| View previous topic :: View next topic |
| Author |
Message |
gordonb3
Apprentice
Joined: 01 Jul 2015
Posts: 185
|
 Posted: Wed Apr 06, 2016 8:07 pm Post subject: [Fix inside] nginx proxy crashing on ARM |
 |
|
No clue what happened. According to logs it appears to have been functioning until three days ago. tcpdump shows that Nginx does forward the request to the second server and a reply is given, but the web browser reports a network reset. There were no software updates nor restarts of either server. I figured I might as well use the opportunity to update both machines and restart but that didn't fix it. I've had this configuration running for a few years and it's completely beyond me how it could suddenly fail.
Any clues where I could start to trace what's happening in Nginx and why the return from the proxied server gets translated into a network reset?
Last edited by gordonb3 on Thu Apr 14, 2016 2:22 pm; edited 1 time in total |
|
| Back to top |
|
 |
gerdesj
l33t
Joined: 29 Sep 2005
Posts: 621
Location: Yeovil, Somerset, UK
|
| Posted: Sat Apr 09, 2016 11:02 pm Post subject: Re: nginx proxy suddenly seized to function |
|
|
| gordonb3 wrote: | No clue what happened. According to logs it appears to have been functioning until three days ago. tcpdump shows that Nginx does forward the request to the second server and a reply is given, but the web browser reports a network reset. There were no software updates nor restarts of either server. I figured I might as well use the opportunity to update both machines and restart but that didn't fix it. I've had this configuration running for a few years and it's completely beyond me how it could suddenly fail.
Any clues where I could start to trace what's happening in Nginx and why the return from the proxied server gets translated into a network reset? |
Not much to go on. No config snippets, nothing. Is the back end web server actually working? |
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Sun Apr 10, 2016 12:02 am Post subject: |
|
|
| Is the backend a HTTP2-capable server by any chance? nginx claims to support that, but it's really really flaky. Those symptoms sound familiar. |
|
| Back to top |
|
|
gordonb3
Apprentice
Joined: 01 Jul 2015
Posts: 185
|
| Posted: Sun Apr 10, 2016 8:11 am Post subject: |
|
|
I don't think config snippets would give more information. It always worked and from one day to the other it seized. The backend server runs Apache 2.4 with php. The Nginx server is configured to serve the static content of the web page. Tcpdump does show Nginx forwarding the request and Apache on the other server answering with a HTTP 200. I currently have iptables configured to route port 80 directly to the backend server and yes the pages run without error but I do want the offload.
| Code: |
location / {
try_files $uri $uri/ @rewrites @proxy;
}
location ~ \.php$ {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass_header Server;
proxy_pass http://10.4.26.253:80;
}
location @proxy {
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_pass_header Server;
proxy_pass http://10.4.26.253:80;
}
|
| Code: | tcpdump -n -i eth1.426 port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.426, link-type EN10MB (Ethernet), capture size 262144 bytes
10:09:17.969838 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [S], seq 3714077759, win 29200, options [mss 1460,sackOK,TS val 39119587 ecr 0,nop,wscale 6], length 0
10:09:17.970195 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [S.], seq 1187160860, ack 3714077760, win 28960, options [mss 1460,sackOK,TS val 31091038 ecr 39119587,nop,wscale 6], length 0
10:09:17.970384 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [.], ack 1, win 457, options [nop,nop,TS val 39119587 ecr 31091038], length 0
10:09:17.971911 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [P.], seq 1:353, ack 1, win 457, options [nop,nop,TS val 39119587 ecr 31091038], length 352: HTTP: GET / HTTP/1.0
10:09:17.972223 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [.], ack 353, win 470, options [nop,nop,TS val 31091038 ecr 39119587], length 0
10:09:18.185590 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [P.], seq 1:6528, ack 353, win 470, options [nop,nop,TS val 31091059 ecr 39119587], length 6527: HTTP: HTTP/1.1 200 OK
10:09:18.185782 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [.], ack 6528, win 661, options [nop,nop,TS val 39119608 ecr 31091059], length 0
10:09:18.185966 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [F.], seq 6528, ack 353, win 470, options [nop,nop,TS val 31091059 ecr 39119587], length 0
10:09:18.187771 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [R.], seq 353, ack 6529, win 661, options [nop,nop,TS val 39119609 ecr 31091059], length 0
|
|
|
| Back to top |
|
|
gordonb3
Apprentice
Joined: 01 Jul 2015
Posts: 185
|
| Posted: Thu Apr 14, 2016 2:00 pm Post subject: |
|
|
I did some further digging into the issue. Adding more verbosity to tcpdump I can see Nginx fetching the page from the backend server. The page size exceeds the ethernet packet size, so I'm seeing two data packets and all seems normal. When Nginx is supposed to pass the page on to the client I'm only seeing a 52 byte (N)ACK packet.
I did note this in the log though:
| Code: | | 2016/04/14 14:32:11 [alert] 11395#0: worker process 11402 exited on signal 11 |
Apparently the Nginx worker dies somewhere between receiving the page from the backend server and passing it on to the client. I disabled the gzip feature in Nginx, but that didn't help.
I have to come back on something I stated in the opening post however. Crazy how time flies sometimes. The logging appears to have been useless because I was looking at the backend server. As it turns out I did in fact update Nginx from 1.9.6 to 1.9.10-r3 (and 1.9.13 since noticing the issue) after the last access logged by Nginx. Luckily I create binary packages for this system (armvtel) so I could reinstall that previous version, and tadaaa. The proxy is working again. So it would seem that an old issue has risen again.
Edit: confirmed. I did a rebuild of Nginx with the following command line and now the proxy also works with 1.9.13
| Code: | | LDFLAGS="-fPIE -pie" CFLAGS="-fPIE -pie" emerge -v nginx |
Didn't want to investigate whether those were compiler flags or linker flags  |
|
| Back to top |
|
|
grobihh
n00b
Joined: 27 Oct 2015
Posts: 5
|
| Posted: Fri May 27, 2016 1:01 pm Post subject: |
|
|
Hi Gordon,
thanks for digging into that issue and for your solution.
I ran into the same problem and starting to think that I am completly stupid.
I am doing a new install and could not refer to a previously working state.
Lost nearly a week before I stumbled over your post which solved my issue.
Did you report that bug to spare other users the same pain?
After some tries I always look into the Gentoo bug tracker to see if there are know issues with problematic packages.
Here I did not find anything.
Also you are refering to an older nginx issue. Did you report it there to the nginx guys?
We really need to post that in the Gentoo bug tracker. There should be a way to set the compiler flags for arm devices in ebuild as long as there is no nginx fix.
I can do that if you do not want to.
Regards,
Grobi |
|
| Back to top |
|
|
grobihh
n00b
Joined: 27 Oct 2015
Posts: 5
|
|
| Back to top |
|
|
gordonb3
Apprentice
Joined: 01 Jul 2015
Posts: 185
|
| Posted: Mon Jul 25, 2016 10:03 pm Post subject: |
|
|
Forgot to look back at this post. Did upgrade nginx somewhere first half of July without issues though. Currently don't have console access to the box, so can't verify the exact version, but am quite sure it was 1.11.x. Good fortune or really fixed?
From what I gather this is not something that was fixed by adding the required compile flags in the ebuild, but since it does appear to be a recurring issue in the nginx code (the source for the compile flags solution was a bug filed on trac.nginx.org related to version 1.6.3) it'd still be preferable to have these fags enforced - at least for ARM. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
