Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Fix inside] nginx proxy crashing on ARM
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Wed Apr 06, 2016 8:07 pm    Post subject: [Fix inside] nginx proxy crashing on ARM Reply with quote

No clue what happened. According to logs it appears to have been functioning until three days ago. tcpdump shows that Nginx does forward the request to the second server and a reply is given, but the web browser reports a network reset. There were no software updates nor restarts of either server. I figured I might as well use the opportunity to update both machines and restart but that didn't fix it. I've had this configuration running for a few years and it's completely beyond me how it could suddenly fail.

Any clues where I could start to trace what's happening in Nginx and why the return from the proxied server gets translated into a network reset?


Last edited by gordonb3 on Thu Apr 14, 2016 2:22 pm; edited 1 time in total
Back to top
View user's profile Send private message
gerdesj
l33t
l33t


Joined: 29 Sep 2005
Posts: 621
Location: Yeovil, Somerset, UK

PostPosted: Sat Apr 09, 2016 11:02 pm    Post subject: Re: nginx proxy suddenly seized to function Reply with quote

gordonb3 wrote:
No clue what happened. According to logs it appears to have been functioning until three days ago. tcpdump shows that Nginx does forward the request to the second server and a reply is given, but the web browser reports a network reset. There were no software updates nor restarts of either server. I figured I might as well use the opportunity to update both machines and restart but that didn't fix it. I've had this configuration running for a few years and it's completely beyond me how it could suddenly fail.

Any clues where I could start to trace what's happening in Nginx and why the return from the proxied server gets translated into a network reset?


Not much to go on. No config snippets, nothing. Is the back end web server actually working?
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5644

PostPosted: Sun Apr 10, 2016 12:02 am    Post subject: Reply with quote

Is the backend a HTTP2-capable server by any chance? nginx claims to support that, but it's really really flaky. Those symptoms sound familiar.
Back to top
View user's profile Send private message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Sun Apr 10, 2016 8:11 am    Post subject: Reply with quote

I don't think config snippets would give more information. It always worked and from one day to the other it seized. The backend server runs Apache 2.4 with php. The Nginx server is configured to serve the static content of the web page. Tcpdump does show Nginx forwarding the request and Apache on the other server answering with a HTTP 200. I currently have iptables configured to route port 80 directly to the backend server and yes the pages run without error but I do want the offload.

Code:

        location / {
                try_files $uri $uri/ @rewrites @proxy;
        }

        location ~ \.php$ {
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header Host $host;
                proxy_pass_header Server;
                proxy_pass http://10.4.26.253:80;
        }

        location @proxy {
                proxy_set_header X-Forwarded-For $remote_addr;
                proxy_set_header Host $host;
                proxy_pass_header Server;
                proxy_pass http://10.4.26.253:80;
        }


Code:
 tcpdump -n -i eth1.426 port 80
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1.426, link-type EN10MB (Ethernet), capture size 262144 bytes
10:09:17.969838 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [S], seq 3714077759, win 29200, options [mss 1460,sackOK,TS val 39119587 ecr 0,nop,wscale 6], length 0
10:09:17.970195 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [S.], seq 1187160860, ack 3714077760, win 28960, options [mss 1460,sackOK,TS val 31091038 ecr 39119587,nop,wscale 6], length 0
10:09:17.970384 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [.], ack 1, win 457, options [nop,nop,TS val 39119587 ecr 31091038], length 0
10:09:17.971911 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [P.], seq 1:353, ack 1, win 457, options [nop,nop,TS val 39119587 ecr 31091038], length 352: HTTP: GET / HTTP/1.0
10:09:17.972223 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [.], ack 353, win 470, options [nop,nop,TS val 31091038 ecr 39119587], length 0
10:09:18.185590 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [P.], seq 1:6528, ack 353, win 470, options [nop,nop,TS val 31091059 ecr 39119587], length 6527: HTTP: HTTP/1.1 200 OK
10:09:18.185782 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [.], ack 6528, win 661, options [nop,nop,TS val 39119608 ecr 31091059], length 0
10:09:18.185966 IP 10.4.26.253.80 > 10.4.26.254.58410: Flags [F.], seq 6528, ack 353, win 470, options [nop,nop,TS val 31091059 ecr 39119587], length 0
10:09:18.187771 IP 10.4.26.254.58410 > 10.4.26.253.80: Flags [R.], seq 353, ack 6529, win 661, options [nop,nop,TS val 39119609 ecr 31091059], length 0
Back to top
View user's profile Send private message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Thu Apr 14, 2016 2:00 pm    Post subject: Reply with quote

I did some further digging into the issue. Adding more verbosity to tcpdump I can see Nginx fetching the page from the backend server. The page size exceeds the ethernet packet size, so I'm seeing two data packets and all seems normal. When Nginx is supposed to pass the page on to the client I'm only seeing a 52 byte (N)ACK packet.

I did note this in the log though:
Code:
2016/04/14 14:32:11 [alert] 11395#0: worker process 11402 exited on signal 11

Apparently the Nginx worker dies somewhere between receiving the page from the backend server and passing it on to the client. I disabled the gzip feature in Nginx, but that didn't help.

I have to come back on something I stated in the opening post however. Crazy how time flies sometimes. The logging appears to have been useless because I was looking at the backend server. As it turns out I did in fact update Nginx from 1.9.6 to 1.9.10-r3 (and 1.9.13 since noticing the issue) after the last access logged by Nginx. Luckily I create binary packages for this system (armvtel) so I could reinstall that previous version, and tadaaa. The proxy is working again. So it would seem that an old issue has risen again.


Edit: confirmed. I did a rebuild of Nginx with the following command line and now the proxy also works with 1.9.13
Code:
LDFLAGS="-fPIE -pie" CFLAGS="-fPIE -pie" emerge -v nginx
Didn't want to investigate whether those were compiler flags or linker flags :oops:
Back to top
View user's profile Send private message
grobihh
n00b
n00b


Joined: 27 Oct 2015
Posts: 5

PostPosted: Fri May 27, 2016 1:01 pm    Post subject: Reply with quote

Hi Gordon,

thanks for digging into that issue and for your solution.

I ran into the same problem and starting to think that I am completly stupid.
I am doing a new install and could not refer to a previously working state.
Lost nearly a week before I stumbled over your post which solved my issue.

Did you report that bug to spare other users the same pain?
After some tries I always look into the Gentoo bug tracker to see if there are know issues with problematic packages.
Here I did not find anything.
Also you are refering to an older nginx issue. Did you report it there to the nginx guys?

We really need to post that in the Gentoo bug tracker. There should be a way to set the compiler flags for arm devices in ebuild as long as there is no nginx fix.
I can do that if you do not want to.

Regards,
Grobi
Back to top
View user's profile Send private message
grobihh
n00b
n00b


Joined: 27 Oct 2015
Posts: 5

PostPosted: Sun Jul 17, 2016 5:40 pm    Post subject: Reply with quote

I created an issue:
https://bugs.gentoo.org/show_bug.cgi?id=589014
Back to top
View user's profile Send private message
gordonb3
Apprentice
Apprentice


Joined: 01 Jul 2015
Posts: 185

PostPosted: Mon Jul 25, 2016 10:03 pm    Post subject: Reply with quote

Forgot to look back at this post. Did upgrade nginx somewhere first half of July without issues though. Currently don't have console access to the box, so can't verify the exact version, but am quite sure it was 1.11.x. Good fortune or really fixed?

From what I gather this is not something that was fixed by adding the required compile flags in the ebuild, but since it does appear to be a recurring issue in the nginx code (the source for the compile flags solution was a bug filed on trac.nginx.org related to version 1.6.3) it'd still be preferable to have these fags enforced - at least for ARM.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum