View previous topic :: View next topic |
Author |
Message |
yabbadabbadont Advocate
Joined: 14 Mar 2003 Posts: 4791 Location: 2 exits past crazy
|
Posted: Mon Aug 17, 2009 4:14 am Post subject: One of the load balanced servers isn't configured correctly. |
|
|
Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly. Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files.
Here is a screen shot to illustrate: http://omploader.org/vMjV6ag
This is awesome... I got the same problem when I hit the Submit button for this post...
See: http://omploader.org/vMjV6bA |
|
Back to top |
|
|
poly_poly-man Advocate
Joined: 06 Dec 2006 Posts: 2477 Location: RIT, NY, US
|
Posted: Mon Aug 17, 2009 5:29 am Post subject: |
|
|
I've noticed konqueror giving "timeout" errors instantly quite a bit today. _________________ iVBORw0KGgoAAAANSUhEUgAAA
avatar: new version of logo - see topic 838248. Potentially still a WiP. |
|
Back to top |
|
|
desultory Bodhisattva
Joined: 04 Nov 2005 Posts: 9410
|
Posted: Mon Aug 17, 2009 7:38 am Post subject: |
|
|
yabbadabbadont wrote: | Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly. | Are you browsing from behind a proxy? If you save the file, what are the contents? If you are encountering the same problem I have been the files should have 0 length.
yabbadabbadont wrote: | Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files. | This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally.
poly_poly-man wrote: | I've noticed konqueror giving "timeout" errors instantly quite a bit today. | This matches my testing with that browser.
If a solution proves elusive during the next day or so, we will probably, temporarily, fall back to a single front end server until we can test the balancing system more thoroughly. |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Mon Aug 17, 2009 7:46 am Post subject: |
|
|
Hello,
I am behind a proxy and I got "zero size reply" messages. _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
yabbadabbadont Advocate
Joined: 14 Mar 2003 Posts: 4791 Location: 2 exits past crazy
|
Posted: Mon Aug 17, 2009 9:22 am Post subject: |
|
|
desultory wrote: | Are you browsing from behind a proxy? If you save the file, what are the contents? If you are encountering the same problem I have been the files should have 0 length. |
Nope. Direct connection to the DSL modem with no proxy configured or running.
It hasn't happened the last few times I've hit the site, so it may have just been some sort of short term thing right after the downtime. I'll post back if I see it again.
Edit: Spoke too soon. It happened right after I posted this when I hit the "View your posts" link. Then it happened twice when I hit the edit button on this post. All three times the php file was zero bytes when saved.
Edit2: It happened when I hit Submit on the first edit... and three times when hitting the edit button to add this edit.... I'm going to just stick to the Ubuntu forums for a while I think. |
|
Back to top |
|
|
richard.scott Veteran
Joined: 19 May 2003 Posts: 1497 Location: Oxfordshire, UK
|
Posted: Mon Aug 17, 2009 10:22 am Post subject: |
|
|
Hi,
I am also getting blank pages!
...I have to refresh each page about 3 times before I see its content.
I'm also behind a proxy of some kind.
However, if I VPN to home and the connect (bypassing the local proxy) then its really quick.
What technology is being used to do the load balancing?
edit: in the few min it took to type this message I then saw "Invalid Session" when I submitted it... not sure if its linked or not.
Rich |
|
Back to top |
|
|
bunder Bodhisattva
Joined: 10 Apr 2004 Posts: 5934
|
Posted: Mon Aug 17, 2009 11:00 am Post subject: |
|
|
i'm also getting blank pages behind my local squid proxy. _________________
Neddyseagoon wrote: | The problem with leaving is that you can only do it once and it reduces your influence. |
banned from #gentoo since sept 2017 |
|
Back to top |
|
|
atrus123 Guru
Joined: 06 Jul 2005 Posts: 339 Location: Annapolis, MD
|
Posted: Mon Aug 17, 2009 11:20 am Post subject: |
|
|
Blank pages ditto! _________________ "I cannot support a movement that exploded spending and borrowing and blames its successor for the debt."
-Andrew Sullivan |
|
Back to top |
|
|
energyman76b Advocate
Joined: 26 Mar 2003 Posts: 2048 Location: Germany
|
Posted: Mon Aug 17, 2009 12:12 pm Post subject: |
|
|
yeah, I am getting lots and lots of timeouts for the last ~12h _________________ Study finds stunning lack of racial, gender, and economic diversity among middle-class white males
I identify as a dirty penismensch. |
|
Back to top |
|
|
poly_poly-man Advocate
Joined: 06 Dec 2006 Posts: 2477 Location: RIT, NY, US
|
Posted: Mon Aug 17, 2009 1:59 pm Post subject: |
|
|
desultory wrote: | If a solution proves elusive during the next day or so, we will probably, temporarily, fall back to a single front end server until we can test the balancing system more thoroughly. | Had you thought of using DNS round-robin? _________________ iVBORw0KGgoAAAANSUhEUgAAA
avatar: new version of logo - see topic 838248. Potentially still a WiP. |
|
Back to top |
|
|
krinn Watchman
Joined: 02 May 2003 Posts: 7470
|
Posted: Mon Aug 17, 2009 3:17 pm Post subject: |
|
|
same problem here, didn't change a thing, and not using proxy... (but my isp should use one) |
|
Back to top |
|
|
mikegpitt Advocate
Joined: 22 May 2004 Posts: 3224
|
Posted: Mon Aug 17, 2009 3:36 pm Post subject: |
|
|
Blank pages here also, both at work and at home. No proxy's... |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Mon Aug 17, 2009 3:44 pm Post subject: |
|
|
It's quite annoying... can't you test before to put directly under production, with a test environment? _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
robbat2 Developer
Joined: 19 Feb 2003 Posts: 82
|
Posted: Mon Aug 17, 2009 5:44 pm Post subject: Re: One of the load balanced servers isn't configured correc |
|
|
yabbadabbadont wrote: | Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly. Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files. | The Apache configuration on the two servers is byte-for-byte identical. If you can reproduce the problem, I'd like to know what MIME type IS being sent with those pages.
desultory wrote: | This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally. | The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets.
poly_poly-man wrote: | Had you thought of using DNS round-robin? | DNS round-robin does not accurately balance the load, due to broken DNS resolvers sorting the results and returning the first one.
For anybody that can trigger it repeatedly, I need to know if any headers are sent with the empty page, or if the entire response is empty.
Also, if you explicitly want to hit one side of the system, there is the DNS entries for [url]forums-web1.gentoo.org[/url] and [url]forums-web2.gentoo.org[/url]. |
|
Back to top |
|
|
bbgermany Veteran
Joined: 21 Feb 2005 Posts: 1844 Location: Oranienburg/Germany
|
Posted: Mon Aug 17, 2009 6:12 pm Post subject: |
|
|
OK, i got a white page right now: https://forums.gentoo.org/viewtopic-t-788306.html
Sourcecode shows nothing. Not even the html tags. Just a white page. The size is 0KB according to the page information.
bb
EDIT: took me three tries to post this. Always got blank posting.php pages. _________________ Desktop: Ryzen 5 5600G, 32GB, 2TB, RX7600
Notebook: Dell XPS 13 9370, 16GB, 1TB
Server #1: Ryzen 5 Pro 4650G, 64GB, 16.5TB
Server #2: Ryzen 4800H, 32GB, 22TB |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Mon Aug 17, 2009 6:18 pm Post subject: |
|
|
Yep, this happens at ANY moment: read message, reply, send reply...
This is TOO ANNOYING... _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
poly_poly-man Advocate
Joined: 06 Dec 2006 Posts: 2477 Location: RIT, NY, US
|
Posted: Mon Aug 17, 2009 6:33 pm Post subject: |
|
|
firing up wireshark now.
EDIT: I'm getting a TCP DUP ACK, and then only outgoing tcp packets, then nothing. http://omploader.org/vMjYzag
Last edited by poly_poly-man on Mon Aug 17, 2009 6:37 pm; edited 1 time in total |
|
Back to top |
|
|
energyman76b Advocate
Joined: 26 Mar 2003 Posts: 2048 Location: Germany
|
Posted: Mon Aug 17, 2009 6:33 pm Post subject: |
|
|
get notification mail, click on link, 3 or 4 instead timeouts before the page loads. Posting something? 5 timeouts can happen. _________________ Study finds stunning lack of racial, gender, and economic diversity among middle-class white males
I identify as a dirty penismensch. |
|
Back to top |
|
|
robbat2 Developer
Joined: 19 Feb 2003 Posts: 82
|
Posted: Mon Aug 17, 2009 8:15 pm Post subject: |
|
|
poly_poly-man wrote: | firing up wireshark now.
EDIT: I'm getting a TCP DUP ACK, and then only outgoing tcp packets, then nothing. http://omploader.org/vMjYzag | Can you make an actual tcpdump packet capture and email it to me please? robbat2@gentoo.org
Please trim it to just the TCP transaction that has the problem, no other traffic. |
|
Back to top |
|
|
robbat2 Developer
Joined: 19 Feb 2003 Posts: 82
|
Posted: Mon Aug 17, 2009 8:34 pm Post subject: |
|
|
I've dropped web2 back out for now, but still using the LB with just web1.
web2 is giving me a bunch of PAX errors |
|
Back to top |
|
|
xaviermiller Bodhisattva
Joined: 23 Jul 2004 Posts: 8706 Location: ~Brussels - Belgique
|
Posted: Tue Aug 18, 2009 6:56 am Post subject: |
|
|
Thank you for your work, robbat2! _________________ Kind regards,
Xavier Miller |
|
Back to top |
|
|
think4urs11 Bodhisattva
Joined: 25 Jun 2003 Posts: 6659 Location: above the cloud
|
Posted: Tue Aug 18, 2009 7:10 am Post subject: |
|
|
robbat2 wrote: | desultory wrote: | This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally. | The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets. |
Issues with session stickyness (as i'm also discovering that bug right now with no proxy in between and working with https while writing this post)?
Is the LB configured to bind particiular clients (by ip, cookie, session-id) to a particiular backend server or is it just a 50:50 'splitter'? _________________ Nothing is secure / Security is always a trade-off with usability / Do not assume anything / Trust no-one, nothing / Paranoia is your friend / Think for yourself |
|
Back to top |
|
|
robbat2 Developer
Joined: 19 Feb 2003 Posts: 82
|
Posted: Tue Aug 18, 2009 7:57 am Post subject: |
|
|
Think4UrS11 wrote: | robbat2 wrote: | desultory wrote: | This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally. | The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets. |
Issues with session stickyness (as i'm also discovering that bug right now with no proxy in between and working with https while writing this post)?
Is the LB configured to bind particiular clients (by ip, cookie, session-id) to a particiular backend server or is it just a 50:50 'splitter'? |
Right now, only web1 is running on the LB. web2 has apache disabled, potentially until it gets a visit from the screwdriver fairy.
If you think you're seeing any persistence issues at all, that's impossible because there can't be any by definition. They would be other problems masquerading as persistence issues.
The configuration in general is a weighted least connections (wlc in IPVS) model, which approximates a 50:50 split in connection durations, not simply connection numbers. There is no persistence configured, and the database replication should be fast enough to carry your sessions over. If it does end up being more problematic, I can enable long-term persistence (so multiple connections take the same path if available) in the www LB side, at the cost of a large tracking table.
ASCII art of the new forums setup
Code: |
Incoming HTTP connections:
--------------------------
X = crossbar switch.
GW = gateway to network.
/-> db1 --> web1
Client -> GW -> IPVS -* X
\-> db2 --> web2
IPVS runs on BOTH database servers, one of which at a time has the (virtual) IP
for the forums.g.o entry. They communicate via VRRP to agree who is in charge.
Outgoing HTTP packets:
----------------------
webN ---> GW ---> client
Backend processing:
-------------------
web1 --> haproxy ---> db1
X
web2 --> haproxy ---> db2
haproxy runs on both web nodes. PHP connects on localhost to haproxy, and gets
an even 50:50 split between database servers. The database connections are
fully proxied.
|
|
|
Back to top |
|
|
poly_poly-man Advocate
Joined: 06 Dec 2006 Posts: 2477 Location: RIT, NY, US
|
Posted: Tue Aug 18, 2009 12:59 pm Post subject: |
|
|
and what exactly is squid doing installed and erroring on web2? seems like it would make more sense without squid... but then again I've never run a high-throughput server (I got 3 hits a day one time!). _________________ iVBORw0KGgoAAAANSUhEUgAAA
avatar: new version of logo - see topic 838248. Potentially still a WiP. |
|
Back to top |
|
|
robbat2 Developer
Joined: 19 Feb 2003 Posts: 82
|
Posted: Tue Aug 18, 2009 5:47 pm Post subject: |
|
|
poly_poly-man wrote: | and what exactly is squid doing installed and erroring on web2? seems like it would make more sense without squid... but then again I've never run a high-throughput server (I got 3 hits a day one time!). |
Squid is installed on both, but not yet enabled, it's for later, to cache the contents of non-static pages. |
|
Back to top |
|
|
|