Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
One of the load balanced servers isn't configured correctly.
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Forums Feedback
View previous topic :: View next topic  
Author Message
yabbadabbadont
Advocate
Advocate


Joined: 14 Mar 2003
Posts: 4791
Location: 2 exits past crazy

PostPosted: Mon Aug 17, 2009 4:14 am    Post subject: One of the load balanced servers isn't configured correctly. Reply with quote

Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly. Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files.

Here is a screen shot to illustrate: http://omploader.org/vMjV6ag

This is awesome... I got the same problem when I hit the Submit button for this post... :lol:

See: http://omploader.org/vMjV6bA
Back to top
View user's profile Send private message
poly_poly-man
Advocate
Advocate


Joined: 06 Dec 2006
Posts: 2477
Location: RIT, NY, US

PostPosted: Mon Aug 17, 2009 5:29 am    Post subject: Reply with quote

I've noticed konqueror giving "timeout" errors instantly quite a bit today.
_________________
iVBORw0KGgoAAAANSUhEUgAAA

avatar: new version of logo - see topic 838248. Potentially still a WiP.
Back to top
View user's profile Send private message
desultory
Bodhisattva
Bodhisattva


Joined: 04 Nov 2005
Posts: 9410

PostPosted: Mon Aug 17, 2009 7:38 am    Post subject: Reply with quote

yabbadabbadont wrote:
Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly.
Are you browsing from behind a proxy? If you save the file, what are the contents? If you are encountering the same problem I have been the files should have 0 length.
yabbadabbadont wrote:
Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files.
This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally.

poly_poly-man wrote:
I've noticed konqueror giving "timeout" errors instantly quite a bit today.
This matches my testing with that browser.

If a solution proves elusive during the next day or so, we will probably, temporarily, fall back to a single front end server until we can test the balancing system more thoroughly.
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8704
Location: ~Brussels - Belgique

PostPosted: Mon Aug 17, 2009 7:46 am    Post subject: Reply with quote

Hello,

I am behind a proxy and I got "zero size reply" messages.
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
yabbadabbadont
Advocate
Advocate


Joined: 14 Mar 2003
Posts: 4791
Location: 2 exits past crazy

PostPosted: Mon Aug 17, 2009 9:22 am    Post subject: Reply with quote

desultory wrote:
Are you browsing from behind a proxy? If you save the file, what are the contents? If you are encountering the same problem I have been the files should have 0 length.

Nope. Direct connection to the DSL modem with no proxy configured or running.

It hasn't happened the last few times I've hit the site, so it may have just been some sort of short term thing right after the downtime. I'll post back if I see it again.

Edit: Spoke too soon. It happened right after I posted this when I hit the "View your posts" link. Then it happened twice when I hit the edit button on this post. All three times the php file was zero bytes when saved.

Edit2: It happened when I hit Submit on the first edit... and three times when hitting the edit button to add this edit.... I'm going to just stick to the Ubuntu forums for a while I think. :lol:
Back to top
View user's profile Send private message
richard.scott
Veteran
Veteran


Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK

PostPosted: Mon Aug 17, 2009 10:22 am    Post subject: Reply with quote

Hi,

I am also getting blank pages!
...I have to refresh each page about 3 times before I see its content.

I'm also behind a proxy of some kind.

However, if I VPN to home and the connect (bypassing the local proxy) then its really quick.

What technology is being used to do the load balancing?

edit: in the few min it took to type this message I then saw "Invalid Session" when I submitted it... not sure if its linked or not.

Rich
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5933

PostPosted: Mon Aug 17, 2009 11:00 am    Post subject: Reply with quote

i'm also getting blank pages behind my local squid proxy.
_________________
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.

banned from #gentoo since sept 2017
Back to top
View user's profile Send private message
atrus123
Guru
Guru


Joined: 06 Jul 2005
Posts: 339
Location: Annapolis, MD

PostPosted: Mon Aug 17, 2009 11:20 am    Post subject: Reply with quote

Blank pages ditto!
_________________
"I cannot support a movement that exploded spending and borrowing and blames its successor for the debt."
-Andrew Sullivan
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2048
Location: Germany

PostPosted: Mon Aug 17, 2009 12:12 pm    Post subject: Reply with quote

yeah, I am getting lots and lots of timeouts for the last ~12h
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
poly_poly-man
Advocate
Advocate


Joined: 06 Dec 2006
Posts: 2477
Location: RIT, NY, US

PostPosted: Mon Aug 17, 2009 1:59 pm    Post subject: Reply with quote

desultory wrote:
If a solution proves elusive during the next day or so, we will probably, temporarily, fall back to a single front end server until we can test the balancing system more thoroughly.
Had you thought of using DNS round-robin?
_________________
iVBORw0KGgoAAAANSUhEUgAAA

avatar: new version of logo - see topic 838248. Potentially still a WiP.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7470

PostPosted: Mon Aug 17, 2009 3:17 pm    Post subject: Reply with quote

same problem here, didn't change a thing, and not using proxy... (but my isp should use one)
Back to top
View user's profile Send private message
mikegpitt
Advocate
Advocate


Joined: 22 May 2004
Posts: 3224

PostPosted: Mon Aug 17, 2009 3:36 pm    Post subject: Reply with quote

Blank pages here also, both at work and at home. No proxy's...
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8704
Location: ~Brussels - Belgique

PostPosted: Mon Aug 17, 2009 3:44 pm    Post subject: Reply with quote

It's quite annoying... can't you test before to put directly under production, with a test environment?
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
robbat2
Developer
Developer


Joined: 19 Feb 2003
Posts: 82

PostPosted: Mon Aug 17, 2009 5:44 pm    Post subject: Re: One of the load balanced servers isn't configured correc Reply with quote

yabbadabbadont wrote:
Every few times when I hit the "View your posts", or other shortcut links, I am prompted by firefox with the save file dialog instead of the page opening correctly. Since it has only started happening since today's down time, I'm guessing that one (or more) of the load balanced servers doesn't have the correct mime type set for php files.
The Apache configuration on the two servers is byte-for-byte identical. If you can reproduce the problem, I'd like to know what MIME type IS being sent with those pages.

desultory wrote:
This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally.
The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets.

poly_poly-man wrote:
Had you thought of using DNS round-robin?
DNS round-robin does not accurately balance the load, due to broken DNS resolvers sorting the results and returning the first one.

For anybody that can trigger it repeatedly, I need to know if any headers are sent with the empty page, or if the entire response is empty.

Also, if you explicitly want to hit one side of the system, there is the DNS entries for [url]forums-web1.gentoo.org[/url] and [url]forums-web2.gentoo.org[/url].
Back to top
View user's profile Send private message
bbgermany
Veteran
Veteran


Joined: 21 Feb 2005
Posts: 1844
Location: Oranienburg/Germany

PostPosted: Mon Aug 17, 2009 6:12 pm    Post subject: Reply with quote

OK, i got a white page right now: https://forums.gentoo.org/viewtopic-t-788306.html

Sourcecode shows nothing. Not even the html tags. Just a white page. The size is 0KB according to the page information.

bb

EDIT: took me three tries to post this. Always got blank posting.php pages.
_________________
Desktop: Ryzen 5 5600G, 32GB, 2TB, RX7600
Notebook: Dell XPS 13 9370, 16GB, 1TB
Server #1: Ryzen 5 Pro 4650G, 64GB, 16.5TB
Server #2: Ryzen 4800H, 32GB, 22TB
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8704
Location: ~Brussels - Belgique

PostPosted: Mon Aug 17, 2009 6:18 pm    Post subject: Reply with quote

Yep, this happens at ANY moment: read message, reply, send reply...

This is TOO ANNOYING... :roll:
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
poly_poly-man
Advocate
Advocate


Joined: 06 Dec 2006
Posts: 2477
Location: RIT, NY, US

PostPosted: Mon Aug 17, 2009 6:33 pm    Post subject: Reply with quote

firing up wireshark now.


EDIT: I'm getting a TCP DUP ACK, and then only outgoing tcp packets, then nothing. http://omploader.org/vMjYzag


Last edited by poly_poly-man on Mon Aug 17, 2009 6:37 pm; edited 1 time in total
Back to top
View user's profile Send private message
energyman76b
Advocate
Advocate


Joined: 26 Mar 2003
Posts: 2048
Location: Germany

PostPosted: Mon Aug 17, 2009 6:33 pm    Post subject: Reply with quote

get notification mail, click on link, 3 or 4 instead timeouts before the page loads. Posting something? 5 timeouts can happen.
_________________
Study finds stunning lack of racial, gender, and economic diversity among middle-class white males

I identify as a dirty penismensch.
Back to top
View user's profile Send private message
robbat2
Developer
Developer


Joined: 19 Feb 2003
Posts: 82

PostPosted: Mon Aug 17, 2009 8:15 pm    Post subject: Reply with quote

poly_poly-man wrote:
firing up wireshark now.
EDIT: I'm getting a TCP DUP ACK, and then only outgoing tcp packets, then nothing. http://omploader.org/vMjYzag
Can you make an actual tcpdump packet capture and email it to me please? robbat2@gentoo.org

Please trim it to just the TCP transaction that has the problem, no other traffic.
Back to top
View user's profile Send private message
robbat2
Developer
Developer


Joined: 19 Feb 2003
Posts: 82

PostPosted: Mon Aug 17, 2009 8:34 pm    Post subject: Reply with quote

I've dropped web2 back out for now, but still using the LB with just web1.

web2 is giving me a bunch of PAX errors :-(
Back to top
View user's profile Send private message
xaviermiller
Bodhisattva
Bodhisattva


Joined: 23 Jul 2004
Posts: 8704
Location: ~Brussels - Belgique

PostPosted: Tue Aug 18, 2009 6:56 am    Post subject: Reply with quote

Thank you for your work, robbat2!
_________________
Kind regards,
Xavier Miller
Back to top
View user's profile Send private message
think4urs11
Bodhisattva
Bodhisattva


Joined: 25 Jun 2003
Posts: 6659
Location: above the cloud

PostPosted: Tue Aug 18, 2009 7:10 am    Post subject: Reply with quote

robbat2 wrote:
desultory wrote:
This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally.
The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets.

Issues with session stickyness (as i'm also discovering that bug right now with no proxy in between and working with https while writing this post)?
Is the LB configured to bind particiular clients (by ip, cookie, session-id) to a particiular backend server or is it just a 50:50 'splitter'?
_________________
Nothing is secure / Security is always a trade-off with usability / Do not assume anything / Trust no-one, nothing / Paranoia is your friend / Think for yourself
Back to top
View user's profile Send private message
robbat2
Developer
Developer


Joined: 19 Feb 2003
Posts: 82

PostPosted: Tue Aug 18, 2009 7:57 am    Post subject: Reply with quote

Think4UrS11 wrote:
robbat2 wrote:
desultory wrote:
This appears to be due to the load balancer itself, as both of the servers behind it seem to be working normally.
The load balancer is purely at a TCP level (IPVS). It does not proxy or even see the response packets.

Issues with session stickyness (as i'm also discovering that bug right now with no proxy in between and working with https while writing this post)?
Is the LB configured to bind particiular clients (by ip, cookie, session-id) to a particiular backend server or is it just a 50:50 'splitter'?


Right now, only web1 is running on the LB. web2 has apache disabled, potentially until it gets a visit from the screwdriver fairy.
If you think you're seeing any persistence issues at all, that's impossible because there can't be any by definition. They would be other problems masquerading as persistence issues.

The configuration in general is a weighted least connections (wlc in IPVS) model, which approximates a 50:50 split in connection durations, not simply connection numbers. There is no persistence configured, and the database replication should be fast enough to carry your sessions over. If it does end up being more problematic, I can enable long-term persistence (so multiple connections take the same path if available) in the www LB side, at the cost of a large tracking table.

ASCII art of the new forums setup
Code:

Incoming HTTP connections:
--------------------------
X = crossbar switch.
GW = gateway to network.

                       /-> db1 --> web1
Client -> GW -> IPVS -*         X
                       \-> db2 --> web2
               
IPVS runs on BOTH database servers, one of which at a time has the (virtual) IP
for the forums.g.o entry. They communicate via VRRP to agree who is in charge.

Outgoing HTTP packets:
----------------------
webN ---> GW ---> client

Backend processing:
-------------------
web1 --> haproxy ---> db1
                  X
web2 --> haproxy ---> db2

haproxy runs on both web nodes. PHP connects on localhost to haproxy, and gets
an even 50:50 split between database servers. The database connections are
fully proxied.
Back to top
View user's profile Send private message
poly_poly-man
Advocate
Advocate


Joined: 06 Dec 2006
Posts: 2477
Location: RIT, NY, US

PostPosted: Tue Aug 18, 2009 12:59 pm    Post subject: Reply with quote

and what exactly is squid doing installed and erroring on web2? seems like it would make more sense without squid... but then again I've never run a high-throughput server (I got 3 hits a day one time!).
_________________
iVBORw0KGgoAAAANSUhEUgAAA

avatar: new version of logo - see topic 838248. Potentially still a WiP.
Back to top
View user's profile Send private message
robbat2
Developer
Developer


Joined: 19 Feb 2003
Posts: 82

PostPosted: Tue Aug 18, 2009 5:47 pm    Post subject: Reply with quote

poly_poly-man wrote:
and what exactly is squid doing installed and erroring on web2? seems like it would make more sense without squid... but then again I've never run a high-throughput server (I got 3 hits a day one time!).

Squid is installed on both, but not yet enabled, it's for later, to cache the contents of non-static pages.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Forums Feedback All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum