Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Forums Keep Dying at 04:00 UTC
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Gentoo Forums Feedback
View previous topic :: View next topic  
Author Message
wswartzendruber
Veteran
Veteran


Joined: 23 Mar 2004
Posts: 1261
Location: Idaho, USA

PostPosted: Tue May 28, 2013 4:14 am    Post subject: Forums Keep Dying at 04:00 UTC Reply with quote

Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.

Comments?
Back to top
View user's profile Send private message
John-Boy
Guru
Guru


Joined: 23 Jun 2004
Posts: 442
Location: Desperately seeking moksha in all the wrong places

PostPosted: Wed May 29, 2013 4:11 am    Post subject: Re: Forums Keep Dying at 04:00 UTC Reply with quote

wswartzendruber wrote:
Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.

Comments?


Having some problems here too
_________________
Like the Roman, I seem to see "the River Tiber foaming with much blood"
Back to top
View user's profile Send private message
sikpuppy
n00b
n00b


Joined: 12 Jun 2012
Posts: 34
Location: Central Coast, NSW

PostPosted: Mon Jun 03, 2013 7:14 am    Post subject: Reply with quote

Wait until 4:01 UTC or get some sleep.
Back to top
View user's profile Send private message
The Doctor
Moderator
Moderator


Joined: 27 Jul 2010
Posts: 2678

PostPosted: Mon Jun 03, 2013 7:20 am    Post subject: Reply with quote

I think I see the same. I wonder what is causing this. Maybe its the planned down time when the Roomba runs over the cord.
_________________
First things first, but not necessarily in that order.

Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Mon Jun 03, 2013 7:28 am    Post subject: Reply with quote

someone didnt follow the phpbb wiki page i wrote up.... /me glares at gentoo.org :D
Back to top
View user's profile Send private message
ppurka
Advocate
Advocate


Joined: 26 Dec 2004
Posts: 3256

PostPosted: Mon Jun 03, 2013 9:22 am    Post subject: Reply with quote

Subscribing to the thread for free entertainment.
_________________
emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/
Back to top
View user's profile Send private message
desultory
Bodhisattva
Bodhisattva


Joined: 04 Nov 2005
Posts: 9410

PostPosted: Tue Jun 04, 2013 2:55 am    Post subject: Reply with quote

wswartzendruber wrote:
Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.

Comments?
It is apparently due to locking issues while preforming the daily backup, which in itself is somewhat understandable given that one of the tables is >16GiB. We are currently exploring some options that might help alleviate the problem, though at the moment no concrete solution is ready for deployment.

666threesixes666 wrote:
someone didnt follow the phpbb wiki page i wrote up.... /me glares at gentoo.org :D
Post a link, destroy my excuse for retaining such ignorance. Seriously, we are open to whatever tweaks we can make to improve responsiveness. Enhancements that we are not able to make might at least prove interesting or, better still, inspire something that we can do.
Back to top
View user's profile Send private message
sikpuppy
n00b
n00b


Joined: 12 Jun 2012
Posts: 34
Location: Central Coast, NSW

PostPosted: Tue Jun 04, 2013 3:04 am    Post subject: Reply with quote

http://wiki.gentoo.org/wiki/PhpBB this one? I don't get how that's going to help, where's the section on this forum crashing at 4am? An easier page to spirit up is "Things to do when the internet is down..". Which can be literally anything.
Back to top
View user's profile Send private message
John-Boy
Guru
Guru


Joined: 23 Jun 2004
Posts: 442
Location: Desperately seeking moksha in all the wrong places

PostPosted: Fri Jun 07, 2013 7:54 pm    Post subject: Reply with quote

Well that was rather a stretch.
_________________
Like the Roman, I seem to see "the River Tiber foaming with much blood"
Back to top
View user's profile Send private message
a3li
Retired Dev
Retired Dev


Joined: 02 Sep 2008
Posts: 122
Location: 독일

PostPosted: Fri Jun 07, 2013 9:05 pm    Post subject: Reply with quote

John-Boy wrote:
Well that was rather a stretch.


Glad you appreciate the hours of effort we put into restoring the database after a machine crash.
_________________
I am Confuism. Do not bother me.
Back to top
View user's profile Send private message
John-Boy
Guru
Guru


Joined: 23 Jun 2004
Posts: 442
Location: Desperately seeking moksha in all the wrong places

PostPosted: Sat Jun 08, 2013 4:48 am    Post subject: Reply with quote

a3li wrote:
Glad you appreciate the hours of effort we put into restoring the database after a machine crash.


I do, normally there's a post or something - couldn't see any mention of it & I was rather curious as to what happened
_________________
Like the Roman, I seem to see "the River Tiber foaming with much blood"
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Sat Jun 08, 2013 4:55 am    Post subject: Reply with quote

how about DRBD????

http://wiki.gentoo.org/wiki/DRBD

http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1

and stop "backing up?"


ive only tested that phpbb wiki in small scale situations, not gigantic forums like this.... and have to patch it with recaptcha sections. =/ punishment.

what about backing up old content to a locked archive, and purging old content?

because i love desultory's response to my generally punky tude....

i realize this is a round about solution to the problem, and have been reported multiple times for having the philosophy means to ends, rather than mean to end.


Last edited by 666threesixes666 on Sat Jun 08, 2013 3:55 pm; edited 1 time in total
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Sat Jun 08, 2013 5:04 am    Post subject: Reply with quote

a3li wrote:
John-Boy wrote:
Well that was rather a stretch.


Glad you appreciate the hours of effort we put into restoring the database after a machine crash.


are you saying you WANT me to bump them 2002 posts?
Back to top
View user's profile Send private message
desultory
Bodhisattva
Bodhisattva


Joined: 04 Nov 2005
Posts: 9410

PostPosted: Wed Jun 12, 2013 3:23 am    Post subject: Reply with quote

Apologies for the delayed response, other things had been somewhat more insistent of my attention.

666threesixes666 wrote:
how about DRBD????

http://wiki.gentoo.org/wiki/DRBD
Interesting, though probably not feasible given the loads it implies.
666threesixes666 wrote:
http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1

and stop "backing up?"
The database servers are running RAID 10 (mirrored striped sets), I will need to confer with the infrastructure team regarding recovery postures (whether the site would be able to remain available during array rebuilds, if so rw or ro, how long it would take, who is actually able to do it, that sort of thing) before considering dropping live backups entirely.
666threesixes666 wrote:
ive only tested that phpbb wiki in small scale situations, not gigantic forums like this.... and have to patch it with recaptcha sections. =/ punishment.
Fair warning, once your site gets big enough, captchas seem to become decreasingly useful, though at some point we expect to once again test that theory.
666threesixes666 wrote:
what about backing up old content to a locked archive, and purging old content?
Doing so cleanly would require significant modifications to the code and would either introduce significant complexity (to the point where we would essentially have to forgo any plans of migrating to phpbb3) and increased storage overhead (as the table that appears to trigger the recurring downtime would largely be cloned at least once, though it would only need backed up when new content was retired to the archive) or no practical gain (as that table would not shrink at all).

For the moment, it seems like our best bet for avoiding recurring downtime will involve altering our backup scheme such that parts of the database would be regenerated in the even of a catastrophic failure. This implies that certain features would be unavailable during recovery, and that the site might be read only in its entirety until everything was regenerated. At least to me this seems a workable trade, given the twin servers keeping the databases on separate RAID 10 arrays, if it does indeed work and the regeneration process is sufficiently rapid.

Now we just need to implement it all.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Wed Jun 12, 2013 4:31 am    Post subject: Reply with quote

drbd is generally kept to LAN setups though it can be done on WAN world wide. i forgot to produce my temporary solution. move the backup to midnight california time, so its not updating @ 8-9pm. since your backing up once every 24 hours, loss of data would be a days worth of registrations and posts, you could make its backups less frequent, since they are not very frequent anyway. or increase frequency to every hour, so down time is in the seconds rather than minutes.

raid is "copy on write" ie update in milliseconds rather than once every 24. drbd on a gigabit switch (or crossover patch cable) would produce a redundant backup of the offending file QUICK.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Thu Jun 13, 2013 4:32 am    Post subject: Reply with quote

probably should of went with scorched earth policy....
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Tue Jun 18, 2013 4:12 am    Post subject: Reply with quote

rsync is a utility software and network protocol for Unix-like systems (with ports to Windows) that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate.

like syncs the 150 posts for the day, 20k? not copy paste 16gb tables?


watch this crazy voodoo video. 8O

http://youtu.be/UTClD8VS-Kw
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Thu Jul 11, 2013 5:02 am    Post subject: Reply with quote

http://sourceforge.net/projects/icebb/

http://bbpress.org/

http://fluxbb.org/about/

http://www.simplemachines.org/

let us not forget there are alternatives to evade these problems in other ways.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6920

PostPosted: Thu Jul 11, 2013 11:09 am    Post subject: Reply with quote

Code talks, bullshit walks. Either produce an answer that doesn't involve hand-converting four million posts or shut up already.
Back to top
View user's profile Send private message
666threesixes666
Veteran
Veteran


Joined: 31 May 2011
Posts: 1248
Location: 42.68n 85.41w

PostPosted: Sun Jul 21, 2013 7:01 am    Post subject: Reply with quote

@ 11:30 i started to investigate phpbb/automod mod installer. it seems broken, with files out of order and wrong names. (like they are building it up with out revision control, or auditing the package before publishing it.) i can get it installed, though its admin panel tab throws errors of "Module file ./../includes/acp/acp_mods.php does not contain correct class [acp_mods]" i messaged phpbb about all of this, and looks like 3.1 will be the answer regarding dropping mods in easily. 3.1 will not be stabilized for a while though on their end.

Ant P. i was suggesting letting phpbb2 rot and become the "locked archive" spoken of previously, and setting up a 2nd forum for migration, or not. like gentoo-wiki.info & gentoo-wiki.com & wiki.gentoo.org
Back to top
View user's profile Send private message
wswartzendruber
Veteran
Veteran


Joined: 23 Mar 2004
Posts: 1261
Location: Idaho, USA

PostPosted: Mon Jul 22, 2013 1:47 am    Post subject: Reply with quote

The Run-Down

It's gotten to the point where in the evening, these forums are dead half the time I try to login to them. What's the point anymore? How do you think new-comers are going to take this?

So yes, let these forums burn. They've been around for ten years now and should finally be laid to rest. And you can't use the argument "well, they function," because they don't. Let's start up new forums, something a little more modern.

And speaking of modern, the past ten years has seen a number of advancements.

WYSIWYG Editor (Optional)

When I'm trying to make a quick post about something, BBCode is annoying. But that's not a global sentiment. Some of us are bad-ass l33t posters and want it. Cool, just make it optional.

OpenID Connect

Hey, guess what happened yesterday? That's right! The Ubuntu forums got wasted and the hackers now have themselves a nice DB dump. I thoroughly enjoyed having to derive a new password for all of my forum accounts. That was fun, let's do it again!

More Info on OpenID Connect

IPv6 Support

This needs to happen sometime, so we may as well make sure that new forum software works with the new protocol.

EDIT: I'm very willing to step-up and help in any way I can.
Back to top
View user's profile Send private message
desultory
Bodhisattva
Bodhisattva


Joined: 04 Nov 2005
Posts: 9410

PostPosted: Tue Jul 23, 2013 3:29 am    Post subject: Reply with quote

wswartzendruber wrote:
The Run-Down

It's gotten to the point where in the evening, these forums are dead half the time I try to login to them. What's the point anymore? How do you think new-comers are going to take this?
Not very well at all.
wswartzendruber wrote:
So yes, let these forums burn. They've been around for ten years now and should finally be laid to rest. And you can't use the argument "well, they function," because they don't. Let's start up new forums, something a little more modern.
The daily downtime is, as previously noted, not due to the forum software; though that in itself hardly negates other deficiencies in it, some of which you pointed out.
wswartzendruber wrote:
And speaking of modern, the past ten years has seen a number of advancements.

WYSIWYG Editor (Optional)

When I'm trying to make a quick post about something, BBCode is annoying. But that's not a global sentiment. Some of us are bad-ass l33t posters and want it. Cool, just make it optional.
If it is part of the package we migrate to, or an existing patchset/plugin/addon freely available for it, it will be available. Conversely, given the amount of work required to make one that would be generally acceptable and generally useful more or less from scratch, I would not expect to see one any time soon otherwise.
wswartzendruber wrote:
OpenID Connect

Hey, guess what happened yesterday? That's right! The Ubuntu forums got wasted and the hackers now have themselves a nice DB dump. I thoroughly enjoyed having to derive a new password for all of my forum accounts. That was fun, let's do it again!

More Info on OpenID Connect
That is not necessarily, yet, on our list of requirements prior to rolling out a, reasonably major, set of changes but it is being considered.

wswartzendruber wrote:
IPv6 Support

This needs to happen sometime, so we may as well make sure that new forum software works with the new protocol.
Agreed, this is the single biggest reason for needing at least major revisions in the existing code base, if not migration to something else entirely.
wswartzendruber wrote:
EDIT: I'm very willing to step-up and help in any way I can.
Thanks for the offer, at the moment I do not know what that might entail aside from testing new features or packages but that could be immensely helpful at times. Expect us when you see us.
Back to top
View user's profile Send private message
wswartzendruber
Veteran
Veteran


Joined: 23 Mar 2004
Posts: 1261
Location: Idaho, USA

PostPosted: Tue Jul 23, 2013 4:35 am    Post subject: Reply with quote

Vanilla Forums has WYSIWYG, OpenID, and IPv6...but if FGO ever moves to that...I'm gouging my eyeballs out.
Back to top
View user's profile Send private message
Bones McCracker
Veteran
Veteran


Joined: 14 Mar 2006
Posts: 1611
Location: U.S.A.

PostPosted: Tue Aug 06, 2013 11:40 am    Post subject: Reply with quote

I nominate wswartzendruber to be an admin.
_________________
patrix_neo wrote:
The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it.
Back to top
View user's profile Send private message
Goverp
Veteran
Veteran


Joined: 07 Mar 2007
Posts: 1972

PostPosted: Wed Aug 07, 2013 7:43 am    Post subject: Reply with quote

desultory wrote:
...
It is apparently due to locking issues while preforming the daily backup, which in itself is somewhat understandable given that one of the tables is >16GiB. We are currently exploring some options that might help alleviate the problem, though at the moment no concrete solution is ready for deployment.
...

This is probably a useless suggestion, and I've no experience to suggest otherwise, but would using LVM2 snapshots help?
_________________
Greybeard
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Gentoo Forums Feedback All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum