View previous topic :: View next topic |
Author |
Message |
wswartzendruber Veteran
Joined: 23 Mar 2004 Posts: 1261 Location: Idaho, USA
|
Posted: Tue May 28, 2013 4:14 am Post subject: Forums Keep Dying at 04:00 UTC |
|
|
Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.
Comments? |
|
Back to top |
|
|
John-Boy Guru
Joined: 23 Jun 2004 Posts: 442 Location: Desperately seeking moksha in all the wrong places
|
Posted: Wed May 29, 2013 4:11 am Post subject: Re: Forums Keep Dying at 04:00 UTC |
|
|
wswartzendruber wrote: | Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.
Comments? |
Having some problems here too _________________ Like the Roman, I seem to see "the River Tiber foaming with much blood" |
|
Back to top |
|
|
sikpuppy n00b
Joined: 12 Jun 2012 Posts: 34 Location: Central Coast, NSW
|
Posted: Mon Jun 03, 2013 7:14 am Post subject: |
|
|
Wait until 4:01 UTC or get some sleep. |
|
Back to top |
|
|
The Doctor Moderator
Joined: 27 Jul 2010 Posts: 2678
|
Posted: Mon Jun 03, 2013 7:20 am Post subject: |
|
|
I think I see the same. I wonder what is causing this. Maybe its the planned down time when the Roomba runs over the cord. _________________ First things first, but not necessarily in that order.
Apologies if I take a while to respond. I'm currently working on the dematerialization circuit for my blue box. |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Mon Jun 03, 2013 7:28 am Post subject: |
|
|
someone didnt follow the phpbb wiki page i wrote up.... /me glares at gentoo.org |
|
Back to top |
|
|
ppurka Advocate
Joined: 26 Dec 2004 Posts: 3256
|
Posted: Mon Jun 03, 2013 9:22 am Post subject: |
|
|
Subscribing to the thread for free entertainment. _________________ emerge --quiet redefined | E17 vids: I, II | Now using kde5 | e is unstable :-/ |
|
Back to top |
|
|
desultory Bodhisattva
Joined: 04 Nov 2005 Posts: 9410
|
Posted: Tue Jun 04, 2013 2:55 am Post subject: |
|
|
wswartzendruber wrote: | Everyday at approximately 04:00 UTC, these forums take a giant dump for several minutes.
Comments? | It is apparently due to locking issues while preforming the daily backup, which in itself is somewhat understandable given that one of the tables is >16GiB. We are currently exploring some options that might help alleviate the problem, though at the moment no concrete solution is ready for deployment.
666threesixes666 wrote: | someone didnt follow the phpbb wiki page i wrote up.... /me glares at gentoo.org | Post a link, destroy my excuse for retaining such ignorance. Seriously, we are open to whatever tweaks we can make to improve responsiveness. Enhancements that we are not able to make might at least prove interesting or, better still, inspire something that we can do. |
|
Back to top |
|
|
sikpuppy n00b
Joined: 12 Jun 2012 Posts: 34 Location: Central Coast, NSW
|
Posted: Tue Jun 04, 2013 3:04 am Post subject: |
|
|
http://wiki.gentoo.org/wiki/PhpBB this one? I don't get how that's going to help, where's the section on this forum crashing at 4am? An easier page to spirit up is "Things to do when the internet is down..". Which can be literally anything. |
|
Back to top |
|
|
John-Boy Guru
Joined: 23 Jun 2004 Posts: 442 Location: Desperately seeking moksha in all the wrong places
|
Posted: Fri Jun 07, 2013 7:54 pm Post subject: |
|
|
Well that was rather a stretch. _________________ Like the Roman, I seem to see "the River Tiber foaming with much blood" |
|
Back to top |
|
|
a3li Retired Dev
Joined: 02 Sep 2008 Posts: 122 Location: 독일
|
Posted: Fri Jun 07, 2013 9:05 pm Post subject: |
|
|
John-Boy wrote: | Well that was rather a stretch. |
Glad you appreciate the hours of effort we put into restoring the database after a machine crash. _________________ I am Confuism. Do not bother me. |
|
Back to top |
|
|
John-Boy Guru
Joined: 23 Jun 2004 Posts: 442 Location: Desperately seeking moksha in all the wrong places
|
Posted: Sat Jun 08, 2013 4:48 am Post subject: |
|
|
a3li wrote: | Glad you appreciate the hours of effort we put into restoring the database after a machine crash. |
I do, normally there's a post or something - couldn't see any mention of it & I was rather curious as to what happened _________________ Like the Roman, I seem to see "the River Tiber foaming with much blood" |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Sat Jun 08, 2013 4:55 am Post subject: |
|
|
how about DRBD????
http://wiki.gentoo.org/wiki/DRBD
http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1
and stop "backing up?"
ive only tested that phpbb wiki in small scale situations, not gigantic forums like this.... and have to patch it with recaptcha sections. =/ punishment.
what about backing up old content to a locked archive, and purging old content?
because i love desultory's response to my generally punky tude....
i realize this is a round about solution to the problem, and have been reported multiple times for having the philosophy means to ends, rather than mean to end.
Last edited by 666threesixes666 on Sat Jun 08, 2013 3:55 pm; edited 1 time in total |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Sat Jun 08, 2013 5:04 am Post subject: |
|
|
a3li wrote: | John-Boy wrote: | Well that was rather a stretch. |
Glad you appreciate the hours of effort we put into restoring the database after a machine crash. |
are you saying you WANT me to bump them 2002 posts? |
|
Back to top |
|
|
desultory Bodhisattva
Joined: 04 Nov 2005 Posts: 9410
|
Posted: Wed Jun 12, 2013 3:23 am Post subject: |
|
|
Apologies for the delayed response, other things had been somewhat more insistent of my attention.
Interesting, though probably not feasible given the loads it implies.
666threesixes666 wrote: | http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_1
and stop "backing up?" | The database servers are running RAID 10 (mirrored striped sets), I will need to confer with the infrastructure team regarding recovery postures (whether the site would be able to remain available during array rebuilds, if so rw or ro, how long it would take, who is actually able to do it, that sort of thing) before considering dropping live backups entirely.
666threesixes666 wrote: | ive only tested that phpbb wiki in small scale situations, not gigantic forums like this.... and have to patch it with recaptcha sections. =/ punishment. | Fair warning, once your site gets big enough, captchas seem to become decreasingly useful, though at some point we expect to once again test that theory.
666threesixes666 wrote: | what about backing up old content to a locked archive, and purging old content? | Doing so cleanly would require significant modifications to the code and would either introduce significant complexity (to the point where we would essentially have to forgo any plans of migrating to phpbb3) and increased storage overhead (as the table that appears to trigger the recurring downtime would largely be cloned at least once, though it would only need backed up when new content was retired to the archive) or no practical gain (as that table would not shrink at all).
For the moment, it seems like our best bet for avoiding recurring downtime will involve altering our backup scheme such that parts of the database would be regenerated in the even of a catastrophic failure. This implies that certain features would be unavailable during recovery, and that the site might be read only in its entirety until everything was regenerated. At least to me this seems a workable trade, given the twin servers keeping the databases on separate RAID 10 arrays, if it does indeed work and the regeneration process is sufficiently rapid.
Now we just need to implement it all. |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Wed Jun 12, 2013 4:31 am Post subject: |
|
|
drbd is generally kept to LAN setups though it can be done on WAN world wide. i forgot to produce my temporary solution. move the backup to midnight california time, so its not updating @ 8-9pm. since your backing up once every 24 hours, loss of data would be a days worth of registrations and posts, you could make its backups less frequent, since they are not very frequent anyway. or increase frequency to every hour, so down time is in the seconds rather than minutes.
raid is "copy on write" ie update in milliseconds rather than once every 24. drbd on a gigabit switch (or crossover patch cable) would produce a redundant backup of the offending file QUICK. |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Thu Jun 13, 2013 4:32 am Post subject: |
|
|
probably should of went with scorched earth policy.... |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Tue Jun 18, 2013 4:12 am Post subject: |
|
|
rsync is a utility software and network protocol for Unix-like systems (with ports to Windows) that synchronizes files and directories from one location to another while minimizing data transfer by using delta encoding when appropriate.
like syncs the 150 posts for the day, 20k? not copy paste 16gb tables?
watch this crazy voodoo video.
http://youtu.be/UTClD8VS-Kw |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
|
Back to top |
|
|
Ant P. Watchman
Joined: 18 Apr 2009 Posts: 6920
|
Posted: Thu Jul 11, 2013 11:09 am Post subject: |
|
|
Code talks, bullshit walks. Either produce an answer that doesn't involve hand-converting four million posts or shut up already. |
|
Back to top |
|
|
666threesixes666 Veteran
Joined: 31 May 2011 Posts: 1248 Location: 42.68n 85.41w
|
Posted: Sun Jul 21, 2013 7:01 am Post subject: |
|
|
@ 11:30 i started to investigate phpbb/automod mod installer. it seems broken, with files out of order and wrong names. (like they are building it up with out revision control, or auditing the package before publishing it.) i can get it installed, though its admin panel tab throws errors of "Module file ./../includes/acp/acp_mods.php does not contain correct class [acp_mods]" i messaged phpbb about all of this, and looks like 3.1 will be the answer regarding dropping mods in easily. 3.1 will not be stabilized for a while though on their end.
Ant P. i was suggesting letting phpbb2 rot and become the "locked archive" spoken of previously, and setting up a 2nd forum for migration, or not. like gentoo-wiki.info & gentoo-wiki.com & wiki.gentoo.org |
|
Back to top |
|
|
wswartzendruber Veteran
Joined: 23 Mar 2004 Posts: 1261 Location: Idaho, USA
|
Posted: Mon Jul 22, 2013 1:47 am Post subject: |
|
|
The Run-Down
It's gotten to the point where in the evening, these forums are dead half the time I try to login to them. What's the point anymore? How do you think new-comers are going to take this?
So yes, let these forums burn. They've been around for ten years now and should finally be laid to rest. And you can't use the argument "well, they function," because they don't. Let's start up new forums, something a little more modern.
And speaking of modern, the past ten years has seen a number of advancements.
WYSIWYG Editor (Optional)
When I'm trying to make a quick post about something, BBCode is annoying. But that's not a global sentiment. Some of us are bad-ass l33t posters and want it. Cool, just make it optional.
OpenID Connect
Hey, guess what happened yesterday? That's right! The Ubuntu forums got wasted and the hackers now have themselves a nice DB dump. I thoroughly enjoyed having to derive a new password for all of my forum accounts. That was fun, let's do it again!
More Info on OpenID Connect
IPv6 Support
This needs to happen sometime, so we may as well make sure that new forum software works with the new protocol.
EDIT: I'm very willing to step-up and help in any way I can. |
|
Back to top |
|
|
desultory Bodhisattva
Joined: 04 Nov 2005 Posts: 9410
|
Posted: Tue Jul 23, 2013 3:29 am Post subject: |
|
|
wswartzendruber wrote: | The Run-Down
It's gotten to the point where in the evening, these forums are dead half the time I try to login to them. What's the point anymore? How do you think new-comers are going to take this? | Not very well at all.
wswartzendruber wrote: | So yes, let these forums burn. They've been around for ten years now and should finally be laid to rest. And you can't use the argument "well, they function," because they don't. Let's start up new forums, something a little more modern. | The daily downtime is, as previously noted, not due to the forum software; though that in itself hardly negates other deficiencies in it, some of which you pointed out.
wswartzendruber wrote: | And speaking of modern, the past ten years has seen a number of advancements.
WYSIWYG Editor (Optional)
When I'm trying to make a quick post about something, BBCode is annoying. But that's not a global sentiment. Some of us are bad-ass l33t posters and want it. Cool, just make it optional. | If it is part of the package we migrate to, or an existing patchset/plugin/addon freely available for it, it will be available. Conversely, given the amount of work required to make one that would be generally acceptable and generally useful more or less from scratch, I would not expect to see one any time soon otherwise.
wswartzendruber wrote: | OpenID Connect
Hey, guess what happened yesterday? That's right! The Ubuntu forums got wasted and the hackers now have themselves a nice DB dump. I thoroughly enjoyed having to derive a new password for all of my forum accounts. That was fun, let's do it again!
More Info on OpenID Connect | That is not necessarily, yet, on our list of requirements prior to rolling out a, reasonably major, set of changes but it is being considered.
wswartzendruber wrote: | IPv6 Support
This needs to happen sometime, so we may as well make sure that new forum software works with the new protocol. | Agreed, this is the single biggest reason for needing at least major revisions in the existing code base, if not migration to something else entirely.
wswartzendruber wrote: | EDIT: I'm very willing to step-up and help in any way I can. | Thanks for the offer, at the moment I do not know what that might entail aside from testing new features or packages but that could be immensely helpful at times. Expect us when you see us. |
|
Back to top |
|
|
wswartzendruber Veteran
Joined: 23 Mar 2004 Posts: 1261 Location: Idaho, USA
|
Posted: Tue Jul 23, 2013 4:35 am Post subject: |
|
|
Vanilla Forums has WYSIWYG, OpenID, and IPv6...but if FGO ever moves to that...I'm gouging my eyeballs out. |
|
Back to top |
|
|
Bones McCracker Veteran
Joined: 14 Mar 2006 Posts: 1611 Location: U.S.A.
|
Posted: Tue Aug 06, 2013 11:40 am Post subject: |
|
|
I nominate wswartzendruber to be an admin. _________________
patrix_neo wrote: | The human thought: I cannot win.
The ratbrain in me : I can only go forward and that's it. |
|
|
Back to top |
|
|
Goverp Advocate
Joined: 07 Mar 2007 Posts: 2007
|
Posted: Wed Aug 07, 2013 7:43 am Post subject: |
|
|
desultory wrote: | ...
It is apparently due to locking issues while preforming the daily backup, which in itself is somewhat understandable given that one of the tables is >16GiB. We are currently exploring some options that might help alleviate the problem, though at the moment no concrete solution is ready for deployment.
... |
This is probably a useless suggestion, and I've no experience to suggest otherwise, but would using LVM2 snapshots help? _________________ Greybeard |
|
Back to top |
|
|
|