Jump to content
Sign in to follow this  
unclecameron

corrupt database (fixed)

Recommended Posts

Okay, so the site was just down for a bit, here's what happened:

1. The underlying operating system had some security and functionality upgrades. Probably 90+% of the time there are really no issues, but for whatever reason this one caused a kernel panic situation (super rare) that required a hard reboot. These aren't very nice because the operating system doesn't do things very cleanly when you're forced to do this, but there's basically no way gracefully around a kernel panic, it's the closest thing to a heart attack that can happen to a server.

2. When the box rebooted, it took its sweet time because it hadn't been rebooted in 259 days (pretty good), so it checked the file system, this took awhile, and is sort of an automatic check that just has to run.

3. After it booted, mysql database server started making rumblings that it was having problems, but then started behaving. Except while the mysql server was behaving, one of the forum tables wasn't, which was why mysql itself didn't throw up any flags.

4. Randy let me know that the database still sucked, so I dug in a bit. While we do very frequent database backups, restoring from backups is a tricky thing because the database is fluid as people post things and change things, so a day old backup even wouldn't really match the rest of the forum structure, basically it could puke even with that frequent of database backups, and even with our frequent full fileset backups. Hazards of the game I guess.

5. I was able to shut down mysql server and do a specific repair on one table that was corrupt, and it worked, so we're back online. Of course, let me know if you still have issues.

6. It's sort of an interesting thing, mostly because we have a freakin ton of posts on this forum, and it's become pretty large, that's a "Good Thing". But now I feel we have to migrate to an active real time database cluster that will keep multiple up-to-the-minute copies of the database, so that if anything happens, we can just point our forum at the hot spare data mirror. Part of the growth process I guess. I don't think Randy or I ever thought it would grow this much, but here we are. We'll let you know what we come up with :)

Share this post


Link to post
Share on other sites

you computer guys are amazing, ps i thought i was doing something stupid or forgot to add some update.... thx.

Share this post


Link to post
Share on other sites

Thanks to Cameron for working it out. Like he said, this site and the amount of photos and posts got much bigger than we ever expected.

I told him I want what's best for the site....even if I have to pay more for an even more upgraded server (or whatever).

Share this post


Link to post
Share on other sites

The site is working great for me these days and now that I've spent the time to learn how to post pictures properly I'm almost set. Just need to figure out how to post more than one picture at a time from Photobucket. Right now I go back and forth from Photobucket to SDAR site for each pictureand have to post the picture then edit the post each time.

CiD

Share this post


Link to post
Share on other sites

The site is working great for me these days and now that I've spent the time to learn how to post pictures properly I'm almost set. Just need to figure out how to post more than one picture at a time from Photobucket. Right now I go back and forth from Photobucket to SDAR site for each pictureand have to post the picture then edit the post each time.

CiD

Check out this page on Photo Bucket.

http://support.photobucket.com/hc/en-us/articles/200244519-Bulk-Actions-Delete-Move-Link-and-Tag

Just did these in bulk.

DSC00014.jpg

DSC00030.jpg

DSC00029.jpg

DSC00028.jpg

Share this post


Link to post
Share on other sites

Thanks for all the hard work you do to keep the site running Uncle Cameron!

How has summer in So. Oregon been?

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information