Jump to content

unclecameron

Members
  • Content Count

    237
  • Joined

  • Last visited

Community Reputation

0 Neutral

About unclecameron

  • Rank
    swarthog

Contact Methods

  • MSN
    unclecameron@hotmail.com
  • Website URL
    http://www.logicalwebhost.com
  • ICQ
    0

Profile Information

  • Interests
    uh, riding...also flying, but it's more expensive, and generally should involve less dirt
  • Gender
    Male

Recent Profile Visitors

4328 profile views
  1. Seems to be stable after almost a day. I fixed a few issues behind the scenes with the redundant hard drives, but seems okay so far Next, we'll plan a software upgrade, but that'll be another day and I'll let everyone know
  2. Okay, we got the huge migration done, went smooth (with some minor bumps), let me know if there are issues.
  3. Okay, so I did a full backup then upgrade to get the latest version which purportedly fixed some security issues, including the one where some of our files where getting hammered with encrypted Javascript re-directs which tried to do nasty things. I wrote a workaround until I could get the upgrade done, but the upgrade now seems to be going okay. Let me know if still see problems and I'll take a look at them as they come up...now go ride
  4. Okay, so the site was just down for a bit, here's what happened: 1. The underlying operating system had some security and functionality upgrades. Probably 90+% of the time there are really no issues, but for whatever reason this one caused a kernel panic situation (super rare) that required a hard reboot. These aren't very nice because the operating system doesn't do things very cleanly when you're forced to do this, but there's basically no way gracefully around a kernel panic, it's the closest thing to a heart attack that can happen to a server. 2. When the box rebooted, it took its sweet time because it hadn't been rebooted in 259 days (pretty good), so it checked the file system, this took awhile, and is sort of an automatic check that just has to run. 3. After it booted, mysql database server started making rumblings that it was having problems, but then started behaving. Except while the mysql server was behaving, one of the forum tables wasn't, which was why mysql itself didn't throw up any flags. 4. Randy let me know that the database still sucked, so I dug in a bit. While we do very frequent database backups, restoring from backups is a tricky thing because the database is fluid as people post things and change things, so a day old backup even wouldn't really match the rest of the forum structure, basically it could puke even with that frequent of database backups, and even with our frequent full fileset backups. Hazards of the game I guess. 5. I was able to shut down mysql server and do a specific repair on one table that was corrupt, and it worked, so we're back online. Of course, let me know if you still have issues. 6. It's sort of an interesting thing, mostly because we have a freakin ton of posts on this forum, and it's become pretty large, that's a "Good Thing". But now I feel we have to migrate to an active real time database cluster that will keep multiple up-to-the-minute copies of the database, so that if anything happens, we can just point our forum at the hot spare data mirror. Part of the growth process I guess. I don't think Randy or I ever thought it would grow this much, but here we are. We'll let you know what we come up with
  5. okay, we did the massive upgrade, which is probably a good thing, now we have to sort out some bugs, which will take some time probably...stay tuned.
  6. Okay, we did it, now I fully expect to see glitches, but we'll work as soon as we can on fixing them. It was a pretty massive upgrade, so we'll continue testing and tweaking throughout the rest of the week and/or until we get it sorted out. Now we'll see if this helps some old issues we were having, like with the uploads and such...(fingers crossed)
  7. Working on the photo upload issue, will do some testing today. Also, in the next couple days we're working on staging a forum update to the latest version, will send out more info on that as we get it scoped and planned, but just a head's up...
  8. The good news is that the site/forum is back up. And it lives on a newer much better server. The bad news is that the more recent nightly backups of the database were corrupt and I lost posts between the 21st of November and now. That's because my automated backup backed up a half-crashed database, every night and didn't tell me when it started to be corrupt. This meant it would not allow me to rebuild on the destination server in a sane way. The good news is I could roll the database day by day back to when it wasn't corrupt. This also means that now when it's been restored, I can start trying to re-import the lost posts Probably though I can start picking through the remaining database wreckage of the semi-corrupted db and restoring lost posts between those two times...probably. Anyway, things don't suck as much now, but I'll be working on restoring the interim suckiness. I thought will nightly database backups and weekly redundant backups offsite to separate servers would be enough. But I was wrong. Soooo, I'll be busy over the next couple days attempting to restore stuff. Thow stuff if you will, this sucked. Now go ride.
  9. unclecameron

    *** scheduled maintenance tonight ***

    Okay that sucked I got it moved over, but because php is sooo much newer on the new site, it didn't like it (since php has been significantly rewritten relatively recently). Anyway, once it was moved I got it ALMOST working, but then it puked, so here we are still back on the old server until I can debug the situation...stay tuned.
  10. unclecameron

    *** scheduled maintenance tonight ***

    okay, going to disable the site during the switchover. Don't panic, if it doesn't work we can switch it back and try again later.
  11. unclecameron

    *** scheduled maintenance tonight ***

    okay, trying this again today in the next couple hours. When I do the final transfer, I'll actually disable the old server website temporarily after I bring the new one up, in case anyone is still logged in and trying to post, this way we won't have orphan posts.
  12. unclecameron

    *** scheduled maintenance tonight ***

    so the migration is going slower than planned, which means I'll probably re-schedule the remaining parts for tomorrow during the workday, when most folks are, um, working, and now forum-ing
  13. Over the next 2 hours you may experience temporary oddities while we migrate this site to a much newer server. Or you may not. Either way, the site has the potential to act oddly in the interim, so don't post anything for a couple hours. Will update once the migration is complete...
  14. Okay, so some of you noticed that the sdar site started having problems with malware, here's a summary of what happened, who was affected, where we are at now, and what happens to keep things like this from happening again: Summary: the server had problems and now it doesn't (but we're still watching it carefully in case nastiness resurfaces) Deeper dive: 1. Due to a vulnerability in the forum software (from a commercial vendor), hackers were able to find an exploit that was used to attack lots of that vendor's software. The vendor subsequently released a patch, which we installed. 2. In the meantime (get your geek hat on) here's what happened. The attackers uploaded a shell into a directory on the forum that then allowed them access to other areas of the site, but not the server itself, so the attack was localized, which is good. 3. Once the shell was active, they inserted bits of code to the top of all the index pages (there are many in the subdirectories of the forum software) that were encrypted so you couldn't read them, but certain browsers could and did, specifically Internet Explorer versions. 4. When we learned there may be problems, we did a virus scan on the server, and it caught the malicious shell and removed it, but upon further investigation we learned that it had dropped the bits of code into the index pages, which weren't detected because they really aren't malware, just things that tried to direct people to where there was some. 5. If people had Internet Explorer and the malware detected that (and they didn't have a javascript blocker of some kind), the malicious code tried to drop a malicious payload that was really residing on a remote server, most of which were located in China, which tried to install a piece of malware called Blackhole, a not very nice piece of turd-tasticness that is generally to be avoided. 6. We wrote a piece of code that went in and removed the malicious code from all the files where we found problems. After a couple hit-and-miss runs, it looks like we got most if not all of the nasty code among the thousands of files we had to rummage through, so it appears we succeeded, at least for the moment, but these things are nasty, so we'll still to keep working on it. 7. We're still monitoring the site quite closely and reports from folks, so keep them coming. 8. If you use Internet Explorer and have another option for a browser, it might be a good idea to use an alternative until we have some more time to test, this will keep you much safer in this particular instance. 9. If you need to keep using Internet Explorer, consider disabling javascript, there are tutorials on how to do that easily, google is your friend here 10. This whole thing sucked, but then again scammers are always looking for chinks in the armor of popular software, so I guess that means we're popular 11. If any of you are having issues in case the nastiness actually got onto your computer and your anti-malware software didn't catch it (you do have anti-malware software - don't you?) then drop me a line and I'll put up some specific steps you can take to make things better, but hopefully you shouldn't have any problems, as most modern anti-malware software would detect and block this. 12. We've stepped up security on the server itself and will continue to monitor its health over the next few days. 13. Go ride already, it's always a good day (or night) for that
  15. unclecameron

    site crashed

    The site was down for a bit just a few minutes ago, a memory exhaustion problem with mysql, but now it's fixed and back up (as you can tell if you're reading this). I will be migrating this site to a new server shortly, which will give us more memory, cpu and disk, which is nice. Before I migrate it, I'll set up a migration window and let folks know. Sorry about that...
×

Important Information