the anatomy of a crash, part 1

In accordance with Finagle's corollary to Murphy's Law, the website broke the day our sysadmin went on a 2 week vacation on another continent. What's most surprising about this is how little it surprised me. First, what happened:

Our library website is using a very old installation of the Joomla content management system (1.5.7 I believe.) Our implementation, for whatever reason, is insecure. I know very little about Joomla or server security, other than to nod sagely and say, "could be an SQL injection attack" (much in the same way dudes will surround an open car hood, though they know nothing about fixing cars, and say, "it's probably the transmission.")

So last Friday, sometime around 11am, our website stopped being an actual website, and started being just a page that displayed the site title. Not so useful for users, I'd imagine. My first instinct is to look at the main index.php page to see if it's been replaced with a different one. I've dealt with this hack in the past, just the result of some asshole saying "LOOK WHAT I CAN DO!" You just delete the new index file they put in, and put yours back in.

When I checked our index file, it was present, not renamed, and all the content was accounted for. At the end of the file there was a php command that was trying to redirect the site to some .biz.tr website, so I took out that code and figured the problem was fixed. Nope, site was still b0rked. I went into all the sub-folders' index files, and found some malicious code in them too, so I decided to just replace all of them with clean backup versions. Still. B0rked. On to the config file. Everything looks fine there, but I replace it with a back up version anyway.

Also it was about this time I sent out an email to the staff that basically said YES I KNOW THE SITE IS DOWN YOU CAN ALL STOP CALLING ME ABOUT IT.

At this point I'm stumped, so I call the head of Media Services, who maintains the servers. He goes in to check which files were accessed at 11am that day. None. Uh, ok. He has me go into the database, to see if the content looks ok, and it does. It occurs to me that I'm able to get into the site from the admin panel, which is a subfolder in the site root, so it's not that the whole site directory is corrupt. Subpages of the actual site, however, are not loading.

We finally realize that this is not going to be an easy fix, so I put up a temporary webpage linking to common services, most of which are on different servers, so they're fine (catalog, database list, LibGuides, and Google forms.)

The head of Media Services then spent his weekend picking through all the myriad of folders on the server to find workable backups of pretty much all the pieces of the site (which, in a content management system, are many.) He then pieced the site back together, file by file. I honestly don't know the details of how he made this happen, because whenever I asked him about it, he sounded like he was going to cry or murder a baby panda, so I'm just gonna let that go. He obviously has some sort of PTSD, and I don't want to poke the painful memories of "the incident." He did mention something about finding out that the site was actually hacked in June, and was only taken down just now by a remotely-issued command that activated the previously-inserted code. Insidious bastards.

I did a Google search for the spam url I found in the main index page, and it's been injected into tons of insecure Joomla installs. I only mention this because people keep asking what kind of douchebag hacker makes it his life work to take down crappy college library websites. It was just a bot that looked for vulnerable targets. Nothing personal, my friends.

The good news is that I learned many lessons from this whole debacle, and have much to share with you along the lines of "how to make sure this doesn't happen to you because it's not fun." I'm going to put that in another post though, because I need to go pour myself a giant tumbler of whiskey right now. Stay tuned...