anatomy of a crash, part deux

So say your site gets hacked, and you try fixing the index and config files, as I mentioned in the last post. And you try checking the server logs to see what files were messed with so you can replace them with backups. And you turn on error-reporting in your CMS to try to see what's going wrong. And you Google some of the malicious code you found in your files. And say none of these fixes work, or yield any useful information. What now?

Well next you'll want to search for common hacks to your specific CMS and version, to see if anyone can walk you through fixing them. Here's a pro-tip though: in the end, most fixes will just tell you to install a fresh version of the software, and if you're in my situation that's not an option, so learn from our fail. Set up your website in such a way that disaster recovery is a relatively easy job, or at LEAST a viable option.

I will admit that much of this advice is based on my experience with Joomla and Wordpress. I have much less experience with Drupal, so some of it may not apply there. If you're a Drupal person, and have advice for keeping your site safe from hackers, please post it in the comments!

1. Keep your site software up to date.
   Also your plug-ins. Also your themes. Because chances are, if they're all from reputable sources, the developers will be addressing vulnerabilities as they pop up. The world of hacking is a shifting landscape, and what's secure today is not necessarily secure tomorrow.
 

2. Keep your customization modular.
   In WordPress, this means using a child theme, rather than making changes to the main theme. When you update a theme, it will override any changes you made to those files. Now you're in a situation where you have stop updating your theme, and are thus breaking RULE NUMBER ONE. You will regret this.
 

3. Keep your site root clean.
   Actually, not just the site root, but all its sub-directories. Part of the problem with our site is that the root folder is cluttered up with custom includes, images, project folders, etc. If you're not the one who put them there (as in my case, where I'm taking over a site from someone who is no longer here) it's hard to know what folders are part of the CMS's software, and which ones are not.
   
   In general, if you re-install the software, it should just ignore these unrelated files and folders, but if the software contains new files and folders that have the same name as yours, you can accidentally overwrite your files. I'd say either place these files one level up, OR, if you want them to have the site root's url, create one folder in the site root, and put all of it in there. Clearly mark that that folder is NOT part of the CMS's file structure.
 

4. Documentation!!!
   Srsly. Updating or re-installing your CMS may not be a difficult process, but YOU may not be around when it needs to be done. YOU may be on another continent, or at another job. YOU may have gotten hit on the head or killed those particular brain cells with alcohol. There are so many pieces to a CMS (plugins, templates, images, forms, database(s), etc,) it's supremely helpful to know which of these need to be backed up in like six places before you re-install, so you don't lose the hours and hours of work you put into customizing them. Which leads to...
 

5. Keep backups of important files and folders.
   Yes, I know you're backing up your entire site on a regular basis, because to not do so would be INSANE, but even so, keep an extra copy of important stuff, JUST. IN. CASE. I have a folder on my desktop with my config file, my entire child theme folder, and my custom plugin folders. WordPress is smart, and names the blank config file something else, so when you update, that default config file doesn't overwrite yours, but still. (Remember to update these backups every time you make a change. I got in that habit anyway, because I keep an entirely local copy of the site to make changes to before making them live, so it's kind of a reflex at this point that when I make a change in one place, I update those files everywhere else.)
 

6. Minimize the use of 3rd party modules or vulnerable code.
   Wherever possible on our WordPress site, I used CSS/jQuery to create my own custom features (like our tabbed search box) rather than install another plugins. Plugins can increase the vulnerability of your site, so use them with caution (and, not to drill it into your head or anything, but keep them updated!) We've also made the switch to Google forms for all our forms, so we have the benefit of their security features (and so the forms are connected to off-site databases, rather than databases on our servers.)
 

7. Create a simple html backup site ready to go at all times.
   Honestly, I never even thought of this until the head of Media Services suggested it. Libraries subscribe to many services that are hosted off-site (such as the catalog, research guides, databases, resource managers, and discovery services.) These services are the core of our business, and are still available even when your site is down. Create a simple site that links to whatever services and resources are still available, as well as basic information like hours and contact info. I just downloaded a free CSS template and created a quick and dirty 2 page website that can be put up during downtime (unless the entire server is down. Then I guess you have to put them elsewhere and do a redirect? ACK! SERVER STUFF FRIGHTENS AND CONFUSES ME.)

So ok, there you have it. I am by no means an expert on the topic of hacking, or disaster recovery, or even web development for that matter. This is just an attempt to learn from my own experiences, and to put what I learned out there, just in case it can help someone else in a similar situation. If anyone else has some advice on these matters, please share in the comments!

the anatomy of a crash, part 1

In accordance with Finagle's corollary to Murphy's Law, the website broke the day our sysadmin went on a 2 week vacation on another continent. What's most surprising about this is how little it surprised me. First, what happened:

Our library website is using a very old installation of the Joomla content management system (1.5.7 I believe.) Our implementation, for whatever reason, is insecure. I know very little about Joomla or server security, other than to nod sagely and say, "could be an SQL injection attack" (much in the same way dudes will surround an open car hood, though they know nothing about fixing cars, and say, "it's probably the transmission.")

So last Friday, sometime around 11am, our website stopped being an actual website, and started being just a page that displayed the site title. Not so useful for users, I'd imagine. My first instinct is to look at the main index.php page to see if it's been replaced with a different one. I've dealt with this hack in the past, just the result of some asshole saying "LOOK WHAT I CAN DO!" You just delete the new index file they put in, and put yours back in.

When I checked our index file, it was present, not renamed, and all the content was accounted for. At the end of the file there was a php command that was trying to redirect the site to some .biz.tr website, so I took out that code and figured the problem was fixed. Nope, site was still b0rked. I went into all the sub-folders' index files, and found some malicious code in them too, so I decided to just replace all of them with clean backup versions. Still. B0rked. On to the config file. Everything looks fine there, but I replace it with a back up version anyway.

Also it was about this time I sent out an email to the staff that basically said YES I KNOW THE SITE IS DOWN YOU CAN ALL STOP CALLING ME ABOUT IT.

At this point I'm stumped, so I call the head of Media Services, who maintains the servers. He goes in to check which files were accessed at 11am that day. None. Uh, ok. He has me go into the database, to see if the content looks ok, and it does. It occurs to me that I'm able to get into the site from the admin panel, which is a subfolder in the site root, so it's not that the whole site directory is corrupt. Subpages of the actual site, however, are not loading.

We finally realize that this is not going to be an easy fix, so I put up a temporary webpage linking to common services, most of which are on different servers, so they're fine (catalog, database list, LibGuides, and Google forms.)

The head of Media Services then spent his weekend picking through all the myriad of folders on the server to find workable backups of pretty much all the pieces of the site (which, in a content management system, are many.) He then pieced the site back together, file by file. I honestly don't know the details of how he made this happen, because whenever I asked him about it, he sounded like he was going to cry or murder a baby panda, so I'm just gonna let that go. He obviously has some sort of PTSD, and I don't want to poke the painful memories of "the incident." He did mention something about finding out that the site was actually hacked in June, and was only taken down just now by a remotely-issued command that activated the previously-inserted code. Insidious bastards.

I did a Google search for the spam url I found in the main index page, and it's been injected into tons of insecure Joomla installs. I only mention this because people keep asking what kind of douchebag hacker makes it his life work to take down crappy college library websites. It was just a bot that looked for vulnerable targets. Nothing personal, my friends.

The good news is that I learned many lessons from this whole debacle, and have much to share with you along the lines of "how to make sure this doesn't happen to you because it's not fun." I'm going to put that in another post though, because I need to go pour myself a giant tumbler of whiskey right now. Stay tuned...

Onward Toward the Next Next Gen Library Website

I want to do a series of posts on designing and building a website for an academic library. I'm mostly doing it for myself, as a way to organize my thoughts on the process (since I will be building a library site this semester,) but I thought it might also be useful to share some of the resources I am and will be collecting. And, of course, it's always helpful to have a place to think out loud, so you can get feedback from outside sources.

I do want to make a point before I get started though; these posts are meant to be critique, not criticism. (Or, at least *constructive* criticism.) I don't want to tear anyone down, or poke fun at anyone's efforts. I understand how hard it is to get things done in libraries, given our limited budgets, and the multiple roles everyone on staff must play. Rarely does a library have the staff or the money to hire a specialist to design and build a website, and even when they do, it's hard to find someone with that skill set who also understands libraries and library culture. I feel lucky that I'm in the unique position of having been a reference and instruction librarian for 7 years, and am now in a job that allows me to use the insights I gained working directly with faculty and students to make our website work better for them.

If sacrifices must be made, due to the afore-mentioned budget and time constraints, I can give you a wee piece of advice on the topic: sacrifice style for function, every time. Choose the easiest-to-use software, and make the most popular resources and services the easiest to find. People may snicker at your color choices (I recommend checking out design seeds to help with that) but that's much better than them being frustrated by not being able to find what they're looking for. Snarky users are not necessarily unhappy users. Frustrated users are pretty much *always* unhappy users.

In searching around, I found (and was directed to) several library sites (both academic and public) that I like, and am using for inspiration. Check out:

Harvard Library
Salt Lake City Public Library
Detroit Public Library

I'll tell you why I like them. With the rise in popularity of open source content management systems like Drupal, Joomla and WordPress, websites got very "boxy." Because these CMS's are organized around interchangeable modules, people tend to just drop those module boxes into their site like a puzzle. The above-three sites (at least 2 of which use Drupal) get around that boxy look by using a white background, so the content boxes blend into the rest of the site. This style also works well with responsive design, which uses stylesheets to create a website that is automatically optimized for whatever screen-size or orientation it is viewed at. The trend towards ubiquitous computing means sites have to be able to easily "jump" from desktop to laptop to tablet to smartphone with minimal sacrifices to functionality.

I think for the next couple of posts I might actually pick one site for each, and go over what I like about it. In the mean time, if you're just getting started with web design (or even if you're an old pro,) I highly recommend taking a look at Hongkiat, which has great design and technical tips, as well as tutorials and guides for all aspects of web design and development.

Getting our foot in the door to Moodle(Rooms)

My awesome co-worker and frequent co-conspirator Barbara Arnett has whipped up a little library resource search box that can be added to courses in our course management software (we just switched to Moodle.) We're currently working on convincing them to add it to the school's course shell template, so it will appear by default in every single course (where each prof will have the option to remove it, if they so desire.)

We're running into some trouble because we're using MoodleRooms to host the CMS, and apparently they charge extra for this sort of thing, but I have to imagine that this would really be useful for students. In the past, the course shells included a link to the library's homepage, with no explanation of how to use the site once they got there. We never really analyzed click-throughs from Blackboard (our old CMS), but the link was fairly buried (as opposed to having the search box prominently located on the side of every screen, as would be the case in Moodle.)

Is anyone else working on something like this? Did you have any trouble getting your school to add it to the template? Do you think it will increase online resource usage?

SCREENSHOTS (click to embiggen):

library link in Blackboard, located only under "Course Content"

library search box in Moodle, located in right-hand column of every page