anatomy of a crash, part deux

So say your site gets hacked, and you try fixing the index and config files, as I mentioned in the last post. And you try checking the server logs to see what files were messed with so you can replace them with backups. And you turn on error-reporting in your CMS to try to see what's going wrong. And you Google some of the malicious code you found in your files. And say none of these fixes work, or yield any useful information. What now?

Well next you'll want to search for common hacks to your specific CMS and version, to see if anyone can walk you through fixing them. Here's a pro-tip though: in the end, most fixes will just tell you to install a fresh version of the software, and if you're in my situation that's not an option, so learn from our fail. Set up your website in such a way that disaster recovery is a relatively easy job, or at LEAST a viable option.

I will admit that much of this advice is based on my experience with Joomla and Wordpress. I have much less experience with Drupal, so some of it may not apply there. If you're a Drupal person, and have advice for keeping your site safe from hackers, please post it in the comments!

1. Keep your site software up to date.
   Also your plug-ins. Also your themes. Because chances are, if they're all from reputable sources, the developers will be addressing vulnerabilities as they pop up. The world of hacking is a shifting landscape, and what's secure today is not necessarily secure tomorrow.
 

2. Keep your customization modular.
   In WordPress, this means using a child theme, rather than making changes to the main theme. When you update a theme, it will override any changes you made to those files. Now you're in a situation where you have stop updating your theme, and are thus breaking RULE NUMBER ONE. You will regret this.
 

3. Keep your site root clean.
   Actually, not just the site root, but all its sub-directories. Part of the problem with our site is that the root folder is cluttered up with custom includes, images, project folders, etc. If you're not the one who put them there (as in my case, where I'm taking over a site from someone who is no longer here) it's hard to know what folders are part of the CMS's software, and which ones are not.
   
   In general, if you re-install the software, it should just ignore these unrelated files and folders, but if the software contains new files and folders that have the same name as yours, you can accidentally overwrite your files. I'd say either place these files one level up, OR, if you want them to have the site root's url, create one folder in the site root, and put all of it in there. Clearly mark that that folder is NOT part of the CMS's file structure.
 

4. Documentation!!!
   Srsly. Updating or re-installing your CMS may not be a difficult process, but YOU may not be around when it needs to be done. YOU may be on another continent, or at another job. YOU may have gotten hit on the head or killed those particular brain cells with alcohol. There are so many pieces to a CMS (plugins, templates, images, forms, database(s), etc,) it's supremely helpful to know which of these need to be backed up in like six places before you re-install, so you don't lose the hours and hours of work you put into customizing them. Which leads to...
 

5. Keep backups of important files and folders.
   Yes, I know you're backing up your entire site on a regular basis, because to not do so would be INSANE, but even so, keep an extra copy of important stuff, JUST. IN. CASE. I have a folder on my desktop with my config file, my entire child theme folder, and my custom plugin folders. WordPress is smart, and names the blank config file something else, so when you update, that default config file doesn't overwrite yours, but still. (Remember to update these backups every time you make a change. I got in that habit anyway, because I keep an entirely local copy of the site to make changes to before making them live, so it's kind of a reflex at this point that when I make a change in one place, I update those files everywhere else.)
 

6. Minimize the use of 3rd party modules or vulnerable code.
   Wherever possible on our WordPress site, I used CSS/jQuery to create my own custom features (like our tabbed search box) rather than install another plugins. Plugins can increase the vulnerability of your site, so use them with caution (and, not to drill it into your head or anything, but keep them updated!) We've also made the switch to Google forms for all our forms, so we have the benefit of their security features (and so the forms are connected to off-site databases, rather than databases on our servers.)
 

7. Create a simple html backup site ready to go at all times.
   Honestly, I never even thought of this until the head of Media Services suggested it. Libraries subscribe to many services that are hosted off-site (such as the catalog, research guides, databases, resource managers, and discovery services.) These services are the core of our business, and are still available even when your site is down. Create a simple site that links to whatever services and resources are still available, as well as basic information like hours and contact info. I just downloaded a free CSS template and created a quick and dirty 2 page website that can be put up during downtime (unless the entire server is down. Then I guess you have to put them elsewhere and do a redirect? ACK! SERVER STUFF FRIGHTENS AND CONFUSES ME.)

So ok, there you have it. I am by no means an expert on the topic of hacking, or disaster recovery, or even web development for that matter. This is just an attempt to learn from my own experiences, and to put what I learned out there, just in case it can help someone else in a similar situation. If anyone else has some advice on these matters, please share in the comments!