If you are one of the regular readers of ShoutMeLoud, you might have noticed frequent downtime and theme changes in last couple of days. There are many reasons for the same and one of them was my hosting server. I was on Linode VPS and despite of being one of the powerful unmanaged VPS server, my site was going down again and again.
Though the reason might be due to some bugged WordPress plugin and also some issues with updated W3 total cache plugin. Last night I started facing some issues with Thesis theme as it was not able to save my design options and I think it was because of some database issue.
Nonetheless, yesterday I almost worked 18 hr in last 24 hrs to fix the issue and in the process I learned lots of things and learned how image hotlinkers can lead to your. My Error log was filled with request originating from other sites who were using ShoutMeLoud skin (Copied and pasted) and many of them didn’t even bother to remove the image links or upload it on their server. In 2 days, my error.log was almost 30Gb, I believe some lucky leecher might have made front page of Digg or similar sites, which created 30+ gigs of error log file.
Anyways, I had almost a nightmare fixing the issue which involves reinstallation of WordPress, completely uninstalling Thesis theme and reinstalling Thesis theme, find plugins which might be causing issues and clean orphaned database tables.
Uninstalling Thesis theme completely:
When we talk about completely uninstalling Thesis theme, it means not only removing theme files but also removing the tables associated in your WordPress database. Since Wp phpmyadmin is no more a secure plugin, I used portable Phpmyadmin to access my site using dashboard. Here I’m sharing a quick tip for those who are trying to do a fresh installation of Thesis on a server which is already running Thesis.
Go to Thesis Manage options and restore default site and design option. You might like open design and site option in a new tab before restoring the default, so that you can quickly copy the settings at later stage. Now deactivate the Thesis theme and go to your database from phpmyadmin. Under Wp_options search for option name Thesis_design_options and Thesis_site_options, delete them.
Once done, you might like to change your custom folder back to custom_sample and activate Thesis. Alternatively, you can remove Thesis theme and install Thesis theme from scratch.
I had to do all this, as Thesis design option was giving some error and there was something wrong with site teaser.
P.s: I also installed WordPress SEO plugin by yoast and used SEO data importer plugin to copy all SEO meta values from Thesis to WordPress SEO. This is to ensure removing associated table with Thesis should not affect my SEO meta values.
Reinstall WordPress blog:
This is just to ensure that every file is intact and there is nothing wrong with the core WordPress file. From your WordPress dashboard click on updates and select the option which says reinstall WordPress.
1 A.M – 4 A.M
By the end of three hours, I have taken a backup of my database, imp WordPress files and Thesis theme. I also completely uninstalled Thesis and reinstalled WordPress, thinking that this might be end of my problems, but seems like this was never supposed to end. ( 1 A.M.)
I reinstalled Thesis, but the problem remains the same, in the meanwhile I also disabled all the plugins to make sure that no plugin is causing issue but everything was useless by that time. It was late and I thought of trying everything from scratch in the morning but to add more to my bad-luck, my site started showing Error establishing database connection and this issue is usually connected to wrong wp-config values. Though I have not touched wp-config file, but to resolve this issue I rechecked my Wp-config file and also restored the file from my backup. I also rebooted my server twice, hoping this might be temp issue but seems like I had no luck last night.
Later one of my Tech guys who manages my VPS, informed me that my error log is full and consumed almost 30+gb of disk space. And this 30Gb error log was created within 8 hrs time, so even my cron job could not empty the log file. That’s the reason why my server was giving “Error establishing Db connection” error.
4 A.M – 7 A.M
So by 4 A.m I have my site backup with default Twenty eleven Theme with no ads and no sidebar for next two hours. In between I used Clean option plugin to delete all orphaned tables and also deleted all Thesis option tables again. I started with fresh installation of Thesis theme and this time everything worked fine. I successfully installed Thesis and Thesis design option was not giving any error. It was 7 A.M by that time and I decided to take a power nap and customize Thesis theme later.
10 A.M-1 P.M
I got up at 10 with the news that my site was down from last one hour, I rebooted the server and realized I forget to activate W3 Total cache plugin and due to load server went down. Ahh!! What a mistake form me. I activated the plugin and our team started fixing theme design. Half n hour more and we are again back with same teaser option issues due to Thesis design option being unable to register the options in database.
Giving up at that point was not an option, so I wanted to try something else. Since I already worked with WordPress, database, Theme and plugin and nothing seemed to be helping me. Finally, I decided to move my blog to another hosting.
Moving to Hostgator from Linode:
( 2 P.M- 6 P.M ) So my obvious choice was Hostgator server and instead of going for VPS or dedicated server, I started with Hostgator Baby plan. As it’s good enough to handle the load for a heavy site. I bought Hostgator plan and we started the migration process. Since Hostgator allows SSH option, so transferring file was a piece of cake. Within 30 mins we were on Hostgator and then we edited our hosts file to make changes.
We installed Thesis, activated plugins and started adding codes to custom_functions and custom.css file one by one. This process also helped us to remove some unnecessary codes from our custom files. We tested and everything worked fine, we changed the name server to our Hostgator server, added MX record to make sure our Google apps is not broken and within sec. we were running successfully on Hostgator. Though if you’re not good with DIY stuff, you can always use Hostgator free hosting migration service.
It’s been 7+ hours and we are still testing load and seeing if Hostgator can handle our site load and so far everything is good.
What could have caused the issue:
Though I’m still not sure if the issue was with my configuration of VPS server or was it because I updated three plugins (Adrotate, W3total cache and Smooth slider), or some bug with database table. I would appreciate if someone wish to add his insight on possible reasons behind this outage and issue.
If you running a big website, it’s always a good idea to have a backup server and complete backup of your file. If you have a developer or Tech guy who maintains your blog, it’s always a good idea to keep track of all the changes happening on your blog.
When ever you’re making any changes in your database, take a complete backup of your database before making even a little change. Also always keep an eye on error.log to monitor possible scripts or domain, causing issues on your server.
Anyhow, it seems like an issue which can happen to anyone and luckily my team and readers were supportive enough to help me out here. In this 16 hr of outage, I got almost 10+ SMS, tons of Emails, Fb message, tweets to inform me about my site downtime and theme. I’m thankful to all who have taken the effort of sending me SMS in the night or communicate the message to me in any possible ways.
Such experiences are always good which makes you learn new things and always add as an experience. Though this issue is not same as Amit Banerjee issue, but you might like to read the similar story from him here.
Have you ever experience such terrible outage when you are running out of options? What helped you and what steps have you taken to resolve such issues.