When I shifted to WpEngine hosting few months back, I shared the complete guide on moving and also added why I’m loving their hosting. If you have missed all these details, here are few things which you should know:
Before moving to WpEngine, I was hosted on a private VPS and one of the freelancer networking guy was managing my server. The performance was good but there was few issues related to downtime and site slowness. More over, one biggest problem with freelancer or less known hosting company is the accountability. So I decided to spend $250/month on WPEngine hosting, and moved ShoutMeLoud to their server. Without any second thought, WPEngine delivers what they promised, and so far I’m in love with the quality of hosting and website performance. Though $250/month is indeed a high-price, and any knowledgable guy would recommend to go for a VPS or dedicated server. In my case, I decided to stick to WPEngine, as I don’t want to see even a second downtime for my main blog, and when quality is everything, price is the least of concerns.
So What’s the problem with WPEngine?
Two of the biggest complain you will hear about WPEngine is Pricing and customer support. In past 2 months, They have significantly improved on their customer support by adding live-chat, but the pricing issue remain the same. After using WPEngine for more than 2 months, I encountered one of the biggest pricing issue with WPEngine. This post is about the problem and the possible solution to the problem.
First of all, WPEngine pricing is based on the visits and the way they count the visits is different from what you see on your Google analytics or WordPress.com stats program. According to ShoutMeLoud Google analytic stats, I get close to 325,000 visits a month, and I opted for WPEngine Business plan which allows up to 400,000 visits a month, and is priced at $250.
Last time when I logged into my WPEngine dashboard, I was surprised to see an extra $188 bill for overage visits, and for this month, I was about to cross the threshold of 400,000 visits. I was quite surprised as, my WordPress.com stats and Google analytics was showing lower than 400,000 visits. Before I raise my concern to their team, I again had a look at their help page related to visits, and I realised I missed understanding few important points. Here is what their visits help page has to say:
- When a human being first arrives on the site and loads the page, staying there for 31 seconds, that’s a visit.
- If that same human then clicks a link and sees another page, that’s not anew visit; that’s part of the same visit.
- If that same human loads the site with a different browsers, that’s still not anew visit; that’s part of the same visit.
- If that same human bookmarks the site, then 11 days later comes back to the site, that is a new visit.
- When a robot loads the site (like a Google or Bing search bot), that’s a visit, but if one robot scans 100 pages quickly, that’s one visit. (You might disagree that a robot is a “visit,” but consider that from a hosting perspective, we still had to process and serve all those pages just like it was a human being, so from a cost or scaling perspective, bots count the same as humans.)
- If a robot scans 20,000 pages over the course of a month, that’s not just one visit. It shouldn’t be 20,000 visits, but neither should it be 1. Something in the range of 100-1,000 visits is acceptable.
- There are additional cases too where the “right thing to do” is less clear. For example, take the case of a “quick bounce.” Suppose a human clicks a link to the site, then before the site has a chance to load the human clicks “back.” Does that count as a visit? Our servers still had to render and attempt to return the page, so in that sense “yes.” But a human didn’t see the site and Google Analytics isn’t going to see that hit, so in that sense “no.” Because we need the notion of a “visit” to correspond to “the amount of computing resources required to serve traffic,” we round off in favor of saying “yes.”
Two important points that I missed reading on the same page was:
- We reset our notion of “unique IP address” every day.
- Robots have the same few IP addresses, so they will be consolidated within one day, but will count again the next day.
Though, everything was well-written but I missed reading the point about resetting the unique I.P. Address every day, and even bot visits are counted as new visits after every 24 hours. ShoutMeLoud is one active blog, and not only Google bots, but there are virtually many Good + bad bots which crawl ShoutMeLoud daily, and I realised that added a lot to my number of visits according to WPEngine.
First of all, this notion of counting visits from WPEngine is not only flawed, but it definitely adds a big hole in the pocked of any existing WPEngine users. As I was considering to move away from WPEngine hosting after knowing how they are counting visits, and certainly this is not a customer-friendly model of pricing, I wanted to see if I could do something to make my WPEngine stats closer to my Google analytic stats. I asked them few questions, which will help any existing WPEngine user or anyone who is planning to purchase hosting from WPEngine:
Q) Image hotlinking: So if an image is hot linked somewhere, and it’s access by a person on site x (which hot linked my images), will that be counted as a visit?
Ans.)Yes, that counts as a visit. Hotlinking can be blocked, but CloudFlare is a simpler solution, with advantages because it works on the DNS level.
Q) If an image which is hosted on my server and is opened anywhere will that be counted as visits?
Ans.) Right, counts as a visit.
Q) CDN: If both the above answers are yes, and if I serve my images using CDN powered by WPEngine will that be counted as visits?
Ans.) Any request that hits CDN instead of the server does not count as a visit. So CDN is a helpful tool as well. But if these hotlinks pre-exist the enabling of CDN, it’s a tougher issue to address. Again, because CloudFlare is at the DNS level, it’s a simple, comprehensive solution.
Q) I see that WpEngine counts bad bots as visits too, do you have a solution for blocking these bad bots/spam/scrapers bots?
Ans.) CloudFlare allows you settings to fight abusive bots. But your bot traffic looks healthy and legit; although you did have over 5k visits from the bot for the popular Chinese search engine “Baidu.”
Q) Why does Google Analytics show a much lower visitor number?
A unique IP is counted as unique in an entire month, while our system looks at unique IPs on a per day basis. They also will not include Google bots or various other bots in their visitors count. GA is used for marketing/advertising purposes and that is why it works so hard to only count what they believe to be human visitors. Our server stats monitoring tracks all unique visitors because from a server impact standpoint, a bot or a human is still pulling data down from the server and using CPU resources in the process.
Q) Is there a problem or something wrong with the GA implementation?
Ans.) Absolutely not. They have a specific purpose, to try and show you how many human visitors have come to your site. Our own stats has a specific purpose to show from a server impact perspective how much data is being transferred over our servers
Q) Also wondering if you could look into my above question related to WordPress.com stats program? Is it also flawed? or they work different than yours?
Ans.) I wouldn’t say those stats are flawed. It’s just a tool that’s designed more for advertising purposes, for example to measure “reach”. A good way to get stats that accurately reflect the traffic that actually hits the server is to see the stats (like “Daily Usage CSV”) available in your User Portal.
How to lower down WpEngine Bills?
First of all, despite of liking everything about WpEngine, I hate the way they are over-charging their customers. I always considered WPEngine to be a customer-centric company, and they have implemented many great features like Website security, free DNS, Website backup, Staging area, and many more to give important features to their customers. But, sadly they have not implemented any measures to block the bad bots from accessing your website. None the less, they are charging their customers for such visits, which doesn’t even matter.
WPEngine team, if you are listening to this, I would love to see you implementing such features in your future iteration.
Anyways, to sum everything in simple words:
- WPEngine charges for bad bots visits.
- They reset their visit counters every 24 hours for human and for bots.
- They will charge you for hot linked images or images visits from Google image search.
- They don’t have any server level implementation for blocking bad bots visits
Now, after going through all this; I had a choice to move away from WPEngine, or keep paying the price. But, there is no fun leaving something which is good, and without making an extra effort. So first, I tried finding few .htaccess code to block bad bots visits. Alas! I couldn’t find any updated list of bad bots, so .htaccess was not a viable option. For now, I have implemented Cloudflare CDN, and will see how WpEngine stats shows the difference. If it all works out, I will be happy to stick with them, else there is no thought on saying Goodbye to WPEngine. So, if you are an existing WPEngine customer, I highly recommend you to implement cloud flare CDN which is free (Also suggested by WPEngine staff), and it will help you to lower down your hosting bill.
- Also read: How to prevent Image hot linking from specific domains using .htaccess
- Use WP-Picshield plugin to prevent Image hot linking
Is WPEngine a good hosting company or bad?
There is a very thin line between deciding anything as bad or good. My part of job is to share my experience, and in terms of hosting quality WpEngine is par excellence, but in terms of pricing, I don’t find it reasonable due to over-pricing. According to my opinion, any busy site will usually face the issue of high-bill on WPEngine hosting, as busy websites usually gets more bot visits, more hot linked images due to copy-paste posts. Undoubtedly, it might not be a very cost-effective solution for a busy website, where as a small WordPress website will not face such issue.
It’s important for any existing WPEngine users to know how they are charging, and count visits. More over, you should definitely implement measures like Cloudflare CDN or any custom solution to block bots visits to keep your limited hosting bill in check. If you are an existing WPEngine customer, I would love to hear your review and opinion regarding their pricing plans and if you have faced any issue with their over-charging, I would love to hear your story.
Get Free Blogging updates in your Email