We all already know that websites with better loading times rank higher than others.
Theme vendors are offering themes with speeds that would be quite good, especially for a news website or portal. But when we tested the theme with demo content; however, regardless of how hard we tried, speeds and scores never reached those on the vendor’s website. This led us to believe that theme vendors sometimes set up demos to make their websites appear faster than they really are.
So, the question is, do you really get what you pay for?
We set out to test this idea, using the following method to produce our loading-time comparison data:
- Test the speed of 25 themes on the vendors’ websites.
- Test the speed of the themes on a shared hosting account.
- Test the speed of the themes on a low-budget virtual private server (VPS) running Apache.
- Test the speed of the themes on a highly optimized cloud server running NGINX and Varnish and located in a top-class data center with very low latency.
After conducting all speed tests, my team compared speeds between the different platforms and also tried to find signs of “cheating.” We used online services to detect content management systems (CMS), as well as manually analyzed the source code.
We used three popular tools to test the websites: Google PageSpeed Insights (GPI), GTmetrix and Pingdom. For this report, I’ll refer mainly to Google, partly to show what it favors, although when it comes to accuracy and analysis options, GTmetrix and Pingdom are far superior.
The score shows how well your website performs:
- Green: 100 to 85
This is a very well-optimized website. Scores above 90 are very rare for websites that are not static — that is, those that use a CMS.
- Orange: 84 to 60
If a website scores at the lower end of this range, it might be on the slow side. If it falls in the upper range — say, around 73 — it’s considered to be relatively quick.
- Red: 59 to 0
This is definitely a slow or poorly optimized website. Even on a fast system or connection, users will notice this. Websites whose structure has not been thought through and media-laden news websites often fall in this range. Though rare, I have seen websites get a GPI score of less than 1 (one).
While we’re on the subject, I might as well throw in some criticism of GPI. It is very sparse in its feedback, especially for dynamic websites, which most of us use today, and sometimes the advice on optimization is borderline rubbish.
GTmetrix and Pingdom, on the other hand, are more thorough and provide much more data. Nevertheless, an excellent (speed) ranking with them doesn’t necessarily correlate with a similar score from GPI. For example, consider that GTmetrix will add points to a low score if a website uses a CDN. A website that uses a CDN might get marginally slower if a user is geographically close to the server, but for the broad majority of people, it will indeed get faster. Google does not seem to account for this.
In fact, the whole subject is a bit tricky, because many seem to treat a GPI score as a measure of speed, although I would consider it to be more a measure of optimization.
So, evidently, GTmetrix and Pingdom just look at optimization.
What’s A Good Speed Or Score?
What’s a good score, and are there any benchmarks? On my blog, I’ve tested Alexa’s top-100 shopping websites, measuring loading time and GPI score. Here are the results for the top 10 and bottom 10:
I measured the loading time of Alexa’s top-100 shopping websites, and these were the 10 quickest and 10 slowest.
The chart below shows the GPI scores for the same websites.
GPI gave these same websites these scores.
Back to the tests. All tests were conducted when we could assume that traffic to the (demo) website in question was low; if we did get abnormal results, we would crosscheck on another occasion. All of the tests were performed with the caching plugin in use on the demo website, if any. Also, note that the VPS’ used were 100% idle — they had no traffic during the tests.
Here is all of the raw data from the test.
The summary below shows the differences (not the actual GPI scores) between the three different server configurations and the reference (i.e. demo) websites.
Findings And Thoughts
Does a theme generally perform worse than the demo when you host it yourself? Do some theme vendors simply make static HTML versions of their demos to boost loading time and speed metrics?
Here are some of my key findings.
- The average score of the budget VPS and shared host was 13% and 26% lower, respectively, than the reference websites. Only the NGINX server came close to matching the reference websites, with an average 6% lower score.
- For the optimized VPS, the two websites with the highest deviations had a negative score difference of 27 and 22 points.
- For the budget VPS, the two websites with the highest deviations had a negative score difference of 35 and 20 points.
- For the shared server, the two websites with the highest deviations had a negative score difference of 33 and a whopping 53 points.
- On the optimized VPS, 19 themes (approximately 80%) had a score lower than the reference websites.
- On the budget VPS, as above, 19 themes (approximately 80%) had a score lower than the reference websites.
- On the shared host, 22 themes (approximately 88%) had a score lower than the reference websites.
As mentioned above, we wanted to check not only the speeds of the themes on different platforms, but also whether we are being misled to believe that they are quicker. In as many as 5 of the tested themes (20%), we found signs of cheating.
Based on what we detected, here are the tricks employed by these theme vendors:
- Some created a pure static website — a demo with no dynamic parts at all. This would increase the GPI score by a whole lot, and loading time would, of course, lower significantly. This is what we anticipated because we were mislead by this ourselves.
- In two cases, websites seemed to have multiple optimized files that could not be retrieved. They would block calls to the files unless the referrer showed that the request was coming from one of the demo pages. However, by faking our referrer to reflect the optimized files’ required request source, we got around the block. Yes, the files were different from what is offered out of the box.
- Some websites masquerade as WordPress by inserting comments and familiar footprints in the code. Comments might be drawn from sources such as Yoast’s SEO plugin and W3 Total Cache, and also the generator tags might be set as
wordpress. This seemed to fool some of the online CMS identifiers out there.
- A few vendor also seem to block the GPI bot, which seems kind of odd.
As you might already have noticed, we have not named names in this survey. We didn’t write this to out any company, but rather to show the importance of doing one’s due diligence when buying a theme that is marketed as being “fast loading” or “well optimized.”
This was not an “official” test either. Accordingly, we cannot conclusively say that the page-loading “cheats” we detected will continue to be present or whether they were intentional. Some cases seemed quite suspicious, though. We also tested some free, non-commercial themes, such as Frank, which follow what we would expect. This reinforces our hypothesis that some themes aren’t as fast as what is advertised on the demo websites. When running small tests on themes for other CMS’, like Joomla, we had the same findings.
SO, ARE THEME VENDORS SCAMMING US?
Whether theme vendors are scamming us depends on whether they intentionally want to mislead buyers. There is certainly a big difference between the figures claimed by vendors on their sales websites and the speeds actually experienced by buyers. Double-dealers might have a lot of leeway, in practical terms. Why? Because most buyers don’t care; they buy based merely on appearance. Other buyers are very lenient; if a website gets a GPI score of 70 or higher, then they don’t care. But for serious online publishers, a seemingly small difference in ranking could mean a difference of thousands of dollars every month.
However, keep in mind that developers in general do what they can to obfuscate the technology behind a website — that is, they hide the fact that they are using WordPress. I expect that top-class theme shops optimize just as much as we do when setting up their server. The average Joe being hosted on an oversold server would, thus, have a hard time getting anywhere close to the performance figures cited by theme vendors.
Still, this ordeal set my company back quite some time when we built our website; and, in this case, I am quite certain we were right. I sent a quite detailed email to the theme’s vendor, telling them what we had done and ending by saying that things didn’t really add up. We never heard a peep from them, but a few days later their demo website was performing a whole lot worse.
Once again, when looking for a theme, do a fair amount of research. Look through the review websites, look for websites with the same theme, and compare.