Are You Getting Cheated When Buying A WordPress Theme?

The theme we started out with seemed very good, offering speeds that were quite good, especially for a news website or portal. We tested the theme with demo content; however, regardless of how hard we tried, speeds and scores never reached those on the vendor’s website. This led us to believe that theme vendors sometimes set up demos to make their websites appear faster than they really are.

So, the question is, when we shop for a theme, do we get what’s on the tin?

Test Method

We set out to test this idea, using the following method to produce our loading-time comparison data:

• Test the speed of 25 themes on the vendors’ websites.
• Test the speed of the themes on a shared hosting account.
• Test the speed of the themes on a low-budget virtual private server (VPS) running Apache.
• Test the speed of the themes on a highly optimized cloud server running NGINX and Varnish and located in a top-class data center with very low latency.

After conducting all speed tests, my team compared speeds between the different platforms and also tried to find signs of “cheating.” We used online services to detect content management systems (CMS), as well as manually analyzed the source code.

Tools Used

We used three popular tools to test the websites: Google PageSpeed Insights (GPI), GTmetrix and Pingdom. For this report, I’ll refer mainly to Google, partly to show what it favors, although when it comes to accuracy and analysis options, GTmetrix and Pingdom are far superior.

GPI will survey a website and then give it a score between 1 and 100. The score depends not only on speed, but on quality of code — minimization of HTML, CSS and JavaScript, caching, compression and so on.

The score shows how well your website performs:

• Green: 100 to 85 This is a very well-optimized website. Scores above 90 are very rare for websites that are not static — that is, those that use a CMS.
• Orange: 84 to 60 If a website scores at the lower end of this range, it might be on the slow side. If it falls in the upper range — say, around 73 — it’s considered to be relatively quick.
• Red: 59 to 0 This is definitely a slow or poorly optimized website. Even on a fast system or connection, users will notice this. Websites whose structure has not been thought through and media-laden news websites often fall in this range. Though rare, I have seen websites get a GPI score of less than 1 (one).

While we’re on the subject, I might as well throw in some criticism of GPI. It is very sparse in its feedback, especially for dynamic websites, which most of us use today, and sometimes the advice on optimization is borderline rubbish.

GTmetrix and Pingdom, on the other hand, are more thorough and provide much more data. Nevertheless, an excellent (speed) ranking with them doesn’t necessarily correlate with a similar score from GPI. For example, consider that GTmetrix will add points to a low score if a website uses a CDN. A website that uses a CDN might get marginally slower if a user is geographically close to the server, but for the broad majority of people, it will indeed get faster. Google does not seem to account for this.

In fact, the whole subject is a bit tricky, because many seem to treat a GPI score as a measure of speed, although I would consider it to be more a measure of optimization. I have written about this before, and below are some examples showing how speed and page score don’t necessarily go hand in hand.

GTmetrix Example

Pingdom Example

So, evidently, GTmetrix and Pingdom just look at optimization.

What’s A Good Speed Or Score?

What’s a good score, and are there any benchmarks? On my blog, I’ve tested Alexa’s top-100 shopping websites, measuring loading time and GPI score. Here are the results for the top 10 and bottom 10:

The chart below shows the GPI scores for the same websites.

Assessment of the results is quite subjective, but it’s safe to say that anything above 75 can be considered good, and anything below approximately 40 is very bad. However, it very much depends on the category of your website. It’s a sure bet that a personal blog will have a better score than a big portal like the Huffington Post.

Benchmarking

Back to the tests. All tests were conducted when we could assume that traffic to the (demo) website in question was low; if we did get abnormal results, we would crosscheck on another occasion. All of the tests were performed with the caching plugin in use on the demo website, if any. Also, note that the VPS’ used were 100% idle — they had no traffic during the tests.

Here is all of the raw data from the test.

The summary below shows the differences (not the actual GPI scores) between the three different server configurations and the reference (i.e. demo) websites.

Findings And Thoughts

Does a theme generally perform worse than the demo when you host it yourself? Do some theme vendors simply make static HTML versions of their demos to boost loading time and speed metrics?

Here are some of my key findings.

• The average score of the budget VPS and shared host was 13% and 26% lower, respectively, than the reference websites. Only the NGINX server came close to matching the reference websites, with an average 6% lower score.
• For the optimized VPS, the two websites with the highest deviations had a negative score difference of 27 and 22 points.
• For the budget VPS, the two websites with the highest deviations had a negative score difference of 35 and 20 points.
• For the shared server, the two websites with the highest deviations had a negative score difference of 33 and a whopping 53 points.
• On the optimized VPS, 19 themes (approximately 80%) had a score lower than the reference websites.
• On the budget VPS, as above, 19 themes (approximately 80%) had a score lower than the reference websites.
• On the shared host, 22 themes (approximately 88%) had a score lower than the reference websites.

Tricks Employed

As mentioned above, we wanted to check not only the speeds of the themes on different platforms, but also whether we are being misled to believe that they are quicker. In as many as 5 of the tested themes (20%), we found signs of cheating.

Based on what we detected, here are the tricks employed by these theme vendors:

• Some created a pure static website — a demo with no dynamic parts at all. This would increase the GPI score by a whole lot, and loading time would, of course, lower significantly. This is what we anticipated because we were mislead by this ourselves.
• Some run local optimization of JavaScript, CSS, HTML, etc. This means that the theme delivered to the client is different than the one used on the demo website. I wouldn’t consider this to be cheating, however, and so didn’t count it as such.
• In two cases, websites seemed to have multiple optimized files that could not be retrieved. They would block calls to the files unless the referrer showed that the request was coming from one of the demo pages. However, by faking our referrer to reflect the optimized files’ required request source, we got around the block. Yes, the files were different from what is offered out of the box.
• Some websites masquerade as WordPress by inserting comments and familiar footprints in the code. Comments might be drawn from sources such as Yoast’s SEO plugin and W3 Total Cache, and also the generator tags might be set as wordpress. This seemed to fool some of the online CMS identifiers out there.
• A few vendor also seem to block the GPI bot, which seems kind of odd.

Summary

As you might already have noticed, we have not named names in this survey. We didn’t write this to out any company, but rather to show the importance of doing one’s due diligence when buying a theme that is marketed as being “fast loading” or “well optimized.”

This was not an “official” test either. Accordingly, we cannot conclusively say that the page-loading “cheats” we detected will continue to be present or whether they were intentional. Some cases seemed quite suspicious, though. We also tested some free, non-commercial themes, such as Frank, which follow what we would expect. This reinforces our hypothesis that some themes aren’t as fast as what is advertised on the demo websites. When running small tests on themes for other CMS’, like Joomla, we had the same findings.

So, Are Theme Vendors Scamming Us?

Whether theme vendors are scamming us depends on whether they intentionally want to mislead buyers. There is certainly a big difference between the figures claimed by vendors on their sales websites and the speeds actually experienced by buyers. Double-dealers might have a lot of leeway, in practical terms. Why? Because most buyers don’t care; they buy based merely on appearance. Other buyers are very lenient; if a website gets a GPI score of 70 or higher, then they don’t care. But for serious online publishers, a seemingly small difference in ranking could mean a difference of thousands of dollars every month.

However, keep in mind that developers in general do what they can to obfuscate the technology behind a website — that is, they hide the fact that they are using WordPress. I expect that top-class theme shops optimize just as much as we do when setting up their server. The average Joe being hosted on an oversold server would, thus, have a hard time getting anywhere close to the performance figures cited by theme vendors.

Still, this ordeal set my company back quite some time when we built our website; and, in this case, I am quite certain we were right. I sent a quite detailed email to the theme’s vendor, telling them what we had done and ending by saying that things didn’t really add up. We never heard a peep from them, but a few days later their demo website was performing a whole lot worse.

Once again, when looking for a theme, do a fair amount of research. Look through the review websites, look for websites with the same theme, and compare.

I hope you like this article. If you would like to see more detailed data and to download my layman’s infographic, you may do so on my website. And you’ll find two excellent articles by Marcus Taylor here on Smashing Magazine: “What to Consider When Choosing a WordPress Theme” and “How to Speed Up Your WordPress Website.”

TL;DR

• Website speed is becoming increasingly important for SEO and usability.
• We tested 25 WordPress themes on three different server configurations not only to compare scores, but to find signs of theme vendors cheating to inflate their score.
• Of the 25 themes tested, five vendors seemed suspicious at best.
• On a budget shared hosting place, 88% of the themes got lower scores than their demos.