Why robots can ruin your email split testing results - Smart Insights Digital Marketing Advice

A reminder to dive deeper into your email testing results

Split testing of email campaigns is a great way to learn and improve your results. Testing works by changing particular campaign elements, sending both the orignal (control) and new version (treatment) and then measuring the difference in results.

However test results can be ruined when there are additional factors that impact one test cell’s results but not the other test cells. When this happens you can pick the wrong winner and end up decreasing campaign performance and revenue.

I was recently running a test and was hit by an external factor that without correction would have meant wrong conclusions were reached.

When diving into the results of one test cell I observed that one email address had clicked five times on every single link in the email. Upon investigation it turned out these clicks were not clicks from a human but clicks by a corporate spam filter. The corporate spam filter was automatically following every link on the email and thus causing a link click to be counted on campaign reports.

Spam filter robots do this to check that the links do not lead to a virus or a site with inappropriate content. Such spam filter automated link clicking is not common and is mostly used by corporate IT, so will affect B2B campaigns more than B2C.

Under normal circumstances a few extra robot clicks would not be an issue to reviewing the success of your campaign. However, by design split test cells are small, so the activity of one address can have a much greater relative impact.

In this case the one email addresses added five clicks to each of the 20 links, giving this one test cell a total of 100 additional clicks. This was enough to skew the result and make this test cell look like a winner. Once spotted the answer was easy, just remove clicks from this particular address before judging the winner.

The take home is always look at all metrics and reports when testing. Review the detail as well as the top line and look for any unexpected patterns that could be due to an external test factor.

http://blog.marketingxd.com/ Pete Austin @marketingXD

The googlebot can also be a big problem when counting email clicks. I’ve seen literally thousands of clicks caused by it, when customers copy a preview or “display in browser” link from an email and paste it into a corporate website. The joke was that we were being DDoS’d by Google.

Two possible solutions to these clicks, and your problem above, are:
* to look at “unique clicks”, not “total clicks”, wherever possible.
* look at the user agent value and exclude known bots

http://www.smartinsights.com/about-dave-chaffey/ Dave Chaffey

Thanks for the Googlebot story and suggestion Pete – we like guys who care about the details.

It makes me wonder whether some lower volume e.g. B2B campaigns (never mind testing) will be skewed by this effect.

http://craig.sullivan@belron.com craig

How big are your test cells? If they are that small, to be affected by this issue, then they are probably worthless for testing email efficacy too.

If you’re tracking conversions and identifying visitors correctly, the duplicate clicks shouldn’t be an issue – only if you are counting clicks, which seems like the wrong metric anyway.

http://twitter.com/tawatson Tim Watson

Hi Craig, thanks for your comment. Glad to see you are thinking about test cell size, since this factor is so often not considered and typcially test cells are used which are too small.

I always apply complete rigour to statistical signficance of results and test cell size (using t-Test and assumption of bernoulli distribution).

To get a feel before running calculations the rule of thumb for significance is a difference of 50 responses. So if comparing clicks, the difference between the two cells must be 50 or more clicks. The robot itself caused 100, enough to skew the result.

In this test I wanted to measure on a conversion event on the website, as you suggest, but the client was not able to get this implemented in time. The aim should be to measure as close as possible to the marketing objective, I totally agree with you.

The clicks were used here as the best proxy to the marketing objective that was set.

I will be doing a future post on cell size, significance and the factors to be considered in ensuring reliability of results.

http://www.emailcenteruk.com Sean Duffy

I agree with the comment that if your test is being affected by this then your sample size is too small to be reliable anyway.

More important is unsubscribe links, and those wonderful one-click and you are suppressed forever. Bots/Filters can ‘click’ on these and therefore unsubscribe that recipient.

That is more worrying than split test issues.

Simple solution is to use a ‘Are you sure you want to unsubscribe?’ page

http://twitter.com/tawatson Tim Watson

Hi Sean, great point on the one-click unsubscribe. Yet another reason not to use a one-click process. I never advise this and it wasn’t used on this test campaign thankfully.

As you say much better to use a confirm page where you can also offer alternatives, such as opt-down, preference change, switch channel or just get feedback.

I’m interested to know what size test cells do you normally use and advise?

http://Pure360.com Michael Bairstow

Great post Tim, intrigued though how did your checks reveal the presence of these robotic spam filters? Can you suggest where I could go to back this up?

http://twitter.com/tawatson Tim Watson

Hello Michael, I’ve seen the robot pattern a few times. The tell tales are; many clicks in a period shorter than a human would do, same IP address, same tracked email address and unrealistic patterns. In this case exactly five clicks on every link, including privacy policy, unsubscribe, show in browser and all six share to social links.

One of the points of the post though is in regard of testing. Always be aware of internal and external factors that can upset test results. Look deeper into the metrics and not just at the top line.

http://getintheinbox.com/2011/06/01/email-marketing-deliverability-i-want-to-build-a-community-around-one-email/ Andy T

From working with Michael (comment above) we’ve also see similar spikes in opens from the tracking gif. After reading this article we’re starting to think it might not be inaccurate to speculate that filters are also following image paths.

Funny about the one click optout being activated by the filters: various legal consultants and best practice documents go on about how everyone should do it and then the technology we’re trying to work with to increase the recipient experience, contradicts it.
It’s one of those times when you either sign and get on with it or try to fight off the hysteria as you laugh it off
Pingback: Double opt-in – why and when? [1/2] | E-Mail Marketing Tipps

A reminder to dive deeper into your email testing results

Related posts

Advice A-Z