Email open rate, as useful as your appendix

5 cases-in-point show why you need other metrics to review email marketing effectiveness

There have been a couple of heated debates I’ve been part of in the email community just recently, one about open rates and another about subject line length. In this post I’m reflecting on the open rate debate. I’ve added an update to the original post showing more proof that open rate fixation can be misleading.

It may surprise you that the community is very split on the value of the open rate metric, seemingly, a cornerstone metric since the dawn of email marketing. It’s a metric cited in every email marketing book and on every mail marketing course… So why is its value in doubt? Ken Magill reported on some of the open rate debate.

Let me summarise some of the reasons why open rates are not important.

  • Opens are almost never the objective of a campaign. Possible exceptions might be in publishing and list hygiene, when identifying active addresses to solve deliverability problems.
  • Higher open rates do not always mean higher click and conversion rates. Open and click/conversion can and do go in opposite directions. Why bother with open rate when you can focus on click or conversion?
  • Open rates can be misleading. Through focusing on driving higher open rates this can potentially leave you with less business.

Update: 13th November 2012

I’ve recently published some research for emailvision reviewing whether open rates are a good predictor of success in email marketing across 50 million retail emails from 196 campaigns.  The headline finding is…

Open rates wrongly predict success 53% of the time

As you might expect, there is a much closer correlation between clickthrough rate and open rate. This graph from the research shows that the open rate is relatively unchanging; it typically varies by just 12%. So if the open rate average is 20% that means it varies up to 22.4%. However conversion rates vary hugely in comparison, by some 70%. If average conversion rate is 0.2% then it varies up to 0.34%. To understand campaign performance the conversion rate is a must.

Why can a higher open rate mislead and deliver less business? Take an example, I talked to someone recently who was delighted they had increased their open and click rates. I asked how. The answer was they removed people who hadn’t opened recently. I asked if they had conversion and revenue figures to compare. I wasn’t surprised when I didn’t get an answer…

The five different cases below illustrate many of the problems with open rates.

Case 1 – List hygiene and inactives

List size 10,000
Number of people who open 2,000
Open rate 20%

A common percentage of inactives is 40% of the base. Let’s remove those 4000 people from campaigns.

List size 6,000
Number of people who open 2,000
Open rate 33%

That’s a 50%+ increase in open rate. Will it have resulted in 50% more revenue? Absolutely not. In fact most likely the revenue will decrease, because every month a percentage of the apparently inactive customers do come back again.

The open rate metric and increase in this case does not represent business success.

Case 2 – Monthly reporting and averages

Campaign open rates are sometimes averaged over the month for reporting. For example these three campaigns:

Campaign A  60%
Campaign B  20%
Campaign C  10%

This gives the monthly open rate, calculated as the average of open rates, of 30% = (60+20+10)/3.

Let’s add real numbers to the above figures

Delivered Opened Open rate
Campaign A 1000 600 60%
Campaign B 2000 400 20%
Campaign C 6000 600 10%

 

Total emails sent 10,000 with total opens 1600, giving an open rate over all campaigns of 16%. The commonly used average of opens, at 30%, is almost double the real value.

Case 3 – Same rates, but one campaign is better

Two campaigns with identical open rates. Which campaign was more successful? Ok, so you’re getting wise and want to know the send size before you decide? That’s fair, here are more campaign metrics.

Delivered Open rate Click rate Conversion rate
Campaign X 10000 15% 5% 0.25%
Campaign Y 10000 15% 5% 0.25%

 

They still look the same. However, campaign Y is more successful. Because the average order value for campaign Y was double that of Campaign X.

Case 4 – The impact of frequency variation and on open and click rate

Customer segment A receives a campaign with open rate 20% and click rate 5%.

Customer segment B receive two campaigns in the same period and because of email increased marketing pressure the open rate drops to 15% and click rate to 4%.

Which was more successful?

If the list size is 10,000, then the number of opens for segment A was 2000 and for segment B was 3000 across the two campaigns. Making campaigns to segment B more successful, although the campaign open rate is lower.

Case 5 – Subject line A/B split tests

Surely the open rate is a useful measure of the subject line quality?

Well, not really here either. Take the case where one subject line gets lower opens and higher clicks than the other. The better subject line is the one with higher clicks and lower open rate. Fewer people may have opened but the ones that did open were the right ones, the right target for the offer and call to action.

It would be even better to evaluate the subject line on conversions, however test cell sample sizes don’t normally permit that.

If not open rate then what?

To sum it up, the better metrics to focus on are:

  • Total revenue from email
  • Revenue per 1000 emails sent (customers) per campaign
  • Email revenue per customer per month
  • Average customer order value
  • Conversion rate
  • List size and growth
  • Percentage of email base purchased in the last month
  • Percentage of email base clicked/opened in the last month

Notice how these metrics consider money and customers not campaigns. Measure campaigns to understand what campaigning strategies work and measure customers too, as customers are more fundamental to business health.

What other metrics, whether new ones or old ones should have time called on them?

Next time around I’m going to look at the debate around subject line length and explain just what the right length is for a subject line.

  • http://twitter.com/SVJennings Sharon Jennings

    Let’s not forget that opens themselves just aren’t accurate. It’s based on images downloading, so you can open an email, but by not downloading images it’s not counted. And conversely, you can NOT open an email, but in the preview pane the images download and it IS counted as an open. But great article, been telling people open rates are useless for ages!

    • http://www.smartinsights.com/ Dave Chaffey

      Thanks Sharon, yes another weakness of opens… keep up the evangelising about what people should be measuring!
      Dave

      • Shivachi

        Hi Dave,
        we send out a daily newsletter that has on average 6 products laid out on a single page. We then proceed to track the click rate for each product and accumulate this data over time.
        However, we’ve noted something curious.

        There seems to be no correlation between the popularity of a product as indicated by the clicks it receives and the revenue the product brings in or the number of buyers it attracts.
        We’ve had some products perform dismally in terms of clicks yet attract a record number of buyers and vice versa.

        My two questions are:
        1. Is it wise to keep collecting this data?
        2. How can we use this click rate data more meaningfully?

        • http://www.smartinsights.com/ Dave Chaffey

          Hi Sivachi,

          That doesn’t sound right – are you tracking the clicks using an email package or using Google Analytics or similar. I recommend tracking the links in the enewsletter as explained here: http://www.smartinsights.com/email-marketing/email-marketing-analytics/email-campaign-tracking-with-google-analytics/

          You can then setup an Advanced Segment in Google Analytics so that you are only reviewing traffic from email and no other channels, so should be more accurate.

          Does that help?

          Dave

          • Shivachi

            I think that does help.
            I will proceed as you advise and then track the performance and see.

            Thanks Dave.

  • Guest

    Let’s not forget that Open Rates just aren’t accurate. As they rely on images downloading, you can have false positives and false negatives.

  • Pingback: Email open rate, as useful as your appendix – Smart Insights Digital … | Internet Marketing News | Conversion Assistants()

  • http://twitter.com/SparkstoneWeb SparkstoneTechnology

    We vary rarely measure open rates as our clients want us to focus almost exclusively on the ROI for the cost of building and sending the email.

    So instead we focus on conversion* rates from visitors arriving at a website via a link within an email. It’s important to restrict these analytics figures to within 5-7 days after the email delivery, as customers could visit on more than one occasion after opening the email and any cookies stored by their browser could skew the results and make the email look mare successful than it actually was.

    *extra note – we find that e-commerce conversions from our emails are usually 50-60% higher than the standard conversion rate for any particular website… so email marketing is still by far the most cost effective and trackable way of marketing.

    • http://twitter.com/tawatson Tim Watson

      Spot on, thanks for adding the comment, great to hear your clients are enlightened.

  • Pingback: Daily Delivery – Better Than Open Rates | Deliverability.com()

  • http://twitter.com/philipstorey Philip Storey

    Any metric that is analysed in isolation, is only going to deceive you. The fallacy of one number, and all that… http://thejonathanmacdonald.blogspot.co.uk/2010/11/fallacy-of-one-number.html

  • http://www.emailvendorselection.com Jordie van Rijn

    I wrote an article on the problems with average statistics a while back
    http://www.emailmonday.com/dont-settle-for-average-email-marketing-statistics

    Its not only the averages over time which cause problems, but also the averages over groups. The best stats tools let you analyse data after the send so you can identify behaviour of select groups and make it actionable.

    • http://www.smartinsights.com/ Dave Chaffey

      Hi Jordie,

      Yes that’s a good point and it shows where email open rates (and other KPIs) are useful – e.g. across webmail readers to show problems in deliverability or rendering or response rates across segment to show how your email is being responded to by different groups.
      I think a lot of email systems aren’t good at encouraging review of segmented results. Anyone say which are the best = easiest to prompt and visualise this?
      Dave

  • http://twitter.com/PeterMasseyBudd Peter Massey

    Hi Dave – have you blogged anything on “leaky buckets”? – An infographic to see % leakage & value leakage at every step from approach through click through conversion to conversation to converted sale to activated customer in one view. V powerful and gives marketeers ability to learn to match the level of effort at each stage to max thoughput all the way to revenue. And to do that with their sales colleagues as its one stream

  • John Bollinger

    One problem with your hypothesis in Case 1 is that inbox delivery rate is not considered when discussing list hygiene based on inactives. While eliminating inactives may cause missing out on possible sales from a “percentage of the apparently inactive customers” that come back each month, not eliminating inactive addresses has a far greater chance of negatively affecting a sender’s reputation by increasing invalids and complaint rates while diluting the open and click rates that are critical in maintaining a good reputation and good inbox delivery rates. Lost revenue from junk folder placement is much greater than gains from a few inactive customers coming back.

    • http://twitter.com/tawatson Tim Watson

      Hi John, I agree this is a possibility and inbox placement needs to be checked, as if that is indeed negatively affected by a high level of long term inactives then removing inactives is productive. For some clients have I removed deep inactive when this is exactly the case.

      I didn’t make this clear enough perhaps in the post though I did point out the possible use of open rate “when identifying active addresses to solve deliverability problems”.

      So to fully qualify I should say; don’t remove inactives unless you have inbox placement problem to resolve.

      Thank you for picking this up and highlighting this.

  • Pingback: Reporting and Metrics – the open rate debate | DMA Email … | Email Marketing()

  • http://twitter.com/tawatson Tim Watson

    Related to this post about open rates I’ve posted an analysis of open rates against conversion rates here http://blog.emailvision.com/eng/open-rates-wrongly-predict-success-53-time

  • Pingback: Latest marketing developments – November 2012 | IDM Marketing Blog()

  • Pingback: 4 indicatori care te ajuta la optimizarea newsletterului - Conectoo | Conectoo()

  • Pingback: Digital Marketing News – Latest Update 4 | Target Internet()

  • Pingback: Digital Marketing News – Latest Update 5 | Target Internet()

  • Pingback: Ecommerce Links: October 2012 « MindCorp | Newsfeed()

  • Pingback: Email Marketing – How Much is Too Much?()

Feedback Form
Feedback Form