Email Split Testing: As Simple as A & B

Three great little email split testing stories…
Picture the scene: Your cart abandonment campaign is up and running and the SaleCycle elves are busy at work retargeting your customers and recovering those pesky abandoned carts…
But what’s next? How about a bit of optimisation to help you recover even more sales!
A/B testing is a relatively simple way to grow and optimise your campaign, whether you’re looking to improve open rates or simply change your tone of voice, testing is a great way to ensure your campaign is performing to it’s full potential!
Three SaleCycle clients have kindly allowed us to share their split testing stories, describing the testing carried out and the all important findings. Here’s the tale of three tests…

---

1 Idle Times

Getting the right timing for your email is crucial, not only for grabbing a customer’s attention while purchasing is still at the forefront of their mind but also to ensure your emails get the strongest open, click through and conversion rates.
There are many variables to take into consideration when testing delivery times, but you can focus on two:
1) Consider the competition – If you know your average customer is likely to have abandoned in order to shop around/compare prices then you may want to reach out to them as quickly as possible – ideally within 20/30 minutes
2) Consider the price point – If you are selling products which generally have a very high order value then customers may be taking a little longer to think things through and you may wish to retarget them between 2 to 4 hours rather than straight away.

---

Cath Kidston’s Story

Fashion and home retailer Cath Kidston wanted to test a 1 day idle time against shorter timescales to see how their customer base behaved and interactive with their abandonment emails.
Cath Kidston tested sending their emails at 24 hours, 12 hours, 6 hours, 4 hours and 2 hours, and here’s how the results shaped up:

The results demonstrated a pretty clear uplift in Open Rates as the idle times got shorter as it was closer to the time of abandonment when the customer was still within their shopping mindset, and while it was interesting that the open rate did drop a little towards the 4 and 2 hour marks the conversion rates were considerably higher at this point.
The difference between the default of 24 hours and 12 hour test was striking enough with an 11% uplift in the open rate and a 45% increase in the open from send. However the shortest idle time of 2 hours was the clear winner overall with a 13.7% increase in the open rate and a whopping 38% increase in the conversion from click.
Based on the strength of the conversion rates, Cath Kidston have set the idle time for the campaign at 2 hours and continued to achieve steady and impressive results.

---

2 Subject Lines

As well as the timing of your emails, the subject line is a key component in ensuring you have the best chance of making a connection with your customer. It can make or break how they perceive the content of the email and their subsequent interaction and it’s vital to be clear about you’re hoping to achieve – there are two golden rules we always recommend at SaleCycle
1) Get to the point – keep your subject line short and simple where applicable, think of how it will appear within someone’s inbox where space is limited as well as making a quick first impression. The general rule is to try and keep your subject line to a maximum of 50 characters.
2) Be Open & Honest – You may find during split testing that often the subject line with the highest open rate has the lowest click through rate – this is a sure sign that the customer’s expectation of the email content based on the subject line doesn’t match the reality which can then affect the click through and conversion rates. There’s no point promising something with an awesome subject line that isn’t delivered in the email!

---

Monsoon’s Story

Fashion retailer Monsoon ran a series of A/B tests to optimise the subject line performance on their campaign. They were looking for the impact different subject lines could have, not only on open rates but also what customers did once they opened the email – here’s what they tried:

“Your Monsoon Shopping Basket”

“Your [item] from Monsoon”

“Can we help?”

So, how did the results stack up? Which one is your money on?
Well, pulling through the name of an item within their abandoned basket had a really positive impact with customers – a 16% increase in the open rate with a whopping 20% increase in the click through rate, and most importantly a 50% increase in the conversion from send rate.
By contrast the customer service focused “Can we help?” had a negative effect on performance, despite seeing a 5% increase in the open rate we then saw a 2% drop in click through rates and a 1.5% drop in the conversion from send – and all this using the same creatives for all variations!

No prizes for guessing which subject line Monsoon chose to continue with going forward…!

---

3 The Creative

So you’ve hit your customers with an email at the best possible time and the subject line is going down a storm, but how do you ensure that you get that all-important click to bring them back to the site to complete their order?
Testing the layout, look and feel of your email can help improve both your click through and conversion rates, here are a few things to consider testing:
– How long is your opening text? Does it get straight to the point or does a more lengthy in-depth engaging copy work better with your target audience
– How personalised are the emails? Does the inclusion of a customer’s name within the text improve their engagement?
– How high is the basket within the email? If it falls below the fold it may be worth testing the layout to try and improve interaction with the basket
– Text links vs. Buttons/Images – if a customer has their emails set to automatically block images is it still clear what the email is about? It may be worth testing a hyperlink within the opening text to see if this helps improve the number of clicks. Similarly you may wish to test replacing any call to action buttons with text cells to test interaction.
– Button layout – do your call to action buttons work better near the top of the email or next to the basket? Do they work better on the left hand side or on the right?

---

Tesco Broadband’s Story

Tesco Broadband wanted to test the layout of their emails to see how a shopping cart focussed email performed against customer service content promoting their USPs.
The SaleCycle design wizz-kids produced a slightly different template to include 5 bullet points highlighting the benefits of switching to Tesco Broadband. The assumption was that a softer, customer service lead approach would work well given the nature of the service the customer was looking to sign up for but the test results were surprising.
Interestingly the open rates and the click through rates were higher on the USP template but the conversion from send dropped by 14% with the conversion from click dropping even further by 16% when compared to the standard short text with shopping cart format.
Testing on this is on-going with the ultimate aim perhaps being a combination of the two styles to compliment both Tesco’s great products and the USPs that make them special.

---

Takeaways

The key factor of good testing is to only test one variable at a time – so the results can be clearly attributed to that change – and to test for as long as possible to gather a decent pot of data for comparison and analysis.
So why not give it a go, after all your customers really do know best!
TWEET THIS BLOG

---