This is what Airbnb, Spotify and Google have in common
They don't always A/B test
Hi there,
As an early-stage startup, you might not always have the resources need to test and grow.
Traffic is one of those resources.
It’s a must for startups - many don’t have the traffic needed to run A/B tests, but find themselves wasting resources on non-significant differences or unreliable results by running them anyway.
But don’t fret if this sounds like you.
Airbnb, Spotify and Google all have high levels of traffic. A luxury for them.
But they don’t always A/B test. And you don’t need to either.
I loved seeing examples from these big brands on how they use alternative approaches to testing in a great article by Convert.com: Optimization Beyond A/B Testing: The Answer To Testing On Low-Traffic Sites.
So if you don’t have any traffic to test, here are some tips from Convert.
AirBnb: Observe user behaviour
Observing user behaviour helped AirBnb see that hosts were sending photos for check-in instructions.
So in response, they created a tool for hosts to create visual guides making check-ins more seamless.
Google: Use user feedback
For their education platform, they used user feedback from teachers to improve Google Meet by adding attendance taking and hand raising to improve the user experience.
Spotify: Use Human perspective
Based on human perspective, Spotify created a “Good Morning” feature using machine learning to recommend songs (interestingly the initial A/B test failed, but this had a positive impact!)
But qualitative methods aren’t enough
Qualitative methods are super useful, but on their own aren’t enough. Don’t use just one method to understand the impact of your changes.
Methods vary in showing more behavioural or attitudinal insights, as well as being more qualitative or quantitative.
So combining mixed-methods paints a fuller picture of whatever you are trying to investigate.
So what are some alternative methods we can use? Convert advises…
The user focused A/B test
Rich Page shares in the article an unusual approach:
Create a new page based on research
Run a preference test on 100 users to ask what they prefer and why
Use recording tools to observe reactions to each
Also use an on-site survey to collect insights
Afterwards run five user tests to gather further qualitative insights
Compile the data to understand patterns and roll out the best one
✅ Pros: user-centric and feedback focused. You don’t need much traffic and it provides plenty of actionable insights
❌ Cons: It’s quite complex and resource intensive to analyse. You need external tools and don’t rely much on quantitative data
User testing
User testing doesn’t have to be as time-consuming or as much of a big project as people think.
It’s important to consistently conduct smaller rounds of testing to see how your changes are being perceived.
My top tip? Use tools like Respondent.io for finding individuals for either moderated or unmoderated user testing.
✅ Pros: user-centric and feedback focused. Gives a lot of ideas and allows you to understand the why behind feedback
❌ Cons: small sample size which tends to work better when you have someone to guide the user and not bias them
Pre / post tests
André Vieira shared his approach in the article:
Define key metrics to measure impact
Establish a clear baseline
Implement the change
Monitor post change metrics
Compare and analyse before / after
✅ Pros: very simple and flexible to use and gain insights with no traffic splitting needed
❌ Cons: you need to be really careful of external factors as there is no control group, meaning it can be hard to know what caused the difference you see
If you want to learn more about Convert’s suggestions for testing on low traffic sites, check out the full article below.
Convert Experience
If you want that extra guidance with your A/B testing, I recommend checking out Convert Experience.
While I do partner with Convert transparently, I hugely recommend them for A/B testing too.
Convert Experience is an experimentation platform that provides comprehensive features and support to run your tests across multiple channels. It’s great value for money and has a 4x faster support desk than average.
It’ll let you set up a personalised website experience in minutes, to test personalised messaging to different audiences and run multiple types of experiment on them including:
A/B testing
Split testing
Multivariate testing
Multipage experiments
Personalisations
Full stack experiments
You can get a 15-day Free Trial below.
Recommendation
Convert has tonnes of brilliant resources that will guide you through your testing struggles.
Here are some of my favourite articles to help with similar problems:
How to A/B Test on Low-Traffic Sites - guides you through other options for testing such as through email marketing, through social media and through paid ads
The AI Playbook for Research, CRO and Experimentation - a free playbook compiling 3000+ hours of research that provides a clear, actionable guide for employing AI tools without focusing excessively on the hypotheticals of AI
Demystifying a Culture of Experimentation - a fascinating lecture covering big brands like AirBnb on how they’ve created a culture of experimentation that ensures that stakeholders are engaged in the experimentation process
Quick Guide to Low Traffic Testing - a full, free guide including 5 more tips from Jonny Longden on how to test on low traffic sites
-
So remember, even though Airbnb, Spotify and Google have the luxury of traffic, they still use alternative approaches. And you can too.
Chat soon,
Daphne