Experimentation is at the heart of innovation at Deliveroo. By continuously testing new ideas, we refine our products, mitigate risks, and ensure data-driven decision-making.

Experimentation at Deliveroo has significantly boosted our innovation as a company. It has allowed us to explore new ideas, protected us from unforeseen challenges by preventing wasted time and resources on ineffective initiatives, and promoted intellectual honesty by validating our assumptions.
Over time, the breadth of our experimentation programme, our tooling and our culture have undergone significant development.
Each year, we run hundreds of experiments, testing a wide array of features. These experiments range from simple UI modifications to complex algorithm adjustments.
We employ a variety of methods, from standard A/B testing to advanced techniques like interleaving, multi-armed bandits and switchbacks.
We’ve moved from teams using ad-hoc methods for deploying and analysing experiments to a mature experimentation platform, which standardises and automates much of the experiment process.

But successful experimentation requires more than just technical expertise and tooling. It requires an experimentation “culture” – a shared set of principles and values that embeds experiments in the broader organisational context, and ensures that experimentation is done in a way that is maximally impactful.
As we have matured technically and increasingly democratised our technical capabilities, we have also evolved a set of such principles, which codifies that cultural side.
These enable us to uphold high-quality experimentation whilst achieving our current goal of broadening the pool of Deliveroo employees capable of running experiments.

Principle 1: Every experiment starts with a clear hypothesis and success criteria.

Experiments work best when they are specific. We formulate our hypothesis based on past experiments, anecdotes, user research, and competitive analysis – leveraging existing learnings helps us design better experiments. We avoid experimenting aimlessly just to ‘see what happens’, risking learning nothing.
Experiments are a highly valuable resource and unstructured experimentation is wasteful.

We set success criteria for rollout before the experiment starts. It may be tempting to decide on the roll out decision rule once results are observed; but in that case, we are more likely to be biased towards forming a hypothesis that supports the observed impact, rather than admitting the experiment results challenged our initial assumptions.

Table 1: Good vs. bad alternative hypothesis for an “Order again” carousel

Principle 2: You don’t have to be a data