Booking.com wraps every product change in an A/B test — 1,000+ live at once

At Booking.com, experimentation is not a tool some teams use — it is how the company decides. For more than ten years the travel platform has run evidence-based product development, and by 2017 its departments were running and analysing more than a thousand concurrent experiments on any given day, across mobile apps, hotelier tools, customer-service phone lines and internal systems.

The deliberate choice was democratization. The experiment platform acts as a searchable repository of every success and failure since the very first experiment, so anyone can consult history and form new hypotheses. Generic APIs abstract away recruitment, randomization and reporting, and each department has its own dedicated support team, so running a test does not require permission from product management.

Trust came from engineering discipline: two separately maintained pipelines compute the same metrics and are monitored against each other, a pool of A/A experiments validates the statistics, and safeguards such as pre-registration and peer review of successes curb p-hacking. Everyone in the company is empowered to stop a harmful experiment themselves.

The result was a company where 'every change, from entire redesigns and infrastructure changes to bug fixes, is wrapped in an experiment,' as the engineers wrote in a 2017 paper — turning product development into a stream of measured bets rather than a series of opinions.