Experiments Team

Experiments Team

See what we're building

People

What we're building

  • No-code experiments / Visual editor

    A visual editor for experiments would allow users to test changes to their website / app without having to touch the code.

    Project updates

    No updates yet. Engineers are currently hard at work, so check back soon!

Recently shipped

Relative deltas and credible intervals added to A/B tests

Juraj would like everyone to know that we've now added relative deltas and credible intervals to our A/B testing tool. In order for you to understand how cool that is though, some explanation may be needed...

A relative delta is the percentage change in conversion rate between the control and test variants. So, a bigger delta means a bigger impact for an experiment.

The credible interval is...complicated. Basically, an experiment measures a certain value (like a conversion rate) and the true value isn't actually the result that's displayed - that's just an approximation because an experiment only measures a small sample of the population. The credible interval gives you a better look at the true data by showing a likely range for the results, as well as a probability percentage that reflects certainty.

Relative deltas are pretty obviously useful for a lot of situations where you want to understand the broad improvement, but credible internal is a more advanced metric which is useful for getting into the nitty-gritty of statistical significance.

Finally, we've also made it easier to ship the winning variant when your experiment reaches a significant result, via a shiny new 'Make decision' modal, which you can see above. Snazzy!

Goals

Q4 2024 Objectives

This quarter, we're focusing on more feature improvements to help users create comprehensive and easy-to-analyze experiments. These improvements will make our product more mature and complete.

Objective 1: Finish HogQL rewrite

We will migrate Experiments to HogQL, making result calculations more reliable and performant. This will also enable the addition of new features listed below.

Objective 2: Multiple experiment goals supported + visualized

We will add support for multiple goal metrics in a single experiment, allowing them to be visualized together and making it easier to interpret results across all metrics simultaneously.

Objective 3: Reusable experiment metrics

We will add the ability to create metric sets that users can save and reuse. This will reduce friction, improve maintainability, and decrease the likelihood of errors.

Objective 4: Review and adjust methodology with a statistician

We will review our current methodology to ensure it is both accurate and easy for our users to understand.

Handbook

What this team does

We build PostHog's experimentation platform, allowing users run A/B tests and feature experiments easily in their apps. We help teams validate ideas, measure impact, and make informed decisions quickly and confidently.

Some of the things we're working on:

  • Experiments UI – an interface for setting up, managing, and analyzing experiments in PostHog.
  • Experimentation API – backend services that handle experiment creation, user assignment, and result analysis.
  • Statistical analysis – implementing statistical methods to ensure accurate interpretation of results.
  • Documentation – writing documentation and tutorials to help users get the most out of Experiments.

Slack channel

#team-experiments

Feature ownership

You can find out more about the features we (and the other teams) own here