PostHog Experiments: A Practical Method for Onboarding A/B Tests
PostHog experiments let you run A/B tests by splitting users into control and variant groups using feature flags, measuring conversion across multiple success metrics, and deciding winners with Bayesian or frequentist statistical analysis. Applied to onboarding flows, the method tests changes like different signup steps, welcome screens, or activation prompts against the current experience. You ship the variant that lifts your target metric and archive the feature flag.
Overview
PostHog experiments emerged from the PostHog growth team's own work improving their product's onboarding funnel. The method was demonstrated publicly by Joshua, a member of the growth team, in a walkthrough that showed the end-to-end workflow of hypothesizing an onboarding improvement, wiring up feature flags, launching the experiment, and reading the results. While A/B testing as a discipline stretches back to Ronald Fisher's agricultural trials in the 1920s and was refined by internet companies like Google and Amazon in the 2000s, PostHog's contribution is a tightly integrated open-source implementation that bundles feature flags, event tracking, experiment configuration, and statistical analysis into a single platform. This removes the duct-tape layer that teams historically needed between separate tools for flagging, analytics, and experiment evaluation.
The underlying mental model is straightforward but often misapplied. You form a hypothesis about a specific change to your product experience, typically phrased as "If we change X, then metric Y will improve by Z%." You then use a feature flag to randomly assign incoming users to either the control (unchanged experience) or one or more test variants. Both groups use the product simultaneously under identical external conditions. After enough users have passed through each variant, you evaluate whether the observed difference in your target metric is statistically meaningful or just noise. PostHog offers both Bayesian analysis, which gives you a probability that one variant is better, and frequentist analysis, which tests against a null hypothesis with a p-value. The Bayesian approach tends to be more intuitive for product teams because it answers "what's the probability that variant B beats control?" rather than the more convoluted "can we reject the hypothesis that there's no difference?"
What separates PostHog experiments from generic A/B testing advice is the tight coupling between feature flags and experiment lifecycle. In many organizations, the flag system and the analytics system are operated by different teams or even different vendors. This creates friction: engineers toggle flags in LaunchDarkly or a homegrown system, analysts query Amplitude or Mixpanel separately, and the mapping between flag state and metric movement is maintained manually or not at all. PostHog collapses these into one workflow. You create the experiment, PostHog generates the feature flag, you instrument your code to read the flag, PostHog tracks the events, and the experiments tab shows results in context. This integration is especially valuable for onboarding flows, where the experiment lifecycle tends to be short (days to weeks, not months) and the number of simultaneous tests can be high.
Compared to dedicated experimentation platforms like Eppo, Statsig, or Optimizely, PostHog experiments occupy a pragmatic middle ground. Eppo and Statsig offer deeper statistical rigor, warehouse-native architectures, and advanced features like CUPED variance reduction or sequential testing. Optimizely and VWO provide visual editors for marketing teams who want to test without writing code. PostHog's strength is that experimentation is one capability inside a broader product analytics suite, so teams that already use PostHog for event tracking, session replays, or feature flags get experiments essentially for free, with no data integration overhead. For onboarding specifically, this means you can define your experiment, watch session replays of users in each variant, check funnel dropoff by variant, and review the statistical outcome all in one place.
The method has evolved as PostHog itself has matured. Early versions required more manual instrumentation and lacked the guided experiment creation flow that exists today. The addition of minimum sample size calculators, automatic significance detection, and the ability to set multiple metrics per experiment (a primary goal metric plus secondary guardrail metrics) addressed common failure modes where teams either called experiments too early or measured the wrong thing. The growth team's public documentation and tutorial videos also reflect a broader industry shift toward transparency in experimentation methodology, making it easier for smaller teams to adopt practices that were previously gatekept by dedicated data science functions at large tech companies.
Teams that benefit most from this method are product and growth teams at early-to-mid stage companies with enough traffic to reach statistical significance in a reasonable timeframe, who are already using PostHog or considering it, and who want to test changes to high-impact user journeys like onboarding, activation, and upgrade flows. If you have fewer than a few hundred users per week flowing through the funnel you want to test, the math gets slow and the method loses its advantage over qualitative approaches. If you have tens of millions of daily active users and a dedicated data science team, you may outgrow PostHog's built-in analysis and want a warehouse-native platform. For the broad middle of the market, the integrated approach works well.
How It Works
Step 1: Identify the onboarding bottleneck and form a hypothesis
Start by reviewing your onboarding funnel in PostHog (or whatever analytics you have) to find the step with the largest dropoff. Look at session replays of users who abandon at that step to understand why. Common reasons include confusion about what to do next, a step that requires too much effort, or a step whose value isn't clear to the user. ' A good hypothesis is falsifiable and specific enough that you'll know exactly what to measure. Watch out for vague hypotheses like 'improving the UX will increase signups,' which give you no clear success criterion.
Step 2: Define your primary metric, secondary metrics, and guardrails
Choose one primary metric that your hypothesis directly predicts will change. This is typically a conversion rate at a specific funnel step or an activation metric. Then pick 1-3 secondary metrics that you want to monitor for additional insight, such as time-to-complete or downstream engagement. Finally, set guardrail metrics: things you do not want to regress, like overall 7-day retention or support ticket creation rate. Write all of these down before you touch the experiment configuration. The most common mistake here is choosing a vanity metric (like page views) instead of a meaningful outcome metric (like percentage of users who send their first message). Another pitfall is having no guardrails and accidentally shipping a variant that improves step completion but degrades long-term retention.
Step 3: Create the experiment and feature flag in PostHog
Navigate to the Experiments tab in PostHog and create a new experiment. PostHog will generate a linked feature flag automatically. Configure the flag with your variants (typically 'control' and 'test', though you can add more for A/B/n tests). Set the traffic allocation, usually 50/50 for two variants. Use PostHog's sample size calculator to determine how many users you need per variant, based on your baseline conversion rate and the minimum detectable effect you care about. If the required sample size means the experiment will run longer than 4 weeks, consider whether you can increase traffic to the funnel, test a larger change with a bigger expected effect, or accept that experimentation isn't the right tool for this decision. Set targeting rules on the feature flag if you want to limit the experiment to specific user segments, like new users only.
Step 4: Instrument your code to read the feature flag and render variants
In your application code, use PostHog's SDK to check the feature flag value for each user and render the appropriate experience. For a frontend onboarding change, this typically means a conditional block where 'control' renders the existing flow and 'test' renders the new version. Make sure the flag check happens early enough in the user journey that users are assigned before they see any part of the experience being tested. A common bug is checking the flag too late, which means users see a flash of the control experience before being switched to the variant, contaminating the data. Test both variants manually in a development or staging environment before going live. Verify that PostHog is receiving the correct events for each variant by checking the live events stream.
Step 5: Launch the experiment and resist peeking
Activate the experiment in PostHog. From this point, users are randomly assigned to variants as they enter the onboarding flow. Monitor the experiment for technical issues in the first few hours: are events firing correctly, is the flag resolving as expected, are there errors in either variant? Once you've confirmed that the infrastructure is working, step away. Do not check results daily. Set a calendar reminder for when you expect to reach the required sample size. If you absolutely must peek (to check for catastrophic regressions), look only at guardrail metrics, not at the primary metric. The reason for this discipline is well-documented: repeated significance testing without correction inflates your false positive rate far beyond the nominal 5%.
Step 6: Analyze results after reaching sample size
When the experiment reaches the pre-calculated sample size, open the results in PostHog's experiments tab. Review the primary metric first: is there a statistically significant difference? If you chose Bayesian analysis, look for a probability of 95% or higher that one variant beats the other. 05. Then review secondary metrics and guardrails. A variant that improves the primary metric but degrades a guardrail metric is usually not worth shipping. If results are inconclusive (neither variant is clearly better), that is a valid and informative outcome. It means the change you tested doesn't matter enough to users to produce a measurable difference, which frees you to focus on something else. Do not extend the experiment hoping for significance. That's the same as peeking.
Step 7: Ship the winner and clean up
If the variant won, roll the feature flag to 100% of users so everyone gets the improved experience. Then remove the flag check from your codebase entirely and delete the conditional code path for the control. If the control won (the change made things worse), roll back the variant, remove the code, and document what you learned. Archive the experiment in PostHog. This cleanup step is non-negotiable. Every feature flag left in your code is a branch of conditional logic that future developers have to understand and maintain. Teams that skip cleanup end up with codebases where no one knows which flags are still active, which were experiments, and which are permanent feature toggles. Document the experiment's hypothesis, results, and decision in a shared location where future team members can learn from it.
When to Use
- When you have a specific, testable hypothesis about your onboarding flow, like 'moving the team invite step from position 3 to position 5 will increase the percentage of users who complete setup,' and enough weekly signups (typically 200+ per variant per week) to reach statistical significance within 2-4 weeks. The method works best when you have a clear before/after measurement point and the change is scoped to one step or screen.
- When your onboarding funnel analytics show a sharp dropoff at a specific step (for example, 60% of users abandon at the 'connect your data source' screen) and you have a concrete alternative design to test against the current experience. The experiment structure forces you to define exactly what 'better' means before you ship the change, which prevents the common failure mode of redesigning a step, noticing a different metric looks good, and declaring victory.
- When you are running PostHog (or planning to adopt it) for product analytics and want experimentation without integrating a separate A/B testing vendor. The value here is the zero-integration-cost path: your events, feature flags, and experiment analysis all live in one system. If you already have PostHog tracking onboarding events, you can launch an experiment in under an hour without any new instrumentation.
- When you want to test onboarding variants across different user segments, such as new users from organic search versus those from a paid campaign, and you need feature flags that respect cohort-based targeting. PostHog's flag targeting lets you run segment-specific experiments so you can learn whether an onboarding improvement that works for self-serve signups also works for users who arrive via a sales-assisted invite.
- When your team has been making onboarding changes based on intuition or qualitative feedback alone and you want to introduce a repeatable, evidence-based process. The method provides enough structure (hypothesis, flag, metrics, sample size, analysis, cleanup) that even teams without a dedicated data scientist can run credible experiments. It is particularly useful as a first experimentation framework before scaling to more complex setups.
- When you need to run multiple concurrent onboarding experiments, like testing a new welcome modal on step 1 while also testing a different activation prompt on step 4, and you need a system that handles mutual exclusion or independent flag assignment correctly. PostHog's experiment setup lets you control whether users can be in multiple experiments simultaneously.
When Not to Use
- When your weekly signup volume is too low to reach statistical significance in a reasonable timeframe. If you get 50 new users per week and you need 400 per variant for a detectable effect size, you are looking at a 16-week experiment. Over that duration, so many external factors change (product updates, seasonality, marketing shifts) that the result becomes unreliable. In low-traffic situations, you are better served by qualitative user testing, session replay analysis, or larger, more dramatic changes that don't require statistical validation to evaluate.
- When the change you want to test is a fundamental architectural shift, like moving from a self-serve onboarding model to a guided setup wizard with human touchpoints. These changes affect so many variables simultaneously that an A/B test cannot isolate what is working. The method assumes you are testing a scoped change within a stable system. If the system itself is being redesigned, use a phased rollout with cohort-based analysis instead, or simply ship the new approach and compare cohort-level retention over time.
- When you don't yet know what your onboarding funnel looks like. Running experiments before you have reliable funnel instrumentation means you'll measure the wrong things or miss key dropoff points entirely. If you can't confidently say 'X% of users complete step 3 and Y% reach activation,' your priority is building that measurement foundation, not layering experiments on top of noisy data.
- When the experiment could cause real harm to a subset of users. For example, if you are in a healthcare, financial, or safety-critical product context and the variant might degrade a user's ability to complete a critical task, the ethical risk of randomly assigning users to a potentially worse experience outweighs the learning. In these cases, staged rollouts with monitoring and rapid rollback are safer than true randomized experiments.
- When your team lacks the discipline to wait for results. If stakeholders will pressure you to call the experiment after three days because the early numbers look good, you will get false positives and make decisions based on noise. The method only works if you commit to the pre-calculated sample size and the pre-defined success criteria. Without that commitment, you are adding process overhead without gaining decision quality.
Examples
Example: Early-stage SaaS simplifying a 7-step onboarding flow
A B2B project management tool with 800 weekly signups noticed that only 35% of new users completed their 7-step onboarding wizard. Session replays showed users abandoning at step 4, which required connecting a calendar integration. The team hypothesized that making the calendar step optional (moving it to a post-onboarding prompt) would increase completion to 45%. They created a PostHog experiment with two variants: control (mandatory calendar step at position 4) and test (calendar step removed, replaced with a dismissible prompt after onboarding). 2%. Guardrail metrics showed no drop in 14-day retention. They shipped the variant and noted that the key learning wasn't just about the calendar step specifically, but that any step requiring third-party authentication in the middle of onboarding creates disproportionate friction. They applied this insight to remove another OAuth-dependent step in a subsequent experiment.
Example: Developer platform testing two onboarding paths for different personas
A developer tools company with 1,200 weekly signups had two user personas: individual developers and team leads. Their single onboarding flow tried to serve both and served neither well. They used PostHog's feature flag targeting to run a segmented experiment. Users who signed up with a company email and indicated a team size greater than 3 were routed to a team-focused onboarding (create workspace, invite members, set up project). Individual developers (personal email or team size of 1) got a solo-focused flow (create first project, write first query, see results). The primary metric was 7-day activation rate (defined as running at least 3 queries). After 4 weeks, the segmented flows outperformed the one-size-fits-all control by 23% for team leads and 18% for individual developers. The team shipped both paths. Their main takeaway was that they should have invested in persona detection earlier, as the improvement was one of the largest they'd seen from any experiment.
Example: E-commerce platform testing a value-prop-first welcome screen
An e-commerce analytics platform with 500 weekly trial signups had a 28% trial-to-paid conversion rate. The existing onboarding started with account configuration: set timezone, upload logo, enter store URL. The growth team hypothesized that starting with a value demonstration (showing the user a preview dashboard populated with sample data from their store category) would increase trial-to-paid conversion by lifting early engagement. They ran a PostHog experiment where the control saw the configuration-first flow and the test saw a sample dashboard with a prompt to connect their real store to see their own data. After 5 weeks and 1,250 users per variant, trial-to-paid conversion was 31% for the test versus 27% for control. The effect was real but smaller than hoped, and the Bayesian probability was 89%, below their 95% threshold. They decided the result was directionally useful but not conclusive enough to ship. They redesigned the experiment with a more dramatic version of the sample dashboard and ran a follow-up test, which eventually reached 95% confidence at a 34% conversion rate.
Example: Freemium product testing upgrade nudges during onboarding
A design collaboration tool with 3,000 weekly signups and a freemium model wanted to increase the percentage of users who started a paid trial within 14 days. , trying to create a fourth project when free allows three) would outperform a generic upgrade banner shown at the end of onboarding. They ran a PostHog experiment with three variants: control (no upgrade mention during onboarding), variant A (generic banner at the end), and variant B (contextual prompt at the moment of the limit). After 3 weeks with 3,000 users per variant, variant B showed a 12% trial start rate versus 7% for control and 8% for variant A. Importantly, they tracked a guardrail metric of 30-day retention among non-upgraders to make sure the contextual prompt wasn't annoying users who intended to stay on the free plan. Retention was flat across all three groups. They shipped variant B and applied the principle of contextual, moment-of-need prompts to other upgrade touchpoints throughout the product.
Skills in This Method
Running A/B Tests in the PostHog Experiments Tab
Step-by-step walkthrough of creating, launching, and monitoring an A/B test using PostHog's Experiments UI, including variant allocation and goal setup.
Setting Up PostHog Feature Flags for Experiment Variants
How to create and configure feature flags in PostHog to assign users to control and test variants in an A/B experiment.
Comparing PostHog Experiments with Eppo, LaunchDarkly, and Other Platforms
How to evaluate PostHog's experimentation capabilities against dedicated tools like Eppo, Statsig, and LaunchDarkly based on analysis methods, integrations, and pricing.
Shipping the Winning Variant and Cleaning Up Feature Flags
How to roll out the winning experiment variant to 100% of users, remove the losing variant's code, and archive feature flags to keep your codebase clean after an experiment concludes.
Designing Experiment Hypotheses and Success Metrics for Onboarding
How to formulate a clear hypothesis, choose primary and secondary conversion metrics, and define what winning looks like before launching an onboarding A/B test.
Segmenting New User Cohorts for Onboarding Experiments
How to target experiments specifically to new users or sign-up cohorts using PostHog's person properties and cohort filters to avoid contaminating results with existing users.
Interpreting Bayesian and Frequentist Results in PostHog
How to read PostHog's experiment results dashboard, understand credible intervals vs p-values, and decide when an experiment has reached statistical significance.
Integrating PostHog A/B Tests with Webflow and Marketing Pages
How to implement PostHog experiments on no-code or marketing landing pages using the JavaScript snippet, Webflow custom code, and anti-flicker techniques.