Designing Experiment Hypotheses and Success Metrics for PostHog Onboarding Experiments
This skill teaches you how to formulate a testable hypothesis, select the right primary and secondary conversion metrics, and define what winning looks like before launching an onboarding A/B test in PostHog or any experimentation platform.
Start by identifying the single onboarding behavior you want to change, then write a hypothesis in the format 'If we [change], then [metric] will [direction] by [magnitude] because [rationale].' Choose one primary metric tied to that behavior, add two or three secondary metrics to catch unintended effects, and set a minimum detectable effect before launching. This prevents post-hoc rationalization and gives PostHog a clear statistical target.
Outcome: You produce a written experiment brief containing a falsifiable hypothesis, one primary metric with a minimum detectable effect, two or three guardrail or secondary metrics, and explicit pass/fail criteria that your team agrees on before a single user enters the experiment.
Prerequisites
- Basic understanding of A/B testing concepts (control vs. variant, statistical significance)
- Familiarity with PostHog events, actions, and the experiments UI
- An existing onboarding flow with at least one tracked conversion event
- Access to historical onboarding funnel data (baseline conversion rates)
Overview
Every onboarding experiment starts with a decision about what you are trying to learn, not what you are trying to build. The hypothesis and metrics you set before launch determine whether the experiment produces a clear business decision or a pile of ambiguous data you argue about in Slack. This skill sits at the very front of the PostHog Experiments Onboarding A/B Test Method. It happens after you have identified a problem in your onboarding funnel and before you configure feature flags or touch any code. If you skip it or rush through it, the downstream steps (flag setup, variant implementation, statistical analysis) all inherit the ambiguity.
The concrete artifact you produce is an experiment brief, a short document (often a single page, sometimes a Notion card or a PostHog experiment description field) that contains four elements: the hypothesis statement, the primary metric, the secondary and guardrail metrics, and the decision criteria. The hypothesis statement follows a structured template that forces you to name the change, the expected outcome, the expected magnitude, and the reason you believe the change will work. The metrics section names specific PostHog events or actions and states how each one will be calculated (unique users who fire the event divided by unique users who enter the experiment, for example). The decision criteria state the minimum detectable effect, the confidence threshold, and any guardrail thresholds that would override a positive primary result.
When done well, this skill eliminates two of the most common failure modes in posthog onboarding experiments: launching a test with no clear definition of success, and declaring a winner based on whichever metric happened to move. Teams that write experiment briefs before launch ship more decisive experiments, spend less time debating results, and build a searchable library of institutional knowledge about what works in their onboarding flow. The brief also becomes the input for the sibling skill of running A/B tests in the PostHog experiments tab, where you configure the experiment with the metrics you chose here.
The difficulty is intermediate because the mechanics are straightforward but the judgment calls are not. Choosing between a funnel conversion rate and a time-to-activation metric, for instance, requires understanding your onboarding flow's shape and your business model's economics. You need enough historical data to estimate a baseline and enough product intuition to set a realistic minimum detectable effect. If you have never run an experiment before, expect to spend closer to 90 minutes on your first brief. After two or three experiments, the process compresses to about 30 minutes.
How It Works
A hypothesis is not a prediction. It is a structured statement that constrains the experiment so that only certain outcomes count as evidence for or against the change. The classic template is: 'If we [specific change to the onboarding flow], then [primary metric] will [increase or decrease] by at least [minimum detectable effect] within [observation window], because [causal mechanism].' Each clause does real work.
The 'specific change' clause forces you to describe the variant precisely enough that another engineer could build it without asking questions. 'Simplify the onboarding' is not a change. 'Replace the five-field signup form with a single-email form and defer profile completion to after first value moment' is a change. The more specific the change, the more interpretable the result, because you know exactly what caused the movement (or lack of it).
The 'primary metric' clause is where most teams stumble. You need a single number that the experiment is designed to move. In onboarding experiments, the primary metric almost always falls into one of three buckets: a step-completion rate (percentage of new signups who complete a specific onboarding step), a time-to-value metric (median minutes or sessions until a user performs a key activation action), or a downstream retention metric (Day 7 or Day 14 return rate). Choosing the right bucket depends on where the bottleneck sits. If users drop out of a specific step, a step-completion rate is the tightest signal. If users complete every step but never come back, retention is the right lens. If users complete steps slowly and churn before reaching the aha moment, time-to-value reveals the friction. The primary metric should be the one closest to the bottleneck you identified.
Secondary metrics exist to catch side effects. A variant might boost step completion by auto-skipping a configuration step, but if that step was teaching users how to use a core feature, Day 7 retention might drop. Secondary metrics act as guardrails: you will not ship a variant that wins on the primary metric if it degrades a guardrail metric past a threshold you set in advance. Two or three secondary metrics are enough. More than that and you are fishing for significance.
The minimum detectable effect (MDE) is the smallest improvement you care about. It is not a guess at what the variant will achieve. It is a business decision: 'A 2% absolute lift in activation rate is worth the engineering investment to ship this variant permanently; a 0.5% lift is not.' The MDE, combined with your baseline rate and your expected traffic volume, determines how long the experiment needs to run. Setting the MDE too small means the experiment runs for weeks or months. Setting it too large means you will miss real but modest improvements. A practical heuristic for onboarding experiments is that the MDE should be the smallest lift that would justify the effort of building and maintaining the variant. For most teams, that is somewhere between 2 and 5 percentage points on a conversion rate.
The observation window matters because onboarding metrics can have delayed effects. A variant that increases Day 1 activation might not show its retention impact until Day 14. If your primary metric is a Day 7 retention rate, you need at least 7 days of data after the last user enters the experiment, not 7 days after launch. The PostHog Experiments Onboarding A/B Test Method recommends planning for the observation window during hypothesis design so you can set the right experiment duration in the PostHog UI.
Finally, the decision criteria lock in how you will interpret results before you see them. A common framework is: ship the variant if the primary metric improves by at least the MDE with 95% confidence (frequentist) or 95% probability of being better (Bayesian), AND no guardrail metric degrades by more than X%. If the primary metric is inconclusive after the planned sample size, call it a draw and move on. Writing these criteria in advance is the single highest-leverage thing you can do to prevent post-hoc rationalization.
Step-by-Step
Step 1: Identify the onboarding bottleneck you want to address
Open your onboarding funnel in PostHog (Funnels insight, filtered to new users from the last 30-90 days). Walk through each step and note the conversion rate between consecutive steps. The step with the largest absolute drop-off is usually the bottleneck, but context matters: a 40% drop from 'created account' to 'completed profile' might be normal for your product, while a 15% drop from 'invited teammate' to 'sent first message' might be catastrophic for your activation loop. Write down the specific step where the drop occurs, the baseline conversion rate through that step, and a plain-language description of what users are being asked to do at that point.
This is the input for your hypothesis.
Tip: If you do not have 30 days of funnel data, use whatever you have, but note the sample size. Baselines computed from fewer than 200 users per step are unreliable and will inflate your required experiment duration.
Step 2: Articulate the causal mechanism behind your proposed change
Before writing the hypothesis, spend five minutes writing down why you believe the change will work. This is the 'because' clause, and it is the part most teams skip. State the user problem (what friction, confusion, or cognitive load exists at the bottleneck step), the behavioral insight (what you have observed in session recordings, support tickets, or user interviews that supports the diagnosis), and the design logic (how the proposed variant removes or reduces that friction). For example: 'Users abandon the integration setup step because they do not know which integration to choose first.
Session recordings show users clicking between tabs for 45+ seconds before leaving. ' The causal mechanism is what makes the hypothesis falsifiable, because if the variant wins but for a different reason, you still learned something, and if it loses, you know which assumption was wrong.
Tip: If you cannot articulate a causal mechanism, you are not ready to experiment. You are guessing. Go back to qualitative research (session recordings, interviews, support tickets) and find the friction before you test a fix.
Step 3: Write the structured hypothesis statement
' Fill in every bracket. The specific change should be precise enough that an engineer could build the variant from the sentence alone. The primary metric should name the exact PostHog event or funnel step. The direction is almost always 'increase' for conversion or 'decrease' for time-to-value.
The MDE is a number you will determine in Step 5, so leave a placeholder for now and fill it in after you calculate it. , 'within 7 days of signup'). Write the complete hypothesis in the experiment description field in PostHog or in your team's experiment brief template.
Tip: Read the hypothesis out loud to a teammate who is not involved in the experiment. If they cannot explain what the test is and how you will know if it worked, the hypothesis is not specific enough.
Step 4: Choose your primary metric
Your primary metric is the single number the experiment is designed to move. It must be a metric you are already tracking in PostHog (or one you can instrument before launch). For onboarding experiments, the three most common primary metric types are: (1) Step conversion rate, computed as unique users who complete step N+1 divided by unique users who reach step N. (2) Funnel completion rate, computed as unique users who complete the entire onboarding flow divided by unique users who enter it.
, 48 hours of signup). Pick the metric that is closest to the bottleneck from Step 1. If the bottleneck is a specific step, use the step conversion rate. If multiple steps are weak, use the full funnel completion rate.
If the funnel completes but users do not activate, use the activation rate. Write the metric name, the exact PostHog event(s) or funnel definition, and the baseline value from your historical data.
Tip: Avoid composite metrics like 'engagement score' as your primary metric. They are hard to interpret and even harder to act on. If you cannot explain what a 3% lift in your primary metric means in plain language, pick a simpler metric.
Step 5: Set the minimum detectable effect (MDE)
The MDE is the smallest improvement that would justify shipping the variant permanently. It is a business decision, not a statistical one. Start by estimating the value of a one-percentage-point lift in your primary metric. If your onboarding flow sees 1,000 new users per week and your current activation rate is 30%, a 3-percentage-point lift means 30 additional activated users per week.
Is that worth the ongoing maintenance cost of the new variant? If yes, your MDE is 3 percentage points. If you need at least 50 additional activated users per week to justify the work, your MDE is 5 percentage points. Once you have the MDE, use a sample size calculator (PostHog provides one in the experiment setup, or use Evan Miller's calculator) with your baseline rate and MDE to determine the required sample size per variant.
Divide that by your weekly new-user volume to estimate how many weeks the experiment needs to run. If the duration is longer than 6-8 weeks, consider whether you can increase your MDE (accept only larger wins) or increase traffic to the experiment.
Tip: A common trap is setting the MDE at 1% because you want to detect small effects. This often results in experiments that need to run for months and tie up engineering resources. For most onboarding experiments with moderate traffic (500-5,000 new users per week), an MDE of 3-5 percentage points is practical.
Step 6: Select secondary and guardrail metrics
Choose two or three additional metrics that will help you interpret the primary result and catch unintended consequences. , 'time to complete onboarding' as a secondary when the primary is 'activation rate'). , 'Day 7 retention rate,' 'support ticket volume in first 48 hours,' 'feature X adoption rate'). For each secondary or guardrail metric, write the metric name, the PostHog event or insight that computes it, the baseline value, and the threshold at which a guardrail violation would override a positive primary result.
' Add all of these metrics to your PostHog experiment configuration as secondary goals.
Tip: If you are testing a change that simplifies onboarding by removing a step, always include a guardrail metric for the feature that step was teaching. Skipping a tutorial step often boosts short-term conversion but hurts long-term feature adoption.
Step 7: Define the decision criteria and write the experiment brief
Write explicit rules for every possible outcome. The four common outcomes are: (1) Primary metric wins by at least the MDE with sufficient confidence, and no guardrail is violated. Decision: ship the variant. (2) Primary metric wins but a guardrail is violated.
Decision: do not ship, investigate the guardrail degradation. (3) Primary metric is flat or inconclusive after the planned sample size. Decision: do not ship, archive the experiment, and move on. (4) Primary metric loses.
Decision: do not ship, update your understanding of the bottleneck. Write these decision rules into the experiment brief alongside the hypothesis, the metrics, and the baselines. Store the brief where your team can find it (PostHog experiment description, a Notion page, a GitHub issue). This document is the contract your team signs before launch.
It prevents the most damaging failure mode in experimentation: changing the success criteria after you see the data.
Tip: Have the person who proposed the experiment and the person who will analyze the results both review and agree on the decision criteria. Disagreements about what 'winning' means are much cheaper to resolve before launch than after.
Step 8: Validate instrumentation before launch
Before you flip the feature flag, verify that every metric in your brief is actually being tracked correctly. In PostHog, create the funnel insight or trend insight for each metric and confirm it returns data for recent users. Check that the event names in your brief match the event names in PostHog exactly (including casing and property names). Run through the onboarding flow yourself as a test user in both the control and variant states, and confirm that the expected events fire at each step.
If your primary metric is a funnel, verify that the funnel steps are in the correct order and that the conversion window matches your hypothesis's observation window. This step catches the most embarrassing experiment failure: running a test for three weeks and then discovering that the key event was never instrumented in the variant.
Tip: Use PostHog's Live Events view to watch events fire in real time as you walk through the flow. If an event does not appear within a few seconds, check your instrumentation code before proceeding.
Examples
Example: B2B SaaS product with a multi-step onboarding wizard (small team, 800 signups/week)
A project management tool sees 800 new signups per week. The onboarding wizard has five steps: create account, name your workspace, invite teammates, create first project, add first task. Funnel data shows a 38% drop-off between 'invite teammates' and 'create first project.' The team suspects the invite step creates friction for solo evaluators who have no one to invite yet. The team has three engineers and no dedicated data scientist.
' They articulate the causal mechanism: solo evaluators hit the invite step, feel the product requires a team, and leave. Session recordings confirm this, showing users hovering on the 'Skip' button (which is styled as a secondary link and easy to miss) before closing the tab. ' Primary metric: step conversion rate from 'invite teammates' to 'create first project,' baseline 62%. MDE: 5 percentage points.
Secondary metric: teammate invitations sent within 14 days (to check whether skipping hurts eventual team adoption). Guardrail: Day 7 retention must not drop below 20% (baseline 23%). 5 weeks. Decision criteria: ship if primary metric lifts by 5pp+ at 95% confidence and Day 7 retention stays above 20%.
The team writes this into a Notion card, reviews it in a 15-minute standup, and hands it to the engineer configuring the PostHog experiment.
Example: B2C mobile app with a single-screen onboarding (large team, 15,000 installs/week)
A fitness app gets 15,000 new installs per week. The onboarding is a single screen asking users to select their fitness goal (lose weight, build muscle, stay active). After selection, users are taken to a personalized workout plan. The activation metric is 'completed first workout within 48 hours,' currently at 18%. The product team believes the three goal options are too generic and that adding a follow-up question about experience level would improve workout plan relevance and first-workout completion.
2% of new users complete their first workout within 48 hours, n=128,000. They write the causal mechanism: users who receive a workout plan mismatched to their experience level (a beginner getting an advanced routine, or vice versa) abandon the first workout because it feels too hard or too easy. Support tickets and app store reviews confirm complaints about difficulty mismatch. 2%.
MDE: 2 percentage points. Secondary metrics: onboarding completion rate (to check if the extra screen causes drop-off), second workout completion within 7 days. Guardrail: onboarding screen completion rate must not drop by more than 3 percentage points (baseline 89%, threshold 86%). At 15,000 installs/week, a 2pp MDE on an 18% baseline requires roughly 7,500 users per variant, so the experiment reaches adequate power in about one week.
Decision criteria: ship if primary lifts 2pp+ at 95% confidence and onboarding completion stays above 86%. The team documents this in the PostHog experiment description field and sets the experiment to auto-calculate significance using Bayesian analysis.
Example: Developer tool with API-first onboarding (mid-size team, 2,000 signups/week)
A developer analytics platform gets 2,000 new signups per week. Onboarding involves signing up, creating an API key, installing the SDK, and sending the first event. The bottleneck is between 'created API key' and 'first event received,' where conversion drops from 71% to 34%. The team hypothesizes that the SDK installation docs are too long and overwhelming, and that providing a one-click copy-paste snippet for the user's detected framework would reduce friction.
Historical data from the last 90 days shows 34% of users who create an API key send their first event within 72 hours, n=18,000. Session recordings show users spending 3-5 minutes scrolling through documentation before many leave the page entirely. The team writes the mechanism: developers who see a wall of documentation for multiple frameworks experience decision paralysis and defer setup to 'later,' which often means never. ' Primary metric: conversion from 'created API key' to 'first event received' within 72 hours, baseline 34%.
MDE: 4 percentage points. Secondary: median time from API key creation to first event (expecting a decrease). 2%), and 14-day retention must stay above 15% (baseline 18%). Sample size calculation: 4pp MDE on 34% baseline requires about 1,200 per variant.
5 weeks total. Decision criteria are documented in a GitHub issue linked from the PostHog experiment, reviewed by the engineering lead and the PM.
Example: E-commerce platform with seller onboarding (small team, 400 new sellers/week)
An online marketplace onboards 400 new sellers per week. The seller onboarding has four steps: create account, set up store profile, list first product, and receive first order. The team is focused on the gap between 'set up store profile' and 'list first product,' where conversion is 52%. They believe the product listing form is too complex for first-time sellers and want to test a guided listing wizard.
The team reviews 45 days of data: 52% of sellers who complete their store profile list their first product within 14 days, n=2,520. Support tickets reveal that sellers frequently ask which fields are required and what makes a good listing. The mechanism: first-time sellers face a blank product form with 18 fields and no guidance on which fields matter most, leading to form abandonment. ' Primary metric: 14-day first-product listing rate, baseline 52%.
MDE: 6 percentage points (higher than usual because at 400 sellers/week, a smaller MDE would require too long a run time). Secondary: median time from profile completion to first listing. Guardrails: listing quality score (computed from completeness of optional fields) must not drop by more than 10% (baseline 72/100, threshold 65/100), because a simpler form might produce lower-quality listings. Sample size: 6pp MDE on 52% baseline requires about 550 per variant, so the experiment needs roughly 3 weeks at 50/50 split, plus 14 days for the observation window, totaling about 5 weeks.
The team acknowledges this is a long experiment and agrees to resist peeking at results before the planned end date. Decision criteria are written into a shared Google Doc and reviewed with the marketplace operations lead.
Best Practices
Write the hypothesis before you design the variant, not after. The hypothesis constrains the design space and prevents scope creep. When you design first and hypothesize second, you end up writing a hypothesis that justifies the design you already built rather than one that tests a genuine belief about user behavior. Teams that reverse the order ship experiments that change three or four things at once, making it impossible to attribute the result to any single change.
Use one primary metric per experiment, not two or three. Multiple primary metrics create a multiple comparisons problem: the more metrics you treat as primary, the higher the chance that at least one shows a false positive. If you genuinely need to optimize for two metrics simultaneously, run two sequential experiments or use a single composite metric (like a weighted score) and document the weights in your brief.
Set the MDE based on business value, not on what you think the variant can achieve. Your estimate of the variant's effect is almost always wrong (and usually optimistic). The MDE is a threshold of caring, not a prediction. If a 2% lift is not worth shipping, your MDE should be higher than 2%, regardless of whether you think the variant will deliver 10%.
Include at least one guardrail metric that measures something downstream of the onboarding flow. Onboarding optimizations that boost short-term conversion at the expense of long-term retention are net negative. A Day 7 or Day 14 retention guardrail catches this pattern before you ship a variant that inflates vanity metrics while hurting the business.
Document baselines with dates and sample sizes. A baseline of '32% activation rate' is less useful than '32% activation rate, measured from Jan 1 to Mar 31, n=4,200 new users, excluding users from paid acquisition campaigns.' The context prevents future misinterpretation and makes it possible to compare experiments across time periods.
Review the brief as a team before launch, ideally in a 15-minute synchronous review. The review serves two purposes: it catches logical errors in the hypothesis or metric definitions, and it creates shared ownership of the decision criteria. When the team has agreed in advance on what winning looks like, post-experiment debates shrink dramatically.
Archive every experiment brief, including experiments that were inconclusive or lost. The archive becomes your team's institutional memory. When someone proposes a similar experiment six months later, you can point to the prior brief and its results. Over time, the archive reveals patterns about what types of changes tend to work in your onboarding flow and which assumptions keep being wrong.
Common Mistakes
Writing a vague hypothesis like 'The new onboarding flow will improve conversion'
Correction
This hypothesis is unfalsifiable because it does not specify which metric, what magnitude of improvement, or what time window. It also bundles multiple changes under 'new onboarding flow,' making it impossible to attribute the result. Rewrite it using the structured template: name the specific change, the exact metric, the MDE, the observation window, and the causal mechanism. If you cannot fill in every bracket, the hypothesis is not ready.
Choosing a primary metric that is too far downstream from the change
Correction
If you change the signup form, measuring Day 30 revenue as your primary metric means the experiment needs to run for months and the signal is diluted by dozens of confounding factors between signup and purchase. This happens because teams want to tie experiments to business outcomes, which is admirable but impractical for most onboarding tests. Choose a primary metric that is one or two steps downstream from the change, not ten. Use the downstream metric (revenue, retention) as a guardrail or secondary metric instead.
Setting the MDE after seeing early results
Correction
This is a form of p-hacking. 8% lift after one week, and then lower the MDE to 3%, you have invalidated the experiment's statistical guarantees. The temptation arises when early results look promising but do not quite meet the bar. Catch this by locking the MDE in the experiment brief before launch and treating the brief as immutable.
If the result does not meet the MDE, the experiment is inconclusive, full stop.
Using too many secondary metrics (five or more)
Correction
Every additional metric increases the chance that at least one will show a spurious significant result. Teams add excessive secondary metrics because they want to 'learn as much as possible' from each experiment. In practice, five or more secondary metrics produce a wall of data that paralyzes decision-making. Limit yourself to two or three secondary metrics that test specific hypotheses about side effects.
If you want to explore broadly, do that in a separate exploratory analysis after the experiment concludes, clearly labeled as exploratory.
Skipping the instrumentation validation step
Correction
Roughly one in five experiments at early-stage companies ships with at least one metric that is not instrumented correctly in the variant. This is because the variant code is new and may not fire the same events as the control, especially if the variant rearranges or removes UI elements. The result is missing data that makes the experiment uninterpretable. Before every launch, walk through both control and variant flows as a test user, watching PostHog's Live Events to confirm every event fires.
Budget 30 minutes for this step.
Not agreeing on decision criteria before launch, then debating results afterward
Correction
Without pre-registered decision criteria, results become a Rorschach test. The designer sees a 4% lift and wants to ship. The engineer sees a guardrail dip and wants to hold. The PM sees an inconclusive result and wants to extend the test.
These debates waste time and erode trust in experimentation. The fix is simple but requires discipline: write decision criteria in the brief, get explicit sign-off from stakeholders, and treat those criteria as a contract.
Other Skills in This Method
Running A/B Tests in the PostHog Experiments Tab
Step-by-step walkthrough of creating, launching, and monitoring an A/B test using PostHog's Experiments UI, including variant allocation and goal setup.
Setting Up PostHog Feature Flags for Experiment Variants
How to create and configure feature flags in PostHog to assign users to control and test variants in an A/B experiment.
Comparing PostHog Experiments with Eppo, LaunchDarkly, and Other Platforms
How to evaluate PostHog's experimentation capabilities against dedicated tools like Eppo, Statsig, and LaunchDarkly based on analysis methods, integrations, and pricing.
Shipping the Winning Variant and Cleaning Up Feature Flags
How to roll out the winning experiment variant to 100% of users, remove the losing variant's code, and archive feature flags to keep your codebase clean after an experiment concludes.
Segmenting New User Cohorts for Onboarding Experiments
How to target experiments specifically to new users or sign-up cohorts using PostHog's person properties and cohort filters to avoid contaminating results with existing users.
Interpreting Bayesian and Frequentist Results in PostHog
How to read PostHog's experiment results dashboard, understand credible intervals vs p-values, and decide when an experiment has reached statistical significance.
Integrating PostHog A/B Tests with Webflow and Marketing Pages
How to implement PostHog experiments on no-code or marketing landing pages using the JavaScript snippet, Webflow custom code, and anti-flicker techniques.
Frequently Asked Questions
How do I choose between a funnel conversion rate and an activation rate as my primary metric?
Use a funnel conversion rate when the bottleneck is a specific step in your onboarding flow and the change you are testing targets that step directly. Use an activation rate when the bottleneck is not a single step but a general failure to reach the aha moment. The rule of thumb: if your change modifies one screen or interaction, measure the conversion rate through that screen. If your change restructures the entire flow, measure end-to-end activation. Mixing them (changing one step but measuring overall activation) dilutes the signal and makes the experiment harder to interpret.
How long should designing the hypothesis and metrics take for an experienced team?
For a team that has run three or more onboarding experiments, the full process from bottleneck identification through a finalized brief should take 30-45 minutes. The first time, expect 60-90 minutes because you will be establishing templates, finding baseline data, and debating metric definitions. If it takes longer than two hours, you are either over-engineering the hypothesis or lacking the baseline data you need, in which case you should pause and instrument your funnel before proceeding.
Should I design the hypothesis before or after segmenting my user cohorts?
Design the hypothesis first, then refine the cohort. The hypothesis tells you which users are relevant to the test. If your hypothesis is about solo evaluators abandoning a team-invite step, your cohort should exclude users who arrive with a team invitation link. Segmenting first (without a hypothesis) often leads to testing a generic change on a narrow cohort, which reduces power without improving signal. That said, high-level cohort thinking (new signups vs. returning users, free vs. paid) should inform the hypothesis, so the two steps are iterative. See [segmenting new user cohorts](/skills/segmenting-new-user-cohorts-for-onboarding-tests) for the detailed segmentation workflow.
What if my onboarding flow does not have enough traffic to detect a small MDE?
You have three options: raise the MDE (accept that you will only detect large effects), extend the experiment duration (if you can tolerate a longer timeline), or change the metric to one with higher variance reduction (e.g., a step-level conversion rate instead of a full-funnel rate). A fourth option is to batch multiple small changes into a single variant and test the bundle, but this sacrifices your ability to attribute the result to any single change. For flows with fewer than 200 new users per week, consider qualitative methods (user interviews, session recordings, usability tests) instead of statistical experiments.
Why does my hypothesis keep changing after I start building the variant?
This usually happens because the hypothesis was written at too high a level of abstraction. When the engineer starts building, they discover constraints (technical limitations, edge cases, design conflicts) that force the variant to differ from what was originally described. The fix is to involve engineering in the hypothesis drafting step, or at least run the hypothesis past an engineer before finalizing it. A 10-minute feasibility check prevents hypothesis drift during implementation. If the variant must change materially, rewrite the hypothesis before launch rather than proceeding with a mismatch.
Can I use the same guardrail metrics across all my onboarding experiments?
You can and should have a standing set of default guardrails (Day 7 retention, support ticket volume, error rates) that apply to every experiment. But each experiment should also have at least one experiment-specific guardrail based on the change being tested. For example, an experiment that removes a tutorial step should guardrail the feature adoption rate for whatever that tutorial was teaching. Default guardrails catch generic regressions. Experiment-specific guardrails catch the predictable side effects of the particular change you are making.
How do I handle experiments where the primary metric is a time-to-event metric rather than a rate?
Time-to-event metrics (like median minutes to first activation) require different statistical treatment than conversion rates. In PostHog, you can track these as trend metrics rather than funnel metrics. , 'reduce median time to first event from 45 minutes to 35 minutes, a 22% reduction') rather than an absolute number. Be aware that time metrics are typically skewed (a few users take very long), so use medians rather than means, and note that you may need a larger sample size to detect differences in medians. If your team is not comfortable with survival analysis or percentile comparisons, consider converting the time metric into a rate metric ('percentage of users who activate within 48 hours') for simpler analysis.