Validating Your North Star Metric with User Research

Validate North Star metric choices with user research: ask customers what value they get, compare their words with the metric, and fix the gaps.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

A North Star Metric is a claim about customers: when this number rises, more customers are getting the value the product promises. Analytics can show that the number rose. Only research with customers can show whether the behavior it counts is the value they care about. This skill uses interviews and short surveys to validate North Star metric choices, either before a new metric is adopted or as a periodic check on an existing one. The North Star Metric method covers the rest of the cycle.

Amplitude's North Star Playbook gives an example of research changing a North Star. A delivery app had considered metrics such as people opening the app, scheduled deliveries and early deliveries. Customer research showed that the deliveries customers valued most were neither early nor late but simply had no issues. The team called these "Happy Deliveries," found they correlated strongly with retention, and made them the North Star.

The playbook also recommends grounding the value exchanges behind a North Star in research. It suggests capturing the customer experience across touchpoints, including existing research or following up with additional research to understand what users really value. Where the team is guessing about customer journeys, it says to either do more research or make the assumptions explicit.

Research and analytics answer different questions, which is why they work together. The Nielsen Norman Group describes this as triangulation: using multiple sources of data or approaches to make findings more credible, with qualitative methods explaining why people behave as they do and quantitative methods showing what they do. The more consequential the decision, the more it is worth triangulating, and choosing a North Star shapes what every team works on. The output of this skill is a validation verdict with evidence: confirm, refine or replace.

How It Works

Validation starts from a written hypothesis. Take the metric's definition and state what it assumes: that customers who do this behavior are getting value, that doing it more means more value, and that customers who do not do it are getting less. Each assumption becomes a question for research.

The Sean Ellis survey gives a fast first signal of what customers value. As described by Rahul Vohra in First Round Review, it asks "How would you feel if you could no longer use" the product, with answers of very disappointed, somewhat disappointed and not disappointed. Superhuman added follow-up questions, including "What is the main benefit you receive" from the product. The answers of the very disappointed group describe the value the product delivers in customers' own words.

Interviews go deeper. Ask customers to walk through the last time the product was useful to them, what they were trying to get done, and what would have happened without it. Avoid asking about the metric or features directly; people describe value best through stories. Include customers who are highly engaged by the metric's definition and customers who are not, because the gaps show up at the edges.

Mapping connects the two sides. Put customers' descriptions of value next to the behavior the metric counts. Look for three patterns: customers who get value the metric does not see, customers the metric counts as successful who describe little value, and value moments that happen outside the product. The playbook notes that some key value exchanges happen outside the product, such as a concert-goer arriving at their seat for a ticketing app (North Star Playbook).

The verdict combines both kinds of evidence. If customers' value descriptions match the counted behavior and the quantitative link to retention holds, confirm the metric. If they mostly match but the metric counts some low-value behavior, refine its definition, often by adding a quality bar as Happy Deliveries did. If the value customers describe is something the metric cannot see, replace it or reopen the selection process.

Validation repeats. Vohra's team kept measuring its score while it worked on the product (First Round Review). A North Star can drift from value as the product and customer base change, so a light version of this research belongs in each annual review of the metric.

Step-by-Step Guide

Step 1: Write the metric hypothesis

Write the metric's definition and the assumptions behind it as testable statements. For example, "Teams that share at least one report a week get value from shared analysis." List the segments whose experience matters most. Decide what finding would make you confirm, refine or reject the metric before you start. Share the hypothesis with the team so the research answers their questions too.

Step 2: Run a short value survey

Send the Sean Ellis question and a "main benefit" follow-up to active users, following the approach described in First Round Review. Segment responses by whether each respondent meets the North Star's definition. Read the main-benefit answers of the very disappointed group closely. Note any benefit that shows up often but has no link to the metric.

Step 3: Recruit across the metric's edges

Recruit interviewees from three groups: customers who clearly meet the metric's definition, customers just below the threshold, and customers who get value by other routes, such as heavy users of a secondary workflow. Include recently churned customers where you can. Balance segments that matter to the strategy. Keep the recruit criteria written down so the verdict can be traced back.

Step 4: Interview for value stories

Ask customers to describe recent moments when the product helped, what they were trying to do, and what would have happened otherwise. Ask what they would miss most if the product disappeared. Avoid naming the metric or leading them toward it. Capture their exact words, because the language matters when you compare it with the metric.

Step 5: Map customer language to the metric

Put each value story next to the metric's counted behavior and classify it: matched, value the metric misses, or counted behavior with little value. Note value moments that happen outside the product, which the playbook says should not be excluded just because they do not occur in the product. Look for patterns across segments. Count how often each pattern appears in your sample, without treating the counts as statistics.

Step 6: Validate North Star metric fit with data

Check the qualitative patterns against analytics, following the triangulation principle. If interviews suggest a quality bar matters, test whether a version of the metric with that bar tracks retention more closely. If a group gets value the metric misses, size it in the data. Write a verdict of confirm, refine or replace, with the evidence for each part.

Step 7: Share the verdict and update the definition

Present the verdict to the people who own the North Star and its inputs, with a few customer quotes that illustrate each finding. If the definition changes, update the dashboard, the input map and the documentation together. Record open questions for the next review. Schedule the next validation round.

Best Practices

• Decide the decision rule before the research. Writing down what would confirm or reject the metric in advance keeps the verdict honest.
• Interview at the edges. Customers just below the threshold and those who get value by unusual routes show what the metric misses.
• Use customers' own words. Their language often names the quality bar the metric needs, as "no issues" did for Happy Deliveries.
• Pair every qualitative finding with a quantitative check. Triangulation makes the verdict credible to people who distrust small samples.
• Look outside the product. Value moments such as a delivery arriving intact happen away from the interface and still belong in the North Star.
• Repeat on a schedule. A metric validated once can drift as the product and customer base change.

Common Mistakes

• Asking customers about the metric directly: People agree with plausible statements. Ask for stories of value and compare them with the metric yourself.
• Interviewing only power users: The most engaged customers confirm almost any engagement metric. Include customers near the threshold and those who churned.
• Treating interview counts as statistics: A pattern in a small sample is a lead to check. Check it in the data before changing the metric.
• Researching after the metric is locked: Validation after launch makes changes costly and politically hard. Validate candidates before adoption and existing metrics on a schedule.
• Ignoring value that happens off-product: A metric that only counts in-product events can miss the moment customers care about most. Look for proxies that capture it.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: North Star Metric

Related Skills

• Selecting Your North Star Metric
• Identifying and Mapping North Star Input Metrics
• Building a North Star Metric Dashboard
• Connecting Your North Star Metric to the Product Roadmap
• Aligning Cross-Functional Teams Around a North Star
• Evolving Your North Star Metric Across Growth Stages

Sources

• Amplitude: The North Star Playbook
• First Round Review: How Superhuman Built an Engine to Find Product/Market Fit
• Nielsen Norman Group: Triangulation

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.