Most ideas lose when tested. Name the riskiest belief, decide how strong the evidence must be, then pick the cheapest of 55 product experiments that gets you there.
Product experiments are small, planned tests of one belief your plan depends on, with a pass signal written before you start. To choose one, name the belief, name its risk (desirability, usability, feasibility or viability), decide how strong the evidence must be, then pick the cheapest method that reaches that strength. This guide sorts 55 methods that way, with time and effort for each.
Key takeaways
- Most ideas lose when tested. Only one third of the ideas tested at Microsoft improved the metric they were designed to improve, and about 10% of Google's controlled experiments led to business changes (Kohavi and Longbotham, 2023).
- Test the riskiest belief, not the whole idea. Marty Cagan's four risks are value, usability, feasibility and business viability (SVPG, 2017).
- Evidence comes in strengths. What people say is weakest; money or a signature is strongest. Strategyzer's library for Testing Business Ideas rates 44 experiments from 1 to 5 on cost, setup time, run time and evidence strength.
- Cheap does not mean weak. In that library, a concierge test scores 1 of 5 for cost and 5 of 5 for evidence.
- Write the pass signal before you start. Moving it afterwards is how most teams fool themselves.
Why do product ideas need experiments at all?
Most product ideas do not fail because the team built badly. They fail on a guess nobody checked.
The record from teams that measure is humbling. Ron Kohavi and Roger Longbotham collect it in one place: a third of ideas tested at Microsoft moved their target metric, Jim Manzi wrote that about 10% of Google's experiments led to business changes, and Netflix is reported to consider 90% of what it tries to be wrong (Kohavi and Longbotham, 2023). These companies employ excellent product people. Their first guesses still lose most of the time.
Skipping the check costs more later. CB Insights studied 431 venture-backed companies that shut down since 2023 and found reasons for 385 of them: 70% ran out of capital, 43% had poor product-market fit, 29% cited bad timing and 19% had unsustainable unit economics (CB Insights, 2026). Companies could give more than one reason. Poor fit and broken unit economics are exactly the doubts a cheap experiment can probe months before the money runs out.
A product experiment is how you check a guess before you spend real money on it. You do not need a research team, and you do not need all 55 methods in this guide. You need the one test that could change your mind about the belief your plan depends on most.
What is a product experiment?
A product experiment is a small, planned test of one belief, with a result you define before you start. Good product experiments have four parts:
- The belief, written as a sentence that could be false. "Freelancers lose work because follow-ups slip," not "freelancers need this."
- The method: the smallest action that produces evidence about that belief.
- The measure: one number, plus one thing that must not get worse.
- The pass signal: the result that would make you say yes, set in advance.
Teresa Torres puts the core move plainly: stop testing whole ideas and test the assumptions they depend on (Product Talk, 2023). An idea like "a reminder app for vets" holds a dozen beliefs. Most are safe. One or two would kill it, and those are the ones worth an experiment. If you are starting from a feature request, the product problem statement template turns it into a belief you can test.
What are the four product risks an experiment can test?
Every belief carries a kind of risk, and the kind decides which methods fit. Marty Cagan of Silicon Valley Product Group names four (SVPG, 2017):
| Risk | Cagan's question | Strategyzer's word | Usual owner | Cheap first test | Strong test |
|---|---|---|---|---|---|
| Value | Will customers buy it, or choose to use it? | Desirability | Product manager | Customer interview, fake door | Concierge, letter of intent |
| Usability | Can users figure out how to use it? | Tagged as desirability | Product designer | Paper prototype | Usability test on a real build |
| Feasibility | Can engineers build it with the time, skills and technology we have? | Feasibility | Lead engineer | Expert interview, technical spike | Wizard of Oz, mash-up |
| Business viability | Does it work for the rest of the business? | Viability | Product manager | Pricing survey, market research | Funnel test, letter of intent |
Bland and Osterwalder's Testing Business Ideas (Wiley, November 2019, 368 pages) groups risk into three questions: do customers want it, can we build it, and will it earn sustainable revenue (Wiley; Strategyzer, 2025). Its experiment library tags usability-style tests, such as card sorting and clickable prototypes, as desirability. Cagan keeps usability separate, and so do we, because "they want it" and "they can use it" fail in different ways and need different tests.
Torres adds a fifth category, ethical: should we build this at all? Cagan's advice on order is blunt: tackle the big risks early, especially value and business viability. For a deeper look at what counts as proof for each risk, read what counts as evidence for the four product risks.
How strong is the evidence from each kind of experiment?
The higher the step, the more you can trust the result.
Not all evidence is equal. A kind word and a signed letter are both "positive feedback," but only one should move your roadmap. Alex Osterwalder makes the same point: what customers say and what they do are two different things, so interviews and surveys give valuable but weak evidence, while a simulated sale gives strong evidence (Strategyzer, 2017).
| Strength | What it means | Example | Treat it as |
|---|---|---|---|
| What people say | Someone told you something | An interview, a survey | A reason to run a stronger test |
| Early signal | Real behaviour, but cheap or indirect | An ad click, a search trend | A clue about where to look |
| What people do | Someone took a real action | Finishing a task, leaving an email | Good evidence for that specific action |
| Money or a signature | Someone gave up something of value | A payment, a signed letter | The strongest evidence you can get before launch |
Specky uses the same ladder. A belief can only be marked Supported by what people do, or by money or a signature. Anything weaker stays an Early signal, however good it sounded.
Does a stronger test always cost more?
No. Strategyzer's experiment library poster rates every method from 1 to 5, and cost and evidence strength move independently (Strategyzer experiment library). A sample:
| Method | Cost (1-5) | Evidence strength (1-5) | What the pairing tells you |
|---|---|---|---|
| Customer interview | 2 | 1 | Cheap and fast, but only words |
| Discovery survey | 2 | 1 | Counts a problem; proves no demand |
| Paper prototype | 1 | 1 | The cheapest rung for a flow |
| Search trend analysis | 1 | 3 | Strangers' behaviour for almost nothing |
| 404 test | 1 | 3 | Also rated 1 for setup and run time |
| Concierge | 1 | 5 | Low cost, top evidence, but slow to run |
| Wizard of Oz | 2 | 5 | Real service, faked automation |
| Extreme programming spike | 2 | 5 | Strong evidence on feasibility only |
| Presale | 3 | 5 | Money on the table |
| Single-feature MVP | 4 | 5 | Strong, but you are now building |
The lesson: the expensive part of a strong test is usually your time, not your budget. A concierge test costs little money but weeks of manual work.
How do you choose the right product experiment?
Answer five questions, in order:
- What is the belief? Write it as one sentence that could turn out to be false.
- What kind of risk is it? Do they want it (desirability)? Can they use it (usability)? Can we do it (feasibility)? Will it make money (viability)?
- What would it cost to be wrong? A guess that decides the next six months deserves stronger evidence than one that decides a button colour. Barry O'Reilly makes the same point about evidence thresholds: a rocket launch needs a higher bar than a sign-up flow (Thoughtworks, 2014).
- What is the cheapest test that gives enough evidence? Start low on the ladder when you know little. Climb as the stakes rise.
- What would count as a pass? Write it down before you begin. O'Reilly's reason is the right one: signals stated in advance reduce biased readings of the result.
A rule of thumb we use: if being wrong costs a week, what people say is enough. If it costs a quarter, get what people do. If it costs a hire, a year or your runway, get money or a signature.
When you have several beliefs and cannot tell which is riskiest, map them first. Our guide to assumption mapping and the free assumption map tool sort beliefs by impact and evidence, so the test goes where it matters.
Product experiment decision table: which test for which risk and evidence level?
Find the row for your risk. Move right until you reach the evidence strength that being wrong deserves. Pick the cheapest method in that cell.
| Risk | What people say | Early signal | What people do | Money or a signature |
|---|---|---|---|---|
| Desirability: is the problem real? | Customer interview, discovery survey, forum research | Search trends, support tickets, analytics check | Diary study | Move on to the offer row |
| Desirability: do they want this offer? | Product box, brochure, validation survey | Online ad, 404 test, email campaign | Fake door, landing page, feature stub | Concierge, crowdfunding |
| Usability: can they use it? | Rarely useful: people misjudge their own ease of use | Web traffic analysis | Paper prototype, clickable prototype, usability test, card sorting | Not the right question; usability is shown by doing |
| Feasibility: can we deliver it? | Partner and supplier interviews, expert interviews | Technical spike | Pretend to own, mash-up, Wizard of Oz | Concierge (paid delivery by hand) |
| Viability: will it make money? | Pricing survey, sales team feedback, investor feedback | Online market research, presale interest | Funnel test | Letter of intent, referral program, pop-up store |
| Typical time | 1 day to 2 weeks | 1 day to 1 week | 2 days to 3 weeks | 2 to 4 weeks |
| Typical effort | Low | Low | Low to high | Medium to high |
Times and effort come from the method tables below and assume a small team. Read across a row to see how to climb. Read down a column to compare risks at the same strength.
Which product experiments answer which question? The full library of 55
Every method below links to its own guide, with a copyable template, a pass signal set in advance and a worked example.
Desirability: is the problem real, and who has it?
Start here when you are still unsure the pain exists. Most of these cost a few days and no code.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Customer interview | What people say | 1 week | low | You have a hunch and need the real story |
| A day in the life | What people say | 1 week | medium | People cannot describe work they do on autopilot |
| Discovery survey | What people say | 1 week | low | You want to know how common a problem is in one group |
| Survey | What people say | 1 week | low | You know the group and just need to settle one choice |
| Discussion forums | What people say | 2 days | low | You want people's exact words without taking their time |
| Storyboard | What people say | 2 days | low | The situation is hard to explain in words |
| Speed boat | What people say | 1 day | low | You have a product and want polite people to criticise it |
| Search trend analysis | Early signal | 1 day | low | You want a one-day read on whether people look for this |
| Web traffic analysis | Early signal | 1 day | low | You have steady visits and need to pick the step to test |
| Customer support analysis | Early signal | 2 days | low | You have a live product and a full inbox |
| Analytics | Early signal | 1 day | low | The behaviour is already logged |
| Diary study | What people do | 2 weeks | medium | The problem comes and goes, and people misremember it |
Once you have run a handful of interviews, analyse them with a synthesis template before you count anything.
Desirability: will people want this offer?
Use these once the problem looks real. Now you are finding out if your particular answer gets a reaction.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Product box | What people say | 1 day | low | Your team cannot agree on the main benefit |
| Brochure | What people say | 3 days | low | You need to check your offer is clear |
| Validation survey | What people say | 1 week | low | People have seen the offer and you want to hear what would stop them |
| Online ad | Early signal | 1 week | low | You want a reaction from strangers |
| Link tracking | Early signal | 1 week | low | You are comparing messages or channels |
| 404 test | Early signal | 3 days | low | You want the cheapest possible check on curiosity |
| Email campaign | Early signal | 3 days | low | You have a list that chose to hear from you |
| Social media campaign | Early signal | 1 week | low | Your customers gather somewhere you are welcome |
| Data sheet | Early signal | 1 week | low | Your buyers evaluate on numbers |
| Explainer video | Early signal | 1 week | medium | It is easier to show than to describe |
| Fake door | What people do | 1 week | low | You can state the offer in one sentence |
| Feature stub | What people do | 1 week | medium | You want to catch current users in the moment |
| Simple landing page | What people do | 1 week | low | You can send strangers to one clear page |
| Boomerang (competitor testing) | What people do | 1 week | low | Good alternatives already exist |
| Buy a feature | What people do | 2 days | low | Customers say everything matters |
| Split test | What people do | 2 weeks | medium | You have the traffic to prove a change caused a result |
Usability: can people use it, and will they come back?
These put something close to the real thing in front of people. Ease of use and repeat use are separate questions, so read each row carefully.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Paper prototype | What people do | 2 days | low | You are still working out the steps |
| Clickable prototype | What people do | 1 week | medium | The flow makes sense and you want to test it for real |
| Usability test | What people do | 3 days | low | Something is built and people are getting stuck |
| Card sorting | What people do | 3 days | low | You are deciding how to name or organise things |
| 3D print | What people do | 2 weeks | high | It is a thing people hold, wear, or mount |
| Life-sized prototype | What people do | 3 weeks | high | Size and space change the experience |
| Single-feature MVP | What people do | 3 weeks | high | Lighter tests passed and you need to see repeat use |
| Concierge | Money or a signature | 3 weeks | medium | You can deliver the outcome by hand and ask for money |
Feasibility: can we build or deliver it?
Some ideas are loved and still impossible. Use these when the doubt is on your side of the table.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Partner and supplier interviews | What people say | 1 week | low | Your plan depends on someone outside your company |
| Expert stakeholder interviews | What people say | 1 week | low | Someone could block or fund this |
| Extreme programming spike | Early signal | 3 days | medium | One technical unknown could sink the plan |
| Pretend to own | What people do | 1 week | medium | You want to learn what one real delivery takes |
| Mash-up | What people do | 2 weeks | medium | Existing tools could do the job if connected |
| Wizard of Oz | What people do | 2 weeks | medium | The hard part is automating it, and the experience needs testing first |
Viability: will it make money?
Interest is cheap. These tests get closer to the moment someone gives up something of value.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Sales force feedback | What people say | 3 days | low | Deals stall and nobody agrees why |
| Investor interviews and pitches | What people say | 2 weeks | medium | You want your advantage stress-tested |
| Pricing survey | What people say | 1 week | low | You have no idea what to charge |
| Mock purchase intent | What people say | 1 week | low | You want to compare two prices |
| Online market research | Early signal | 3 days | low | You need to know if the sums could ever work |
| Presale interest | Early signal | 1 week | low | You want to see if interest survives a price |
| Funnel test | What people do | 2 weeks | medium | You need to know what a customer costs to reach |
| Referral program | Money or a signature | 2 weeks | medium | You want to know if users will vouch for you |
| Letter of intent | Money or a signature | 4 weeks | medium | You sell to businesses and can reach who signs |
| Crowdfunding | Money or a signature | 4 weeks | high | It is a consumer product and you can deliver |
| Pop-up store | Money or a signature | 2 weeks | high | People need to see or touch it before buying |
Not sure where to start?
If the idea is still a blur, or nothing here fits, begin with one of these.
| Test | Evidence strength | Time | Effort | Reach for it when |
|---|---|---|---|---|
| Lean Canvas | What people say | 1 day | low | The idea is unstructured, or your team disagrees |
| Other | Early signal | 1 week | low | None of the 54 methods above fits |
What does climbing the evidence ladder look like in practice?
This sequence is made up to show the idea. Imagine you think independent gym owners lose members because they do not notice who has stopped coming.
- Five customer interviews confirm owners have tried to fix this. That is what people say, and it costs a week.
- A fake door shows owners will leave an email for an offer. Say 40 of 400 visitors from a gym-owners' newsletter do, which is 10%. Now you have behaviour.
- A concierge test sends weekly lists by hand to five gyms, and three pay for a second month. Now you have commitment.
- Only now do you build, starting with a single-feature MVP. Brief your coding agent with the AI coding agent PRD template, and record why you went ahead in a product decision log.
Using the times in the tables, steps 1 to 3 take about five weeks: one for interviews, one for the fake door, three for the concierge. Each step cost more than the last, and each one was earned by the one before it. If any had failed, you would have stopped early and saved everything after it.
Notice what the sequence skipped. Nobody ran a survey to ask whether gym owners "would use" a churn tool, and nobody built a dashboard to see if it got traffic. Each test answered the next riskiest question and nothing more. For where this sits in a full discovery cycle, see the product discovery process guide.
How many product experiments and participants do you need?
There is no fixed number of experiments. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. Strategyzer's summary of the book suggests a practical cadence: run three to five small experiments on your highest-priority assumptions in the first month, and start with the smallest test that could confirm or kill each one (Strategyzer, 2025).
Pair weak and strong. One "say" test tells you what to look for; one "do" test tells you whether it holds. Two "say" tests that agree are still two opinions.
Participant numbers depend on the method:
- Usability tests: five people per round. Jakob Nielsen's model found five users uncover about 85% of usability problems and 15 find nearly all, so he advises spending the budget on several small rounds (Nielsen Norman Group, 2000).
- Surveys: aim for about 50 valid answers from one named segment before you trust a percentage. Our discovery survey guide explains why.
- Split tests: these need real traffic. Kohavi and Longbotham note that a 50/50 split gives the most statistical power once a short ramp-up has ruled out obvious errors. If you lack the visitors, use a fake door or landing page instead. Our guide to A/B testing covers the maths.
- Letters of intent and presales: a handful can be enough when each one is a real buyer with a budget.
What are the most common mistakes with product experiments?
- Testing the idea instead of the riskiest belief. "Will this work?" cannot fail cleanly. "Will owners pay €50 a month for a weekly list?" can.
- Picking the method you already know. Teams that like surveys survey everything. Let the risk choose the method.
- Treating what people say as what they do. Twenty enthusiastic interviews are still the bottom rung.
- Writing the pass signal afterwards. A result you judge after seeing it is a story, not a test.
- Split-testing without traffic. A few hundred visits rarely prove that a change caused a result.
- Skipping the unglamorous risks. Founders love desirability tests. Cagan's advice is to tackle value and business viability early, and feasibility can sink a loved idea just as fast.
- Climbing too fast. A crowdfunding campaign before a single interview spends weeks to learn what a conversation could have told you.
When should you not run a product experiment?
Skip the experiment when the decision is cheap and easy to reverse; shipping and watching is the test. Our piece on reversible and irreversible product decisions helps you tell the two apart. Skip it, too, when you will build regardless of the result. A test you would ignore is theatre. And when none of the 55 methods fits your setting, such as a year-long public-sector sale, design a custom experiment rather than forcing one.
Is it ethical to test something that does not exist yet?
Several methods, including the fake door, the 404 test, presales and the Wizard of Oz, show people something that is not fully real. Keep them honest:
- Tell people on the next screen that the feature is not ready, and thank them.
- Never take money you cannot refund or deliver on. Say when delivery would happen.
- If a person secretly does the work in a Wizard of Oz test, tell participants afterwards when it would matter to them.
- Ask before recording anyone, and only keep the email addresses people chose to give you, for the purpose you stated.
How to run this in Specky
Fill in a Lean Canvas on the Assumptions page, and Specky drafts the assumptions behind it. Each one appears on a board by impact against evidence, with the riskiest numbered. Click a note, pick a test, and the plan fills in with steps, a pass signal, and a time to run.
Who does the work depends on the method. Some tests are AI interviews in Vouch, at vouch.specky.space. For some, Specky builds a page and collects responses. For some, it reads your connected analytics. For the rest, you run the test and Specky keeps the plan and the result. Each guide above says which.
Recording a result moves the note and updates any bet resting on it. One contradicted assumption marks the whole bet as contradicted. A bet only counts as supported when every assumption under it is.
Frequently asked questions
What is a product experiment?
A product experiment is a small, planned test of one belief about a product, with a pass signal defined before you start. It lets a team learn whether something is true before spending real money on it. A good one names the belief, the method, the number it will measure and the result that would count as a yes.
How do I choose the right product experiment?
Write the belief you are least sure about, name its risk (desirability, usability, feasibility or viability), and decide how strong the evidence must be given what being wrong would cost. Then pick the cheapest method that reaches that strength. Early on, conversations are usually enough. As stakes rise, ask people to act or pay.
What are the four types of product risk?
Marty Cagan's four risks are value (will they buy or use it?), usability (can they figure it out?), feasibility (can we build it with our time, skills and technology?) and business viability (does it work for the rest of the business?). Strategyzer's Testing Business Ideas uses three, desirability, feasibility and viability, and its library tags usability tests as desirability.
What is the strongest evidence in product validation?
Money or a signature is the strongest evidence, such as a payment, a pledge or a signed letter of intent. It beats what people do, which beats what people say. Osterwalder's rule of thumb is that evidence gets stronger the closer it comes to a real purchase, which is why a simulated sale outranks any interview or survey.
Should I start with customer interviews or a landing page?
Start with interviews when you are still unsure the problem is real, because you need people's own words before you can write an offer. Start with a landing page when the problem is clear and you can state your offer in one sentence. Many teams run both inside two weeks: interviews first, then a page built from what they heard.
How many experiments should I run before building?
There is no fixed number. Keep testing until the belief your plan depends on most has evidence strong enough for what being wrong would cost. As a cadence, Strategyzer's book summary suggests three to five small experiments on your highest-priority assumptions in the first month. Stop early when one fails clearly.
Sources
- Silicon Valley Product Group — The Four Big Risks (Marty Cagan, 2017)
- Strategyzer — Testing Business Ideas Experiment Library (poster)
- Wiley — Testing Business Ideas: A Field Guide for Rapid Experimentation (Bland and Osterwalder, 2019)
- Strategyzer — Testing Business Ideas: Book Summary and Key Takeaways (2025)
- Strategyzer — How Strong Is Your Innovation Evidence? (Alex Osterwalder, 2017)
- Kohavi and Longbotham — Online Controlled Experiments and A/B Tests, Encyclopedia of Machine Learning and Data Science (2023)
- CB Insights — Top reasons startups fail (report, March 2026)
- Product Talk — Assumption Testing: Everything You Need to Know to Get Started (Teresa Torres, 2023)
- Thoughtworks — How to Implement Hypothesis-Driven Development (Barry O'Reilly, 2014)
- Nielsen Norman Group — Why You Only Need to Test with 5 Users (2000)
This story was originally published on 9 November 2026 at https://www.specky.space/blog/how-to-choose-a-product-experiment.
Fathy Shalaby is a scientist by training (master's). He worked at LexisNexis and Canva, and writes How to Be a Product Engineer.
