Experiments · Mistakes · Anti-Patterns · Goodhart's Law · PTOS

Dangers and Anti-Patterns in Experiments: How Product Managers Deceive Themselves and Kill the Product

We examine common pitfalls and mistakes in conducting experiments—from lacking result-based decisions to incorrect sampling and missing guardrail metrics.

Dangers and Anti-Patterns in Experiments: How We Lie to Ourselves, Even When Using A/B Tests

Experiments are the foundation of an Outcome-Driven approach. They allow us to receive honest signals from reality and make data-driven decisions. But the mere presence of experiments does not guarantee success. Teams often fall into traps that turn A/B tests and other validations into a theater of self-deception.

Here are the most common anti-patterns that kill the value of experiments.

1. Experiment Without a Result-Based Decision

• What it looks like: The team launches a test, gets results, looks at them, and... nothing happens. The discussion ends with the phrase "interesting, let's think about it."
• Why it's dangerous: If it's not defined beforehand what decision will be made for what result (Go/No-go/Reframe), the experiment turns into a collection of arguments for "why we should continue." This is the Feature Factory in a lab coat.
• Antidote: Before starting any test, define success/failure thresholds and specific actions for each outcome in the Launch Brief or "One-Page Evaluation Sheet."

2. Choosing Metrics and Thresholds After Launch

• What it looks like: "Let's see what grew... Oh, this metric grew a bit, let's make that the target metric!"
• Why it's dangerous: This is not analytics. It's "hindsight advocacy." You are simply looking for justification for work already done.
• Antidote: The target metric, guardrails, and success/failure thresholds must be defined and documented before the experiment begins.

3. Substituting Value with a Click

• What it looks like: The team launches a fake-door test, sees a high CTR, and concludes: "Users really need this feature!"
• Why it's dangerous: A click is interest, not value. Users may click out of curiosity, but this says nothing about their willingness to pay, integrate the solution into their workflow, or use it regularly.
• Antidote: Use the "Ladder of Proof." Understand what level of signal your test provides, and do not mistake a weak signal for irrefutable proof.

4. Incorrect Sampling

• What it looks like: The team shows a new B2B feature to everyone, including freelancers and students, and then wonders why there's "no interest." Or, conversely, they show it only to the most loyal "fans" and get inflated, unrepresentative results.
• Why it's dangerous: You get beautiful but useless garbage instead of a signal.
• Antidote: Clearly define the target segment for the experiment. Ensure these users have the problem you are solving.

5. Missing Guardrail Metrics

• What it looks like: The team optimizes one metric, for example, conversion to registration. They simplify the form, remove complex steps, and... achieve growth! But they don't notice that the retention of new users has halved because irrelevant users started entering the product.
• Why it's dangerous: You win the battle but lose the war. Improving one metric can come at the cost of the product's overall health. This is a classic example of Goodhart's Law.
• Antidote: Every target metric (North Star) must have a "guardrail." Ask yourself: "How can we 'improve' our metric while simultaneously harming the product?" The answer to this question will help you define your guardrails.

6. "Pilot for the Sake of Pilot"

• What it looks like: A pilot project is launched to test a new idea. It has no clear deadlines, success criteria, or result-based decisions. It just "runs."
• Why it's dangerous: Such a pilot never ends. It exists to avoid making the difficult "yes" or "no" decision.
• Antidote: Any pilot is an experiment. It must have its own "One-Page Evaluation Sheet" with goals, thresholds, and a decision.

Experiments are a powerful tool, but only in the hands of those who approach them with discipline and honesty. Avoid these anti-patterns, and your tests will begin to yield not just numbers, but real knowledge and growth.