Metrics · Checklist · Self-deception · Data Reliability · PTOS

'Are We Lying to Ourselves?' Checklist: 7 Questions to Verify the Reliability of Your Product Metrics

A 7-question combat protocol that allows you to quickly check if the team is deceiving itself with numbers and ensure the reliability and transparency of metric-based conclusions.

'Are We Lying to Ourselves?' Checklist: 7 Questions to Verify the Reliability of Your Product Metrics

Metrics should be a tool that makes reality visible, not a means of self-deception. Unfortunately, numbers can easily be misleading if you don't ask the right questions. The main one is: 'How will we know we aren't lying to ourselves with these numbers?'.

Here is a 7-question combat protocol that will help you quickly verify the reliability of your product metrics and ensure you are making decisions based on truthful data, not beautiful but empty charts.

1. Is This Metric About Value or Activity?

This is the main filter that weeds out 'vanity metrics.' Views, clicks, registrations—all are activity. They are nice, but they don't guarantee that the user has received real value.

• Ask yourself: Does this metric measure a value-event — a specific action after which you can honestly say, 'Yes, the product has delivered on its promise'?
• Example: Instead of 'number of app opens' for a fitness tracker, measure 'number of completed workouts'.

2. Is This Metric a Leading Indicator for the Business or Just 'Nearby'?

A good product metric should be a leading indicator of future business results (revenue, retention). It should predict success, not just correlate with it.

• Ask yourself: Is there historical data proving that the growth of this proxy metric preceded the growth of business metrics? Can you explain the mechanism, why this behavior should lead to money?
• Example: An increase in the 'number of messages sent in a team chat' (proxy) should lead to a decrease in the churn of paid teams (business result).

3. Is There an Obvious Way to 'Game' It? (Goodhart's Law)

'When a measure becomes a target, it ceases to be a good measure.' Any metric under pressure will be distorted. Your job is to anticipate this.

• Ask yourself: Can you think of 3-5 ways to 'game' this metric in 5 minutes while making the product worse? (e.g., spam notifications, dark patterns, artificially breaking down actions).
• If yes, your metric is dangerous without safeguards.

4. Are We Looking at Cohorts/Segments or Just the 'Overall Average'?

'Average' numbers often hide the truth. Growth might be driven by one segment while another is churning en masse.

• Ask yourself: How does this metric behave for different user groups (new vs. old, by different acquisition channels, on different pricing plans)?
• Example: 'Average retention of 20%' might hide that the retention of new users is only 5%, while that of old users is 80%.

5. Does This Metric Have 'Guardrails'?

Never use a single metric in a vacuum. Any 'improvement' can have negative side effects.

• Ask yourself: What guardrail metric will show us that we haven't broken something important?
• Example: If you are optimizing for 'conversion to purchase,' your guardrails could be 'number of support tickets' and 'refund rate.' Growth in the main metric at the cost of the guardrails is a bad sign.

6. Do We Trust the Data and Its Interpretation?

Even the best metric is useless if the data for it is collected incorrectly or everyone interprets it differently.

• Ask yourself: Does this metric have a 'passport' (a clear definition, measurement window, segment)? How confident are we in the data quality? Can we explain why the metric increased or decreased (debuggability)?
• Example: What exactly do we consider an 'active user'? Opening the app? Viewing a screen? Performing a value-event?

7. Is There a Predefined Decision for Thresholds?

Metrics are not for observation, but for making decisions.

• Ask yourself: What will we do if the metric reaches the success threshold? The failure threshold? The 'gray area'?
• If there's no answer, you're not doing analytics; you're just collecting numbers.

By running your key metrics through this checklist, you can build a system that helps make informed decisions and protects you and your team from self-deception.