Metrics · Product Analytics · Goodhart's Law · Causality · PTOS

Metrics: How Not to Lie to Yourself with Numbers and Measure Reality

We analyze why metrics can create false confidence and how to distinguish proxy metrics from business metrics to manage the product, not just the 'dashboard weather'.

Metrics: How Not to Lie to Yourself with Numbers, but to Measure Reality

In product development, metrics are our way of connecting with reality. But very often, they turn into a tool for self-deception. We look at beautiful charts going up and think we are doing everything right, while the product slowly dies.

To avoid this, we need to learn to distinguish "good" metrics from "bad" ones and always remember the main principle: a metric must reflect a real change in user behavior, not just our activity.

The Main Question for Any Metric

If the user performed this action, did they really get value?

If the answer is "not sure" or "not always," then you are likely measuring noise, not value.

Value-Event: The Heart of Your Metric System

At the core of any healthy metric system is a Value-Event.

A Value-Event is the minimum, measurable, repeatable user action, after which one can honestly say: "Yes, the product delivered on its promise."

Examples of Value-Events:

• Duolingo: Lesson completed.
• Spotify: Track listened to for ≥ 30 seconds.
• Notion: Document created and saved.
• Marketplace: Message sent to seller.

NOT Value-Events:

• Registration.
• Onboarding completion.
• App opened.
• Clicking the "Start" button.

These are all infrastructure, not value. A user registers not to register, but to solve their problem.

The Metric Ladder: From Proxy to Money

Not all metrics are equally useful. They can be represented as a ladder, where the bottom rung holds the fastest, but most "deceptive" metrics, and the top rung holds the most honest, but slowest ones.

1. Proxy Metrics: These are activity metrics that indirectly may indicate value.

   • Examples: DAU/MAU, time in app, number of clicks.
   • Problem: They are easily "gamed" and often have no connection to real value (see Goodhart's Law). A user might spend a lot of time in the product because your interface is confusing.

2. Value Metrics: These are metrics built around the value-event.

   • Examples: Number of users who performed a value-event; frequency of value-event per user.
   • Strength: They more honestly reflect whether users are getting real benefit.

3. Business Metrics: These are metrics directly tied to money.

   • Examples: Revenue, Customer Lifetime Value (LTV), Customer Acquisition Cost (CAC), Retention.
   • Problem: They are "lagging indicators." You might only see retention drop several months after a failed release.

How to Build a Healthy System?

1. Define your Value-Event. This is the core of your system.
2. Build a hierarchy. Link your proxy metrics to the value-event, and the value-event to business metrics. You need to understand how the growth of one metric affects others.
3. Use Guardrails. Every target metric should have a "guardrail." If you optimize activation, make sure retention doesn't drop. If you're working on retention, make sure the support load doesn't increase.
4. Focus on causality. Don't just observe correlation; prove causality through experiments (e.g., A/B tests).

Remember, numbers don't lie. We lie to ourselves when we choose the wrong numbers or interpret them in a way that suits us. Start by honestly defining value—and your metric system will transform from a distorted mirror into an accurate compass.