Driver tree is a simple and effective tool to drive growth, find focus, and align teams in an organization. It is a map of how different metrics and levers fit together in an organization. Understanding the driver tree is key to driving a successful metric strategy for the organization that can lead to effective experiments.

Let’s consider an eCommerce website as an example with an overarching metric of revenue. This revenue can be increased by Level 1 factors that directly influence revenue.

Let us consider # transactions and avg transaction values as the two key influencing factors. These factors can in turn be impacted by Level 2 factors like price, number of users etc.

Building this driver tree can be a great way to understand the metrics that can contribute to your topline metric. This helps various teams align on success metrics of the experiment that can help move the needle. A data-driven product team can use this mapping to identify where strategic initiatives need to be implemented for the highest impact, thus facilitating prioritization.

An experiment often needs multiple metrics to quantify the impact on the overall product and user experience. Often, there are three major types of metrics we can set in an experiment:

Also known as Primary Metric or Success Metric, it is aligned with the overall company long-term vision and mission. It is a small set of metrics that the company cares about and aims to achieve.

Examples of goal metrics can be revenue, adoption, engagement, or performance. These metrics are typically stable over long periods of time and are lagging in nature. They are also easy to communicate to investors, cross-functional teams, and employees.

Instead of using goal metrics, experiments use driver metrics to evaluate success of experiments. These are surrogate or indirect metrics which are leading indicators of product success.

Examples - teams working on app performance or website engagement might not be able to use revenue as their goal metric. But they should be able to use another metric like latency or daily active users, which can contribute in driving higher revenue in the longer-run. It is important to ensure the driver metric chosen directly drives the goal metrics of the organization.

These metrics ensure that any new feature or product change does not impact the company negatively. These metrics act as a sanity check of the experiment. If a guardrail metric is negatively impacted, it can serve as a red flag highlighting either an experiment assumption that has been violated or some aspects of the experiment not going as planned.

These metrics help us understand why some important metrics, especially the goal metrics, move or do not move, and how to interpret their movements. Diagnostic metrics are typically not a part of the overall optimization strategy but help diagnose the impact of an experiment. They also confirm the assumptions being made for feature changes.

Example - embedding videos on a web page should increase the page latency. Instead of having an clear directional interpretation, it is more important for a good debugging metric to be sensitive so it moves with the experimental changes.

Most experiments require a combination of different types of metrics to design a robust experiment. Often hundreds of metrics are observed and computed for a test. Only a handful of those are actually used to drive the experiment decision. However, the other metrics are needed to validate our assumptions, assess impact on other features, and ensure there’s no detrimental effect on the users. When all these metrics are used in synergy, we can be confident that we are making the right decisions and our product changes add real value to the users, thus impacting their behavior positively.

Designing an experiment often involves optimizing multiple metrics together. This is where the concept of Overall Evaluation Criteria (OEC) comes into play. An OEC is a combination of various weighted metrics together, which serves as a single metric that can be used for an overall decision-making.

Having a single metric focused approach for running experiments isn’t recommended. But having too many metrics in an OEC can also make the decision process difficult. This is because metrics can often move in different directions and the overall impact of the experiment might be hard to judge. Hence, combining 3-5 metrics in an OEC is typically a good approach.

Online experiments are a gold standard for causality and are used by data-driven companies to understand the impact of their products on their users. In this blog, we present how metric strategy ties closely with the design of experiments. We also learnt different types of metrics like goal metrics, driver metrics, guardrail metrics, and diagnostic metrics. Finally, we looked at how multiple metrics can be combined in an OEC to drive decision-making.

<Read Part 1