When a measure becomes a target, it ceases to be a good measure
- Charles Goodhart

The adage above is called "Goodhart's Law." Goodhart is an economist, and the law's context was initially based on economic policies. However, we can apply this law to almost any system, big or small.

Goodhart's law describes scenarios where the way of evaluating a system becomes the target itself. Goodhart's law applies in many cases. If a hospital solely tries to reduce waiting times, they might see reduced quality in patient care. If a school prioritizes high grades in standardized tests, it might observe a deficiency in the completeness of students' knowledge and development. If a business only focuses on driving up the number of sales, it might witness significant churn if customer satisfaction is not accounted for. In the Wells Fargo scandal, employees opened millions of fake user accounts to meet sales targets. This occurred between 2002 and 2016. There are other real-life examples where the measure of success for a system solely becomes the optimization objective, thereby causing unintended consequences.

I like to think that most things can be framed as systems, and almost any system starts with an intentional goal. However, by solely optimizing for this goal, the system may begin to experience unplanned surprises. 

Let's consider another example of an R&D corporation that only employs PhD-level scientists. If the corporation's goal was to hire the best science talent and optimize for this by only hiring PhD scientists, would there be any bad consequences? While there might be a high correlation between having a PhD and being a talented scientist, does having a PhD make one a talented scientist? Goodhart was trying to describe these types of relations between measures and targets. These kinds of questions are necessary when designing and evaluating the systems we use to achieve certain goals.

For systems and processes I work with, I often ask myself: How do we design good evaluations? How do we ensure that the metrics we care about are true indicators of success or failure in our system? How do we know we aren't fooling ourselves? How do we design a system that is adaptive to metric drifts? I am describing 'metric drift' as a scenario where the metrics we track are no longer good measures of the target.

I work in a computational drug discovery lab that builds machine learning models for discovery campaigns based on the drug target of interest. These models achieve very high accuracy during testing, but we've found that they have a high failure rate in real life, i.e., the number of times they incorrectly predict the wrong molecule as a potential drug lead. I often think about why this happens, especially since the models are properly evaluated and trained on carefully curated data. It could be that the metrics we use to assess these models have become what we primarily optimize, and we deviate from the main goal of "success in real life." We’ve also hypothesized that the quality of chemical data makes it hard to model the space of drug-to-activity.

These days I'm more invested in improving closed-loop experimentation and self-driving labs. I use sequential design methods like Bayesian optimization to guide and optimize chemical and biological processes or products. This is another place where Goodhart's Law shows up. Some processes have many inputs that map to two or more outputs. In the case of biopharmaceuticals, the goal is to increase both process output and product quality. It would be a waste if we optimized the yield of the process and the purity or bioactivity of the therapeutic is low. It is also not ideal to have a low process yield. This makes the process expensive and less scalable. Since the goal is to improve overall process productivity, if we only focus on yield or purity as the proxy for success, we might experience undesirable outcomes. This brings me to talking about what I mean by a "good system."

There's a saying that 'the purpose of a system is what it does.' While many people might agree with that statement, there are certain cases where we have desired outcomes, and it would be unwise to settle for emergent properties that are observed. If we define a good system as one where we have some form of control and it functions correctly as desired, then we need to figure out ways to gain this control.

Goodhart’s Law gives us a warning but doesn’t necessarily tell us how to mitigate the effects of what it warns about. To account for what the law states, we must acknowledge several key aspects of the system, including metrics, variations, and correlations.

Metrics: The metrics you track, as a measure of your system's performance or productivity, contribute to whether or not you will be sidetracked. Also, overreliance on a single metric often causes failures. Metrics for systems that interface with people or real-life events are usually gamed. This is also the case for AI agents. An AI agent tasked with increasing the amount of money in a portfolio might resort to actions like deforestation to print more money. That is a popular sentiment among researchers in AI ethics. These types of failures occur when we use a single metric to assess the success of a system. Furthermore, most systems have metrics that are difficult to measure, and we sometimes have to rely on proxy metrics, which are other variables that can adequately describe the hard-to-measure metric. I have observed that this is either an easy or hard problem and can depend on the system. These proxy metrics can sometimes be misleading.

Variations: These can tell us if there are drifts in the system or the metric. To build adaptive systems, you need to track variations. Variations can help you identify if a metric or variable properly accounts for the main goal, and they are also good indicators of anomalies and interesting events

Correlations: Every sufficiently educated person knows the saying, 'Correlation is not causation.' I've learned that problems often arise when you try to optimize for a goal using variables or metrics merely correlated with it. The problem is that correlation alone does not account for other variables. In addition, the correlating variable can become decorrelated in conditions that weren't accounted for during design. Some types of low or high correlation only apply under certain conditions, and if your system interfaces or interacts outside of these conditions, you might observe unplanned behavior.

You might notice that the above section feels like a statistics refresher. However, good data and corresponding analysis are crucial for good design. You can't quantify what you haven't measured or tracked. Goodhart's Law unwraps a challenge in system design: the tension between measurement and achieving the true objectives.

Throughout this text, I've provided examples of how improvement in proxy metrics can distort the true goal we want to achieve. Understanding this challenge gives us tools to build better systems. By employing multiple complementary metrics, carefully tracking variations over time, and deeply understanding the causal (not just correlational) relationships within our systems, we can create better evaluation frameworks. The goal isn't to find perfect metrics but to design adaptive systems that respond when metrics cease to serve their intended purpose. It is also worth mentioning that some systems are simple enough to be aligned with a single metric. However, as we develop more sophisticated systems, especially those incorporating AI and automation, careful alignment of metrics with objectives becomes crucial.

The lesson of Goodhart's Law isn't that measurement is pointless, but that thoughtful system design requires constant tracking and adaptation. We must regularly question our metrics, examine their effects, and be willing to change our evaluation methods as circumstances change. The process of figuring out the right variables and metrics that correctly measure the productivity/quality of a system is typically cut-and-try. Only through this cycle of measurement, observation, and adjustment can we create systems that properly serve their intended purposes rather than becoming married to our "trusted" metrics.

Powered by Froala Editor