Accountability Without Effectiveness

Metrics, metrics, metrics

Governments track extensive quantitative information about their activities and intended outcomes. This data theoretically enables government actors to monitor progress against work plans, strategies, and priority projects while communicating results to leadership and the public.

While metrics serve as essential tracking instruments, several unintended traps can obstruct efforts to generate value, foster innovation, or challenge the status quo. Data collection's primary purpose should be learning — particularly when addressing complex challenges requiring experimentation to discover what works. Yet when data serves only accounting functions, it creates friction.

Ten Metric Traps

1. Using metrics mainly for accountability, not for learning, understanding or advocating. Learning should be the prime directive, from which the rest follows.

2. Deciding on metrics before we even know what the solution is, which constrains discovery.

3. Measuring something just because we can, not because it's important. Not knowing how to measure something doesn't mean it's impossible to measure.

4. Prioritizing quantitative over qualitative methods because we think that quantitative metrics are objective and bias free. While quantitative data reveals what is happening, qualitative data explains why and captures human impact.

5. Micro-managing through metrics — "your performance will be measured by the number of widgets you produce, how you produce them is totally up to you" isn't really giving autonomy or creative freedom to teams.

6. Picking metrics just because they're easy to measure, not basing our decisions on what will provide the most value.

7. Choosing metrics without talking to others or thinking it through. In the rush to meet deadlines we haphazardly decide on key performance indicators without much thought.

8. Measuring outputs and not outcomes — we end up counting beans and determine success based on outputs without tracking quality and effectiveness, or how it affects people.

9. Focusing on short term measures over long term impact. Building follow-up evaluation into projects/programs is critically important, especially when dealing with complex problems.

10. Measuring process efficiency and not experience or effectiveness. If we are only measuring the number of transactions and transaction time then improvement efforts will inevitably be based in optimizing processes. Measuring experience and effectiveness, while more challenging, will help us determine how well processes work for people.

Data Collection's Four Purposes

Organizations collect, analyze and share data for four main reasons: to keep track; to tell others; to show why something is important; and to learn and improve. Many organizations don't use data well. For nearly a decade I have advocated for and trained organizations to be more data-driven. When teaching this work, accountability represents only a quarter of data collection's value. Learning is fundamental to driving continuous improvement and performance excellence. Yet many focus excessively on tracking rather than learning.

Data paves the road to the bottom. It is the lazy way to figure out what to do next. It's obsessed with the short-term. Data gets us the Kardashians.

— Seth Godin

The Effectiveness Gap

Organizations might be executing all required activities and hitting established goals, yet effectiveness remains unchanged: public confidence in government is low; employee satisfaction is low; and public needs remain unmet. This disconnect may stem from overemphasizing transaction counts and output metrics. These measurements should represent only part of a broader approach to learning and improvement.

Hitting the Wrong Targets

When organizations hit all the wrong targets while only measuring target completion, the result becomes clear: government demonstrates high accountability but low effectiveness. Every improvement initiative then emphasizes speed and cost reduction rather than effectiveness or user satisfaction.

How might we start measuring what makes up the relationship the public has with government?