TL:DR — It is impossible for organisations to “demonstrate their impact” if they work in complex environments. Asking them to do so requires them to create a fantasy version of the story of their work. This corruption of data makes doing genuine change work harder because it is difficult to learn and adapt from corrupted data.

Everyone who works to make public service operate more effectively wants to impact the world. Having an impact is a great thing. It’s an important North Star for a person, team or organisation. What we do should make a difference in the world.

However, too often, the way that the concept of “impact” is used by organisations gets in the way of the real-world change that people are trying to create. In previous work, we have assembled evidence which helps to show that using outcomes as performance management tools doesn’t work, and this is similar territory. The idea that we can measure our impact to understand how well we are performing is seductive, but fundamentally flawed.

Using “impact” for any form of performance management makes it much more difficult for organisations to do impactful work. This is the uncomfortable truth that leaders must face if genuine impact is to be created in the world.

This truth is uncomfortable because it would be really, really handy for managers and leaders if we could use impact as a performance management tool. It would make the job of managing public service (or any form of social action) much, much easier. This is why it is such a seductive idea.

However, part of the responsibility of leadership is facing uncomfortable, difficult truths. So, let’s examine the evidence around “demonstrating impact”.

Demonstrate “your” impact

Many teams or organisations seek to demonstrate “their” impact — the difference their work makes in the world. They are often asked to do this by those who fund the work. Sometimes they do it because they want to help their staff see the difference that they make.

There’s only one problem with this. Almost all useful social change is achieved as part of a complex system. In other words, your work is a small part of a much larger web of entangled and interdependent activity and social forces.

The systems map of the outcome of obesity illustrates this perfectly — it shows all the factors contributing to people being obese (or not), and all the relationships between those factors.

This is the reality of trying to make impact in the world — your actions are part of a web of relationships — most of which are beyond your control, many of which are beyond your influence, quite a few of which will be completely invisible to you.

All of these things combine with your actions to create impact in the world. Let’s work this example through using the obesity systems map. Say that you’re one of the people operating in the bottom right corner of this system — you’re providing “healthcare and treatment options” to address obesity. Let’s say you’re delivering weight loss programmes in neighbourhoods. How would you distinguish the impact of your weight loss programme from the influence of all the other factors in this system?

Short answer — you can’t. Someone on your programme sees a film that changes their perspective on the meals they cook. Someone on your programme changes jobs, to a place with a canteen where they only serve healthy options. Someone is made redundant, so they can’t afford to buy organic food. What was the impact of your programme in these situations?

This reveals a fundamental truth about the nature of complex systems. In a complex system, it is impossible to distinguish the effect of particular actors on the overall pattern. This is because complex systems produce emergent, nonlinear behaviour. The tiniest change in input variables creates potentially huge changes in results. Consequently, you can’t produce a reliable counterfactual in a complex system. (You can’t say what would have happened if X wasn’t present). And if you can’t produce a reliable counterfactual, then you cannot reliably identify the impact of your activity.

Contribution analysis — doesn’t really solve the problem

It is increasingly recognised that it is impossible to reliably attribute impact to a particular intervention in a complex system. This is why you will