How can the theory of measurement be used to boost effectiveness?

Impact measurement is a commonplace term, but using it to improve effectiveness takes commitment
impact measurement charities
How can measurement be used to improve effectiveness? Photograph: Elizabeth Young

When an organisation builds impact measurement into job descriptions and includes the issue in staff inductions, there is a better chance that it will operate more effectively.

This is the case at IntoUniversity, an award-winning London education charity that supports disadvantaged students to achieve their aspirations. In the recent New Philanthropy Capital (NPC) report Making An Impact, the charity features as an example of how to monitor impact to improve performance.

While IntoUniversity has looked at its impact since its launch in 2002, it concentrated further on the practice when it decided to expand. It was supported by a qualitative evaluation by the National Foundation for Education Research (NFER) and its funder the Impetus Trust helped focus the commitment to such monitoring.

The charity developed self-reported measures, like asking students to report the changes happening for them, then surveying the issues they outlined and building up responses. It also used a social return on investment analysis and introduced new software and systems to help collate results so questionnaires were automatically converted into Excel files. The charity now has over 10,000 records to help it understand what programmes work best for its young beneficiaries.

Among the insights, for example, is the fact that more intense programmes with fewer students attending more frequently have higher success rates. IntoUniversity can use such data not only to work out how to retain students, but it believes the impact approach helps the charity keep its competitive edge, underlining its goals.

While impact measurement is becoming a commonplace term in the sector, not everybody is always clear how to translate its theory of measurement into the reality of boosting effectiveness.

The NPC's Inspiring Impact programme recently published a code of good impact practice for consultation. The code, developed by organisations including NCVO, NEF, Social Enterprise UK, SROI network, CES, NAVCA, Dartington Social Research Unit, Small Charities Coalition, Macmillan Cancer Support and the Charity Commission, outlines principles like a focus on purpose and the use of "proportionate and appropriate" methods and resources. It encourages organisations to consider the positive, negative, planned and unplanned differences they make and to act on what they find.

So what does a charity that successfully uses impact measurement to improve effectiveness look like? David Pritchard, NPC's head of measurement and evaluation, says: "It is open to the possibility that it may not be very effective. It is difficult to bring about significant improvements and increased impact if people within the charity are defensive about what they do."

A commitment to continuous improvement is a must, Pritchard adds, expanding on one of the key messages in the new code. "There is little point to collecting data and measuring your impact if you are not prepared to take action on the results. That is very hard to do without the right culture." A clear focus and care about what is measured and how is vital, he says. While creating change is hard, being too general or vague about the change you want to make "is a sure way of being ineffective".

Sonia Sodha, head of policy and strategy at Dartington Social Research Unit, agrees that a refined vision is important. She defines impact measurement as "very much about independent measurement of intervention", it is about documenting a specific set of outcomes – "not just about looking at any outcomes".

She adds: "If you're interested in measuring impact of intervention, then you need to look at a randomised control trial." The initial and final outcomes – pre- and post-testing – are useful in clarifying whether or not you are heading in the right direction but a before and after focus in isolation is not enough, she says.

The hallmarks of an organisation that uses measurement to be more effective is a long term evaluation strategy. "It's an organisation that has clear idea of refining its interventions, who is it aimed at how is it trying to achieve [its aims]."

As for the challenges, says Sodha, cost is generally cited as one barrier although she says measurement is not as expensive as some think. This echoes the suggestion in the NPC's new code about the need for "proportionate and appropriate" methods and resources. Other problems occur when organisations evaluation for evaluation's sake so they start measuring without, for example, first ensuring their intervention is well designed in the first place. Equally important is a balance between qualitative measurement and quantitative, the two complement each other but each on its own will not create a full picture of impact.

"The first and probably biggest challenge is for a charity having the courage to try to collect data on its impact in the first place," adds Pritchard. "Investing in measuring and evaluating your impact takes resources and carries risk. What if the results are not as good as you expect? How do you explain that to staff, trustees, and funders or donors?"

There is also the risk, he says, that charities "want to measure it all" because they believe their work has varied positive impacts. "This is both impractical and ill-advised," he ssays, warning that "trying to measure too much because you are not really clear on exactly the changes and benefits that you bring about is almost as bad as not measuring anything".

Another dilemma, says Pritchard, is how to estimate what would happen to your beneficiaries without the work of your charity. "People who work for charities on the frontline see the positive effects of their work daily, it is hard to try to 'prove' that any changes beyond the very short-term happen because of what you as a charity do", he says.

According to IntoUniversity's impact report, a key indicator of success is the number of young people who progress from the programme to university. By tracking its students as they left school in 2012, staff know that 77.1% have a university place. The charity benchmarks this, for example, against the 34% national progression rate for all students from state schools and the 18% rate for all students who had free school meals.

This focus on demonstrating effect is crucial if staff and supporters are to know if they really have the transformative impact they strive for. In an NPC recent analysis of the charity it is described as "one of the best-run organisations that NPC has seen". Why? Not just because of the "effective model", "impressive management" or "strong finances", but also due to its "increasing evidence of its impact".
