When Not to Evaluate
If the project is new
New projects can be very difficult to evaluate, and are unlikely to have a lot of data available. Additionally, it’s likely to be a time when resources are more limited, with people focusing on getting the project to run smoothly.
If the project is constantly changing
Projects that are in flux, often have unclear or inconsistent aims, outcomes or activities. It’s best to wait until these aspects have settled down before attempting to evaluate.
If the findings won’t be useful
If there isn’t a question to answer, then evaluation might not be necessary.
If the evaluation won’t provide any new evidence
If the evaluation is unlikely to tell you anything new, then it may not be necessary; there’s no need to keep testing the same thing.
‘If it looks like a duck, quacks like a duck and walks like a duck - then it is probably a duck."
Keeping it in proportion
It can be tempting to want to evaluate everything all at once, or think that one evaluation will tell us everything we need to know but this is probably unrealistic.
When I was first deciding on a topic for my dissertation, I believed I could solve the answer to crime in three months all by myself. Then I realised that there just wasn’t enough hours in the day! So I scaled back my approach, choosing a topic that was important to me and a current area of concern. I looked at the evidence to see what was already known and where there were still gaps. This helped hone my focus, providing a much more detailed and precise evaluation of the evidence.
We often come across similarly optimistic evaluations and evaluation plans; so broad that they are unlikely to actually provide meaningful insights. So, if you’re just starting out with your evaluation, here are some things to consider:
Be clear about the types of questions you want answered
If you’re looking at processes i.e how your project works, then it’s no good doing an impact evaluation and vice versa.
Keep it simple – develop a small number of indicators to capture change.
It’s better to do one thing really well, than lots of things poorly. What shows you’re having the biggest impact or what outcome relates most to your aim? Could you evaluate that? Or, do you have an area of the programme that is going really well or not so well?
It’s not a one-stop shop
Evaluation is a continuous process, and one evaluation is not going to be able to answer all of your questions.
But you don’t need to keep re-inventing the wheel…
If there is already evidence out there that backs up your evaluation, is it necessary for you to repeat this? For example, we know there’s a really strong link between mentoring and increased confidence, so perhaps you don’t need to measure this outcome but instead focus on another.
In summary, be realistic, consider whether it is necessary to evaluate and what you might be able to find out. Remember that one evaluation is unlikely to tell you the whole story, but done well it can be an extremely powerful tool.
Natalie Arnett, Project Oracle
Natalie is a research officer at Project Oracle. She provides support to a number of different workstreams, including validation of provider evaluations, supporting student research placements and the Evidence Champions Series.
This blog was original published on Project Oracle's website and can be found here.
