I often get the question: ‘What is a good evaluation tool?’. Of course I must answer that I don’t know, or even worse: they don’t exist…
As in the Zimbabwean (Shona) proverb ‘Don’t beat a drum with an axe’ you must be careful with tools: you cannot use ‘one tool fits all’. But that’s unfortunately exactly what many evaluation theorists want to make us believe: ‘Use my tool and you will solve the X,Y or Z problem!’
It does not work like that and that can be shown best with the following diagram:
or in less theoretical language, e.g. for a brave project in Zimbabwe:
Most of the time the ‘first generation’ (quantitative) information can be counted (in the upper left box) and should be counted as well! But this is not much more than what we use to describe as output!
We need ‘Second generation’ information – qualitative , what are peoples ‘perceptions’ – for the upper right and lower-left box, in order to understand more about the outcome (what does the project do with people, what do e.g. trainees do with their acquired skills).
Now we also need to know about the impact: what effect does the project have on the (Zimbabwean in this case) society?
And then we are in dire straits, where a single tool cannot help us anymore: we will need a host of methods / methodologies to find out about it: how else can you measure something that is unknown as a practice and for which very few standards are developed?
In my view the best way is using the ‘back of your brains’: putting all your knowledge and skills on the scale and start working on a tailor-made methodology. Many times in my case that will be fourth generation (see http://www.bwsupport.nl) or one of the many tools we are trying to develop in the fifth generation methodologies with a group of evaluators (see http://www.evaluation-5.net). That can include (parts of) Most significant change, Social return on investment, Developmental, Outcome mapping, logical framework, etc. methodologies.
But the way of operating is virtually always the same: count whatever can be counted – it is certainly necessary, but it is not enough. Personally I prefer qualitative methodology for outcome and a mix of co-creative, constructivist and iterative ways of ‘tackling impact’.