Accountability or evaluation?

Starting up again after the holidays with some philosophy…

Language is an important issue in evaluation I realised just before holidays during the presentations on the INTRAC conference on Monitoring and evaluation.
Just have a look at the words accountability and evaluation…

Where one is developed from the root counting and the other from the word values … you can imagine that they also represent the two main lines in PM&E (Planning, monitoring and evaluation).

Accountability is the issue of the day at the moment: and it’s most of the time narrowed to upward accountability (to the donor) and to counting (measuring) effects…
The introduction of the director of the IOB (Dutch ministry of foreign affairs ‘Inspection for development cooperation’ – what’s in a name…) was completely pushing towards accountability: showing the need for thorough research into the impact of developmental interventions. Roughly speaking: the main aim is to find proof of results in developmental projects, in order to convince politicians to budget for international cooperation.  That’s in itself a noble thing: without money there is a lot less possible. But although I consider it necessary, I think its not sufficient.

And there we arrive at the values of evaluation: I think in the end it is more effective to go back to the roots of International Cooperation (and solidarity): somewhere after World War two it was an outcry of ordinary citizens in the richer countries (like The Netherlands) that they found it not compatible with their values to see large part of people in the (then Third) World succumbing to famine and suffering poverty in general. That led to the start of many a ‘development organisation’ like (Oxfam) Novib etc. where the public set aside some of their own wealth in order to show their practical solidarity with people living in poverty.

I see evaluation as a process where we should go back to our joint values and have a look together whether we accomplished what we intended together (as joint stakeholders, first and foremost local people and organisations at the place where project or programmes are running). So there is no space for corruption, there is no space for negligence and inefficiency – no difference here with the accountability paradigm. The difference is in the way of assessment and what’s more in the goal of the assessment. In my view we should focus on a joint understanding of what has been reached and on a joint effort to reach more. Hence my interest in new forms of evaluation (see www.evaluation-5.net) and my constructivist methodological approach (see www.bwsupport.nl).

The results are also different: of the accountability inspection the intended result is to convince politicians (if they can be convinced at all…), of the evaluation the intended result is to do it better together – both local and international stakeholders who negotiate about their joint goals.

But of course that would need a change in paradigm in  International Cooperation: the wording International Cooperation in itself is almost entirely that of a state-to-state cooperation: the public itself is less involved nowadays: most investments in solidarity are taxpayers money nowadays, decisions are taken by politicians. If people want to change that we ought to ‘put our money where our mouth is’: more projects need to be directly financed by the people who want to express their solidarity, and should not be left to politicians. Fortunately there are signs here and there (1% club, more solidarity groups, etc.) of a (re-)developing paradigm: it’s not only nostalgia…

So there is a lot in words, thinking through the very intentions behind them….

Next time more practical: quantitative and qualitative impact assessment

Sharing 5th generation experience

With my colleagues Rosien Herweijer and Marlen Arkesteijn from evaluation 5.o we wrote a paper for the upcoming INTRAC conference on monitoring and evaluation titled “Fifth generation Evaluation of a HIV/AIDS prevention programme among LGTBI in 15 countries Africa and Latin America”. In the paper we describe how we used constructivist evaluation methodologies to facilitate the evaluation of the International Programme of Schorer Foundation, Amsterdam generating a process that produced real learning among all involved

Evaluation Tools

I often get the question: ‘What is a good evaluation tool?’. Of course I must answer that I don’t know, or even worse: they don’t exist…

As in the Zimbabwean (Shona) proverb ‘Don’t beat a drum with an axe’ you must be careful with tools: you cannot use ‘one tool fits all’. But that’s unfortunately exactly what many evaluation theorists want to make us believe: ‘Use my tool and you will solve the X,Y or Z problem!’
It does not work like that and that can be shown best with the following diagram:

or in less theoretical language, e.g. for a brave project in Zimbabwe:

Most of the time the ‘first generation’ (quantitative) information can be counted (in the upper left box) and should be counted as well! But this is not much more than what we use to describe as output!

We need ‘Second generation’ information – qualitative , what are peoples ‘perceptions’ – for the upper right and lower-left box, in order to understand more about the outcome (what does the project do with people, what do e.g. trainees do with their acquired skills).

Now we also need to know about the impact: what effect does the project have on the (Zimbabwean in this case) society?

And then we are in dire straits, where a single tool cannot help us anymore: we will need a host of methods / methodologies to find out about it: how else can you measure something that is unknown as a practice and for which very few standards are developed?

In my view the best way is using the ‘back of your brains’: putting all your knowledge and skills on the scale and start working on a tailor-made methodology. Many times in my case that will be fourth generation (see http://www.bwsupport.nl) or one of the many tools we are trying to develop in the fifth generation methodologies with a group of evaluators (see http://www.evaluation-5.net). That can include (parts of) Most significant change, Social return on investment, Developmental, Outcome mapping, logical framework, etc. methodologies. 

But the way of operating is virtually always the same: count whatever can be counted – it is certainly necessary, but it is not enough. Personally  I prefer qualitative methodology for outcome and a mix of co-creative, constructivist and iterative ways of ‘tackling impact’.

Media, development and evaluation

At the moment I’m working with 2 African Media Organisations: one in West, one in Southern Africa. They are quite different in almost everything, but the most interesting is the goals they are after:
One wants to open up the debate in society in order to promote social justice. That is what media are not always after, specifically not commercial media… The second is an exile newspaper, and they only want one thing: to give a voice to the voiceless, yes, but then tell the facts like they are, whoever is involved.

It makes me think of an old, ongoing debate in journalism: are media meant to promote something – like social justice, or development? – or are media there in the first place to ‘tell it like it is’ ?.
Or may be it can be linked: can you promote social justice by ‘telling it like it is’?

There is a lot of (philosophical) theory written around this theme, in the first place by the German philosopher Habermas (since 1989: The structural transformation of the Public Sphere), but the discussion is still alive and kicking…

Most concretely: do we want to develop the media (1) or do we want to promote development through media (2) or do we want to to ‘tell it like it is’ also where development is concerned? I am quite convinced of the last goal, and much less about the first 2.

In my view the media should develop themselves (where they can use some help sometimes), the development should also develop itself, so let the media stick to the core business; ‘telling it like it is’!

Why is this so necessary? Not because I think there is only one truth in this multifaceted world. Not because I think any media can ‘tell the truth’. The reason is simply that there is one thing media must have, and that is credibility. If you do not trust the people who are ‘sifting’ the news for you, why would you read them, look at them, listen to them?

If people get the right or (or even wrong) idea about whatever media that they are speaking ‘their master’s voice’ – even if that master is a charity of good will – those media will instantly loose credibility.

Now both organisations I’m working with do solve that problem in a certain way: one is above all supporting (community) media that themselves are independent ‘without fear or favour’ (as far as possible in autocratic Africa). The danger they’re in is that they themselves become the master – small news organisations are willing to talk their talk…

The other is so opposed to the powers that be, that they almost become a ‘mouthpiece’ of the opposition, thus also becoming someone’s ‘pet dog’ – at least in the perception of readership.

The only answer I have is that it’s a balancing act, time and again – as it is in my job: evaluating them. Are we the donor’s pet dog? Or are we only talking the talk of the beneficiary? It’s a balancing act between those two, although the solution is not ‘staying in the middle’, but rather telling it like it is: credibility is also our best friend and it can only be earned by staying deadly honest.

#Twevaluation 2: How useful is Twitter?

At the 1% Event on 17 september 2010, the twitterati sent tweets like: “Great energy #1pce” and “Good buzz @ #1pce”,’ Tiina Urm speaking #1pce, http://www.youtube.com/watch?v=A5GryIDl0qY”.
Overall, Twitter is used to inform the followers on what is going on. The tool is great for short announcements, and for referring to Youtube movies and websites. Quick, in 140 characters (less if you leave space for re-tweeters), a message with an impression. Furthermore, thanks to the hashtag #, anyone can quickly see all the tweets about a specific subject.

For monitoring purposes Twitter is a tool to assess the mood at an event. Are people mentioning what is happening? Which activities are they mentioning most? How are they mentioning the activity?

What can Twitter add as an evaluation tool?
Tweets need to be collected immediately after an event with the Twitter Search function and saved elsewhere; leave it for a few days and the majority of tweets has already disappeared. The tweets can then be counted, individual twitterati counted, mood assessed, and comments counted and analysed. We noticed that only a few tweets had some form of content that was useful for evaluation purposes, beyond very general announcements. Out of the 900 tweets only 96 had some form of content/mood that was useful. Furthermore, all comments tend to be positive, in line with Twitter culture.

The questions that the twevaluators asked were seldom answered. Even personal replies with an additional question were not answered. Somehow, that does not go well with the medium. Consequently, the evaluation questions formulated for the day were not answered by Twitter.

What worked well was the screen in the conference main hall. Everyone could see the tweets, and the re-tweets by the tweetdeck. That added to the mood of “It is happening now!” and it added to the inclusion of all participants. During the day, we became aware that lots of participants never twittered, and did not have an account. One even twittered afterwards that he did not dare to use his old fashioned telephone in such a hip environment!
We may need more visuals for the ongoing outcome of the tweets real time, for example a time line with the best activities or a top 10 on the screens that evokes more evaluative tweets; this might encourage more participants’ involvement.

This first #Twevaluation experiment has generated new questions. We now know that twitter has possibilities: reading all tweets for energy, mood and positive feedback. Next time, the twevaluators would:
· Just use the general hashtag, the specific #twevaluation added nothing
· Not ask specific questions but just a few general ones, to be repeated during the day
· Add an “open computer” for participants who wish to Twitter but never did it before or did not bring their own computer/telephone
· Align the questions of other evaluation tools like the questionnaire with the Twitter questions (most inspiring, most useful, number of new contacts)
· Add a visual tool on the screen, to show aggregate data all day long (encouraging more participants to vote for the most useful part of the program, for example)
· Involve the bloggers more, to generate more tweets related to content
· Count the total outreach of all the twitterati by checking the number of followers
· Consider sending the twitterati or/and their followers a follow-up question, a few days after the event

Gisela Dütting
Bob van der Winden

Black slaves, black coffee, black oil: Angolan – Dutch relationships over 4 centuries.

Intervention during Angola Conference on 23 September 2010 in Wageningen by Astrid Schipper and Bob van der Winden (see also http://www.ces.uc.pt/conferenciaangola2009 )

Slaves, coffee and oil…
Of course you all know that Luanda was occupied by the Dutch between 1641 and 1648.
That was when ships of the West and the East Indian Company sought to overtake Portuguese dominance in the Angolan slave trade. Mutual interest to ‘rule out and destroy’ the Portuguese resulted in a treaty with the re-known Angolan queen Rainha Ginga. The queen would hand over half of the slaves she made to the Dutch. In return the Dutch would support the queen against the Portuguese. Towards the end of 1648 the Portuguese forced the Dutch out of Angola, but the slave trade had flourished for all those years and would flourish for a long time still…  After the Dutch colony Indonesia gained independence the Dutch moved to Angola for their coffee. Prior to the liberation war Angola was one of the worlds’ main coffee producers and the Netherlands was one of the main buyers of Angolan coffee. The Dutch Angola Committee (later Holland Committee on Southern Africa and NiZA) put pressure on major Dutch coffee roasters and distributors to stop selling coffee from Angola. Albert Heijn and Douwe Egberts were so afraid that they stopped imports even before the campaign had been publicly launched. This coffee boycott became a textbook example of how a small group of civil activists can achieve results with amazing speed… The third wave of Dutch – Angolan trade in a black commodity is going on to date: oil. In 2008 Angola’s export income is said to be $67 billion and $60 billion comes from oil. The value of Dutch oil-import has risen from 57 million euro in 2005 to almost 1,2 billion euro in 2008. No wonder that Dutch ambassadors openly stated that ‘energy security’ is the Dutch department of Foreign Affairs’ first priority.

From trade to aid and back

During the first 30 years after liberation Dutch policy focused predominantly on providing humanitarian aid to the war torn population. Besides some modest support in the areas of human rights and good governance, the bilateral aid has now virtually dried up: The amounts provided went down from 12 million euro in 2002 to a last input of 2,5 million euro in 2005.

Meanwhile the bilateral focus of the Dutch Government has shifted in Angola from aid (back) to trade and several facilities have been developed to promote this. The Dutch interest in Angolan oil and Liquefied Natural Gas (LNG) and the Angolan interest in Dutch “know how” and the technological expertise to cultivate and transport these are key factors for an advantageous mutual cooperation. Also the Dutch tradition in maritime transports and in the management of ports is of great interest for Angola. For example Dutch company Heerema Maritime has just finished the construction of a shipyard in Porto Amboim (Kwanza Sul) This shipyard is now providing assistance to Angola’s offshore oil and gas industry; Royal Boskalis has been awarded a contract for the dredging of the Soyo Liquefied Natural Gas port in North Angola…. In the meantime Angola has also moved up in the ranks of Dutch export and is currently the second largest market for Dutch (mainly consumer) products in Africa, after South Africa.  For the past years the Dutch Embassy published a newsletter focusing on business opportunities in Angola. The title is ‘Time is now!’ But looking closely at the Netherlands – Angolan trade ties over the past centuries it may be clear that there is no such thing as ‘Time is now!’ For centuries the Netherlands got a very good deal out of its linkages with Angola. From the treaty with Rainha Ginga to the deals struck by the 2009 Dutch trade mission headed by the Dutch Minister of Economic Affairs.

Whereas the general public seems to think that nations like China have intensive trade relations with Angola because they do not bother about all sorts of human rights, honestly, we should recognize that the Netherlands has not fared badly either while trading with Angola. The question is ‘is that a bad thing’? After all ‘no aid but trade’ is a slogan that has been welcomed since a long time by large part of the ‘third world movement’… Once Dutch companies start taking ‘Corporate Social Responsibility’ more serious this might be even a good development …

‘Reversed Boston tea-party’
Finally: what does the (re-)structuring of the relationships between Angolan and Dutch government ‘from aid to trade’ mean for Civil Society here and there? David Sogge and I wrote about it it in the book  ‘Southern Africa: Civil Society, politics and donor strategies’: We conclude that the civil domain in today’s Angola is highly constrained. Government is not accountable to its citizens: their income comes from offshore. Dambisa Moyo descibes this in ‘Dead Aid’ as the reversed Boston tea-party: no representation without taxation… We therefore argue that international development agencies, currently fixated on project-based development strategies, should put much greater priority on enlarging and protecting public domains expanding the ‘public arena’ and making it transparent. Popular leverage and a shift in the balance of forces in the arena is important, but a viable objective in the middle run would be to enhance responsiveness of the Angolan state to citizens’ needs, e.g. by supporting social movements to become strong enough to hold the government accountable, thus setting precedents for wider formal democracy in the longer run. Thus focusing on the direct contribution of Dutch civil society to the strengthening of Angolan civil society. May be this in the end could historically seen also become a continuum… like the trade relations between both countries….

Crowd-sourcing by Twitterati at 1%event

Margreet van der Pijl of the 1% club wrote in her blog Quality improvement through Crowd involvement how the 1% club is looking for new ways of evaluation. Gisela Dütting (http://giseladutting.wordpress.com) and I took up the challenge and decided to start with the 1% event last friday, 17 September. We counted on the many ‘twitterati’ in the crowd and started formulating questions to be asked during the day.

Indeed among the 300 participants there were 110 individual twitterati who produced 900 tweets all together…

We asked general questions with the hashtag #1pce and #twevaluation: that did not work – no answers…

We also directed questions to people who had for instance indicated that they liked a specific presentation or open space workshop (e.g. ‘what did you like specifically’): all together during the day: 4 answers (as many as we got on one general question, more or less a ‘control’ question about the quality of the lunch…).

Our conclusions:

The 900 tweets could after the event be retrieved and analyzed – there were 96 tweets that contained enough content to make analysis possible. The fact that 900 tweets were produced on 1 event (which made the event the 5th largest event on Twitter in Holland that day) is in itself significant for instance.

But also the emphasis on the good feeling during the meeting and the enormous amount of positive energy during the day speaks to the imagination.

Tiina Urms presentation on cleaning up the WHOLE of Estonia in one day (onepercentblog.com/tag/tiina-urm/) was by far the most popular presentation for the twitterati.

But twitter as a tool? We really need to improve it for next time – if it is at all possible to use it for monitoring of an event…. (‘instant crowd sourcing’)?

Gisela Dütting and Bob van der Winden

VIDEO evaluation

There are as many ways in video evaluation as in any other branch of this sport…. You” find a good introduction in : http://www.evaluators5-0.net/index.php?page=Power%20of%20Images&id=94
by my good friend and colleague Marlen Arkesteijn.

What we intend to do with a small team of students and HA people is following, monitoring and start with evaluating the RDNA-activities @HA2010 by video, asking people’s cooperation for an interview and a short video (1 minute) after each event. We will then make short video compilations of what people have to say and place them on the website of HA2010, inviting others to comment… If your comments are interesting enough (!?!) you might be asked for an interview as well!
The final objective of this monitoring event is to try to find out what it does to people to be at events like this: what do they ‘learn’ – do they see practical implementation possibilities? What happens inside their heads (after the hangover…) and what happens later in their practice?
Don’t hesitate to comment and come up with tips: you’re most welcome!

From development to transparency…

Too often people think that Civil Society are formal NGO’s: following Habermas’ theory of public sphere we take civil society as a space rather than a set of organisations. It may be thus termed the civil domain. Civil domains are nested in a larger sphere, the arena. In this arena conflicts of power are fought out – in cold and hot wars. This two-sphere model lends itself to an analysis of power in and around a country or an issue.

Many constraints arise in the interplay of economic, political and military forces in a ‘public arena’, where state and non-state actors contest for power in ways strongly influenced by geo-politics. Oil and diamonds, etc. as well as security. In that sense natural resources have – de facto – the same effect as development aid (especially where it is used for budget or sector support). The effect is disastrous for democracy in a way that Dambisa Moyo describes as the ‘reversed Boston tea party’: ‘no representation without taxation’: money for public expenditure does not come from the local taxpayer but comes ‘from above’: ergo: the government holds the civilians ransom, the base for clientelism is there.
International development agencies are currently fixated on project-based strategies, including support to NGOs as vehicles for relief and ‘development’ services and for ‘advocacy’. We argue that the priorities should instead fall on domains and arenas: multiple – courageous – actors, rather than institutes.
We – as development practitioners – have a problem here, that cannot be solved simply and that is mainly given by the catch 22 we’re in: we know that we should support activists and grassroots organisations, but our money comes from governments that have made accountability more important than effectiveness …
Because of the sometimes absurd regulations we are almost obliged to spend the money to large NGO’s that we know to be ineffective…. (but write good reports…).
So I don’t know the solution for the overall problem, but I know that some – smaller – northern NGO’s are trying to bridge the gap I described and that sometimes through ‘umbrella NGO’s’ the money can effectively end up where it needs to be.
What my argument boils down to: let’s act together, let’s go on, or let’s find more ways to support those people who deserve our support and to people that have the courage to enlarge the public domain trough media, public debates and everything the like.
We admire courageous people in non-democratic and corrupt states – so let’s try to be a little bit courageous and defy the actual discourse in international cooperation where it works to the detriment of those who really need it.

The paradox of aid….

Some time ago I was in Nairobi for an evaluation of a global youth organization. I worked mainly with youth groups in Kibera and Mathare respectively, two ‘one million inhabitants slums’.

These hard working youngsters (all under 30) do hardly get any support from the international development organizations and not a shilling from their own government. Still they have over 50 members each, run small schools for orphans, organize ‘micro-credit groups’ themselves, have some money put aside for income generating loans (with 500 euro you can start a complete water distribution point, where 10 youngsters are busy with), run health advice for single mothers, etc. etc.

At night, back in my hotel I read in the papers that the minister of education has answered to the accusations of DFID from Britain that 30 million pounds have disappeared: He admits that it is at least 1 million pounds and that officials in his ministry have received the money for receipts manufactured by themselves. By the way: the minister stated that he would not resign, because he did not want to suffer for the misbehavior of his juniors… so much for the ministerial responsibility…

So the bilateral support for education in Kenya is now withheld by the UK government… Anyhow I saw with my own eyes that ‘education for all’ in Kenya (that is the programme Britain contributes to) means ‘income for the few’: no cent had been spent by the ministry of education nor by any civil society organizations to the 150 school kids in Mathare: shacks serve as schools (educated by volunteers from the community)…
And of course this kind of things do not only happen to British development aid, it happens all over the place.

What do I see in this example – one of many?
• In the first place it supports Patrick Chabals analysis (see ‘Southern Africa: Civil Society, Politics and donor strategies’(2009) to which I also contributed an article on Angola) that ‘power is about the control of resources’ – development aid is but one of the resources the political class controls – through the state…
• Further I saw once again that where the need is highest seemingly ‘chaotic’ and disorganized groups of volunteers take the lead and do something. Groups that are very seldom linked with registered charities and NGO’s (that can receive our international funds).
• In the third place it is clear that in the ‘aid arena’ the grassroots organizations are indeed in jeopardy (I quote Chabal again) because they do not meet our bureaucratic criteria – which paradoxically have come into life above all because of public outcries – specifically because of frauds like this story! So: because of the fact grassroots activists get no money from the government (that is subsidized by development agencies) aid on al large scale is frozen – so they will never get any money…. Do you still wonder why many people in sheer disbelief want to go and bring their own money to Africa? I can understand it better and better….