Oct 17, 2012
Velayuthan Sivagnanasothy, Secretary for Ministry of Traditional Industries and Small Enterprise Development in Sri Lanka, shares his insights on evaluation of development programmes.
Velayuthan Sivagnanasothy, Secretary for Ministry of Traditional Industries and Small Enterprise Development in Sri Lanka, was recently interviewed by 3ie in Colombo. Siva has held several senior positions in government and was the lead for instituting Managing for Development Results and establishing the national electronic Project Monitoring System in Sri Lanka. He offered his insights on evaluation of development programmes in this interview. Excerpts:
You have been both a policymaker and evidence broker with the government of Sri Lanka. Tell us about your experience in impact evaluations in Sri Lanka?
VS: Traditionally, Sri Lanka had a system of Monitoring and Evaluation (M&E) where we assessed projects on the basis of outcomes and sustainability. But when I was Director General of the Ministry of Plan Implementation, we were keen to introduce evaluation in government. We got help from the Asian Development Bank (ADB) and the UNDP. The ADB trained our officials and helped to establish our Evaluation Information System (EIS). We collected evaluation reports prepared by the government, donors and research organizations, synthesized those evaluations and added them to the EIS database. Key findings were highlighted to help policymakers learn and understand the findings quickly. The projects were also rated on different scales of success – as per the ADB guidelines. We now follow the Organisation for Economic Co-operation and Development (OECD) criteria and methodology. We look at the relevance of the project from the national development strategy point of view as well as the theory of change, the balance of components to deliver the output, and programme theory. There is also the principle of efficiency: whether the project has been implemented within the time and estimated cost. Also, the effectiveness of the intervention is examined through immediate outcomes. We also look at whether the project has the potential to produce long-term impacts and sustainability.
How important are independent evaluations?
VS: In the early 1990s, a World Bank team came to Sri Lanka on an evaluation mission. I was asked to be part of the team but the evaluators were unhappy. They thought participation of insiders in the evaluation process was inappropriate.
Participation of government is essential for evaluation. A number of such independent, objective evaluations were done earlier and reports were produced – but who consumed those reports? Nobody took ownership of evaluation findings. Now the donor community has realized the need for joint evaluations. OECD recommended joint evaluation methodologies to make sure that the government of the partner country participates in the evaluation. This helps not only to strengthen ownership but also to use the findings.
Do governments want only successful programmes to be evaluated?
VS: No. Unsuccessful and non-performing projects or problem projects are also evaluated. With regard to problem projects, I remember the Southern Transport Development Project that the Centre for Poverty Analysis (CEPA) was evaluating. This involved building the country’s first major expressway in Sri Lanka. It was a problem project. There were resettlement issues. If projects are delayed, sometimes the government wants to know what kind of challenges the project is facing. Sometimes innovative projects and those that are doing well are also evaluated – planners want to know how a project is successful and if it can be mainstreamed. There are certain criteria for the selection of projects for evaluation. The criteria may include concerns such as replicability, problem projects, innovative projects, projects that have policy connection.
What happens when there are negative findings or no impact?
VS: There is a general perception that if there are negative findings, treasury and government will reduce funding. There was the case in the production village programme – where each village produced one product. Common Service Centres were set up in the villages but some were not successful as there was no cost recovery mechanism and they were either closed down or scaled down because of poor management practices. Some of the production village projects were performing poorly. But the project was evaluated and additional funds sanctioned. So, negative evaluation findings were used to get additional money to support and address some of the important issues to strengthen the Project’s theory of change.
The flip side is that negative findings can be taken up by the media to criticize the government. Politicians in opposition can also use it to discredit the government. Hence, initially evaluation reports were confidential and circulation was restricted.
At what stage of the impact evaluation does the researcher involve the policymaker?
VS: I have seen some donors involve the government at the end when you disseminate the findings. Well, at that stage the government may not accept the findings. The best time to involve them is at the upstream, the very beginning when the nature and scope of the evaluation is discussed.
That way, the policymaker is aware of the evaluation questions necessary for the decision makers. He also gets involved in the scoping session and the entire process can be integrated into the planning and operational process. This way, even if there are negative findings, it is not a shock to the policymakers.
Dissemination workshops are important both upstream and downstream. Evaluation should address the needs of the government. Those lessons must be integrated into the operation and maintenance process – throughout the cycle. Even at the point of dissemination, it is important to have a wider group of policymakers for dissemination workshops. It is here that we inform the policymaker about what worked well, and what did not. Therefore, throughout the cycle, the policymaker needs to be involved.
Can you give examples of evaluations in Sri Lank that had an impact?
VS: There was the $30-40 million Aquaculture Development Project funded by the ADB. Hatcheries and breeding centres were set up to produce fingerlings in the backyard of the houses. The fish were reared and harvest for consumption. It was supposed to improve the nutritional status of the people. But half way through the project, a section of the Buddhist clergy objected. They claimed that killing fish for the purpose of eating did not go well with the Buddhist philosophy. At that time, the President decided the project was to be discontinued. The government then handed it over to the private sector. The private entrepreneurs entered the scene and began cultivating and exporting ornamental fish. So whereas the idea was to improve the nutritional levels of the people, it went on to achieve some other objectives.
An evaluation of the programme showed that it is important to take into account religious, cultural, social factors in the planning process.
There was also the case of the rural electrification project. This was originally meant to provide electricity to rural people. High tension power lines were set up in far flung areas but it was found that people did not have the purchasing power and continued to use the traditional kerosene lamps. An evaluation of the project showed that there were only 35-40 percent of connection rates and hence the financial rate of return was very low. So a soft component, a bank loan was added to help consumers to obtain power connections. Soon a rapid increase in connection rates was evident. In other words, the programme theory was improved and it became effective. Hence, a simple evaluation can change a programme, strengthen its results chain and contribute to its success.