MASH Project Foundation organized a Workshop on Measuring Impact: Evaluating Development Programs Through Mixed Methods Research on Friday, 24th July 2020,
The workshop was designed for college students, researchers and working professionals who are looking to understand evaluations and impact measurement in the development sector landscape.
The workshop seeked to answer 3 broad questions – Why we evaluate, What we evaluate and how we evaluate in the context of impact measurement in development programs. It delved into how impact within a development program can be visualized using the Theory of Change. It will focus on every step of evaluating development programs- data collection, developing the evaluation framework, mixed methods research, and data triangulation.
The workshop was conducted by Ms. Rai Sengupta – Rai is a development consultant and a national level keynote speaker. A graduate of Economics from Shri Ram College of Commerce, University of Delhi, she currently works with an international development consulting firm based out of New Delhi. Rai is part of national-level teams evaluating schemes implemented by the Ministries of Women and Child Development, and Rural Development – on behalf of NITI Aayog, Government of India. She was recently part of a three-member technical team evaluating a maternal and child cash transfer programme in Myanmar, on behalf of UNICEF.
Rai introduced the workshop by stating that the workshop will help the participants learn about research methodology and impact measurement. She began the session by questioning the participants whether evaluation is an old or a recent phenomenon? She explained that ancient Egyptians evaluated grain and livestock production 5000 years ago. She further added that evaluation of education and social programs began in the 1800s in several Anglo-Saxon countries. About 50 years ago, the need for evaluations increased after World War 2.
She proceeded by asking another question – What are the reasons for measuring impact to which the participants responded with answers such as to check progress, track efficacy and efficiency, to strategize, learn from existing practices or to understand the program better.
Rai explained the 4 reasons to evaluate are :
- Decisional purposes – to pave the way for decisions on the continuation, termination or reshaping of a policy or programme.
- Managerial purposes – to achieve a more rational distribution of resources.
- Ethical purposes – to learn from past mistakes and to bring transparency and accountability into the system.
- Educational purposes – to educate, motivate and check the operational efficiency of public agencies for better operations in the future.
Rai decoded the concept of impact by putting up few questions:
- How do you define impact?
- Is impact the same as outcomes? If no, then what could be the possible difference?
- Is the impact of an intervention the same as the outputs of an intervention?
- How does one measure impact?
She answered the above questions by explaining the concept of Theory of Change which can be understood as a description of how and why a desired change is expected to happen in a particular context. She supplemented the theory with examples for better understanding.
Theory of Change of a programme is important to be known for the programme to be successful.
She explained that the framework to understand the Theory of Change is as follows – Inputs – Outputs – Outcomes – Impact. Outcomes & outputs are short term while impact is the long term overall accumulation of the outcomes. One input can have multiple outputs/outcomes/impacts.
Theory of Change, also known as the Log frame analysis, describes:
- Inputs as the resources required for a programme or policy.
- Outputs as the tangible products as a result of the inputs.
- Outcomes as the behavioral changes that result from the outputs.
- And, impact as the long-term changes that derive from an accumulation of outcomes.
Rai put up a quiz question for the attendees to develop a better understanding of the Theory of Change. She took help of a few examples to explain the indicators of input, output, outcome and impact. She further shared the case study of the Maternal and Child Cash Transfer(MCCT) Programme implemented in Chin and Rakhine States of Myanmar. Along with the reasons to implement, she described the interventions used for the success of the programme as well. In the first phase of the intervention, the Government of Myanmar provided monthly cash transfer of 10 USD to all pregnant and lactating women which continued till the child reaches 24 months of age. In the second phase, every beneficiary was entitled to be a member of a Mother Support Group(MSG) and to attend monthly/quarterly SBCC sessions on nutrition, health and hygiene. After this, Rai gave the inputs for the MCCT programme and the participants were able to apply the theory and fill up the blanks of output, outcomes and impact correctly.
Taking the two phases of intervention as the inputs, other indicators can be derived as:
- The outputs can be mothers having greater purchasing power and greater awareness of health practices.
- The outcomes can be mothers purchasing nutritional food for themselves and their child and following healthy child rearing practices.
- The impact can be a decline in the rate of child malnutrition in Myanmar.
An importance of the Theory of Change is that it helps us to differentiate between the impact and output/outcome of an intervention. A formative evaluation looks into the ways in which a program, policy or project is implemented whereas a summative evaluation determines the extent to which anticipated results were realized. Formative evaluation measures only three of all the four indicators whereas summative evaluation can measure all of the four indicators.
To judge the type of evaluation to be used –
If we need to measure the long term impact then only summative evaluation can be used whereas formative can be used for process evaluation. After explaining the topic, she took up a quiz question to check the knowledge level of the attendees on the above explained concept.
Methodology of evaluation follows 6 steps –
Step 1. Start with the principles: This step involves mixed-methods research, participatory approach, theory focused and utilization focused.
Step 2. Map the stakeholders: Brainstorming and mapping out stakeholders who affect or are affected by the intervention to gain an insight on how the intervention is progressing.
They can be the developers of the intervention, funders, implementing agencies, on-ground staff, direct beneficiaries, indirect beneficiaries, general public and potential adopters of the intervention.
Case Study – Rai further discussed the stakeholders of the MCCT Programme as the Government of Myanmar, pregnant and lactating women, implementing officials who disburse the cash, husbands and other family members, midwives who provided SBCC training sessions, and members of the community.
Step 3 . Choose your tools: The next step talks about how to get the required information from the stakeholders. This can be done by choosing the appropriate tools.
Primary data is directly collected through surveys, discussions, interviews etc., while the secondary data is indirectly taken up from the internet, libraries, reports, online portals, etc.
Case study –
Primary data can be either qualitative or quantitative.
Qualitative data includes:-
- Key informant interviews for policy makers
- Focus group discussions for midwives
- In-depth interviews for implementing staff
Quantitative data includes:-
- Beneficiary survey for the pregnant/lactating women
- Household survey for the husband/family members/community members
Secondary data consists of:
- Programme documents such as programme guidelines
- Sectoral reports and studies on past evaluations in the maternal and child nutrition space.
- Data from the Myanmar Information Management Unit
Step 4. Design your tools with sensitivity: Points to remember while designing the data collection process –
- Training of enumerators on ethical standards and guidelines
- Interactions with all stakeholders with prior consent
- Informing participants of their rights to anonymity
- Obtaining the consent of parents for interviewing children
Case study – In the context of the MCCT Programme, data collection would have to be sensitively conducted given the threat of ethnic conflict and armed violence in Rakhine State.
Step 5. Triangulate findings: Data triangulation refers to the process when data on the same topic is collected from multiple sources and by different methods. This helps to reduce respondent bias.
Case study – In the example of MCCT Programme, to understand if the cash transfer is effectively taking place, cross-validation of the data can be done by asking beneficiary women and their family members about the cash received every month and the government officials about the cash given out every month.
Step 6. Build conclusions and inferences: Combining insights from various data sources to form conclusions.
Case study – In the Myanmar case study, to understand if the cash transfer leads to women empowerment, triangulation of various insights can be done to make it clear that pregnant/lactating women do not have a role in the usage of the cash received. The mother-in-laws play a key role in determining the cash usage. These conclusions are supported by secondary literature.
Rai continued the quiz questions to judge if the attendees have been able to grasp the knowledge imparted through the workshop. Finally, she wrapped up the workshop with some key points to be remembered while measuring impact. Sauhard Chaudhary from team MASH, concluded the workshop on the note that the Theory of Change as well as the six steps of research methodology can be easily understood through case studies and each one of us can use these in our day-to-day research projects.