n

Program evaluation

30/12/2013 CATHOLIC RELIEF SERVICES - CRS
Print Friendly, PDF & Email

Program evaluation

Purpose:
The purpose of thisScope of Work (SOW)   outlines the conditions and responsibilities of the consultant to conduct a qualitative and quantitative mid-term evaluation of CRS/Mali’s USDA funded International Food for Education and Child Nutrition program, implemented by Catholic Relief Service.  The evaluation period is November 4, 2013- December 16, 2013
Background:
In October 2011, CRS began implementing Phase II of the USDA funded International Food for Education and Child Nutrition program managed through the Foreign Agriculture Service (FAS).  CRS/Mali has been awarded commodities and funding by USDA in response to recurrent food crises in the Sahel region, high levels of malnutrition, and low and inequitable levels of educational access and quality.  Of Malians aged 15 and above, 26 percent are literate, and the vast majority of these literate adults reside in urban and peri-urban areas.  With such extremely high illiteracy rates among adults, it is often a challenge for illiterate parents to be involved in their children’s learning and school performance. Additionally, households are largely dependent upon subsistence agriculture, and malnutrition is high in the target area, due to diet, poor hygiene practices, and illness.
This project aims to address needs in the areas of health, nutrition and education through school feeding, take-home rations for girls and nomads (both boys and girls), health training and nutritional supplements (Vitamin A and de-worming), and capacity building in the schools and communities through trainings and Savings and Internal Lending Committees (SILC) to secure quality education of the school children in the targeted regions. To ensure access to food aid and promote higher quality education, CRS/Mali’s education interventions aim to:
Achieve higher enrollment and attendance rates with emphasis on girls and nomads through monthly take home rations and performance prizes to girls.
Improve student achievement by providing report card formats that are accessible to illiterate parents and that encourage them to become more involved in their child’s education
Improve, children’s health and nutrition through PD/Hearth sessions and student health and nutrition through Vitamin A and de-worming distributions
Build capacity for communities to undertake development projects and improve the school infrastructure
Train women‘s SILC groups to generate income for women and increase household savings and access to credit enabling caregivers to pay for school fees, contributions to school canteens, supplies, clothes, shoes and purchasing local nutritious foods for household consumption.
The project seeks to achieve these objectives through implementation of activities that span education, health, capacity building, and commodity sectors, with cross cutting emphasis on gender and livelihoods. All schools in the GFE program benefit from:
– school meals (distribution of hot lunch)
– systematic deworming and micronutrient supplements (Vit A and Iron),
– training for Canteen Committees (CGS) and  teachers in health, hygiene, nutrition
– training for CGS on roles and responsibilities, micro-planning, resource mobilization, management of the community-run canteen, health, hygiene, nutrition, girls education, etc
The program implements additional activities in selected schools that include:
– a report card format  for girl students (to increase girls attendance and success rates)
– Saving and Internal Lending Communities
– PD/Hearth
– Distribution of performance prizes to one student from each grade level (recognizes children who have had the greatest improvement in school participation or academic achievement or recognized as a top performer).
The baseline survey was conducted by CRS in December/Jan 2012, and data was collected through interviews, focus group discussions, observations, and anthropometric measurement. Program participants were randomly selected (students, teachers, households, School Management Committee members, members, caregivers, and school directors) from 100 of the 310 participating schools (32%).  An education team surveyed 1,881 households (884 in Koulikoro region and 997 in Mopti region); measurements were taken from 469 children 6-36 months (244 in the Koulikoro region (132 boys and 112 girls) and 225 from the Mopti region (125 boys and 100 girls). Lastly, the micro-finance component collected data from 1350 mothers (874 from Koulikoro region and 476 from Mopti region) of school- aged children participating in the Food for Education program. Due to a teachers’ strike for salary increases, the surveys were conducted in two phases, from December 12-22, 2012 and from March 19-26, 2012. The primary objective of the baseline survey was to define the current level of knowledge and practice related to several USDA and CRS monitoring indicators.
Objectives of the Midterm Evaluation:
The primary objective of the qualitative and quantitative midterm evaluation will be to determine (1) the relevance of all interventions, (2) the effectiveness of implementation strategies and activities, (3) the efficiency of the project (4) the achievements of the interventions to date  (5) and the possibility for sustainability of initiated actions..  Furthermore, these objectives will assess the programs delivery of project outcomes and based on this assessment take decisions on the future orientation of the project. The mid-term will evaluate the full scope of the programming including partner and community structures. Data will be collected by reviewing CRS documents and comparing them to, interviews, focus group discussions, and observations from a random sample of schools.
Relevance
Relevance is defined by the extent to which the project activities are suited to the priorities of the target group recipient and donor. Key questions to address are:
Are the activities and outputs of the project consistent with the overall goals and the attainment of its objectives? (Based on literature review)
Does the program meet communities and government priorities? (Based on a literature review and interviews with government and community representatives)
Are stakeholders (management committee, parents, teachers, local authorities and partners) satisfied with their participation in the program?  Why or why not?
Are program targets emphasizing on the most vulnerable populations?
Effectiveness
Effectiveness is a measure of the extent to which project activities attain its objectives. Key questions are:
According to available project records, and qualitative data collected, identify to what extent  project  objectives  and the yearly benchmark indicators are likely to be achieved?
The project has 17 key indicators that will be assessed in addition to pilot program activities indicators. They include the following:
Number of pupils enrolled in  project supported schools ;
Number of children promoted to the next grade level at the end of the 2012 school year ;
Number of children enrolled in the 2012 school year that continued in enrollment in 2013 ;
Number of children who are not attending school.  Definition: the number of school age children in a community, who do not attend school, including both those children who are enrolled and not enrolled compared to the number of children attending.
Percent of children in sample villages who are attending USDA supported classrooms/schools;
Number of  USDA supported schools implementing health and nutrition education for students;
Percentage of government supported schools implementing health and nutrition education for students
Number of schools with established Vitamin A and de-worming programs;
Number of children who received  Vitamin A and de-worming medication in project supported schools*;
Number of school health and nutrition sessions held with adult community members;
Number of functioning PD/Hearth groups according to  intervention criteria
Number of functioning SILC groups  according to intervention criteria
Number of Committee de Gestion Scolaire (CGS), Parent Teacher Associations (PTAs) or  similar “school” governance structures contributing to their school as a result of USDA assistance;
Number of Committee de Gestion Scolarie (CGS) that are functioning and providing support to the canteen ;
Number of school gardens that have been established as a result of USDA support;
Number of  school cantines and warehouses that have been constructed or rehabilitated as a result of USDA support;  and
Number of CGS/SILC who provide stipulated level of contribution (e.g fuel, cooking implements, condiments, ingredients etc.
Are the implementation strategies relevant and effective enough to improve: 1) enrolment and attendance among pupils, particularly girls?  2) Community participation and engagement? 3) A better learning environment? Are there other strategies which would work better and had more impacts?
Are there some internal and /or external factors that hindered the efficient implementation of project activities to this point?
Is the management system effective? Is the coordination mechanism for the partnership effective?
Has program implementation been effectively monitored? How well have the monitoring and evaluation mechanism in place helped the implementation of the project thus far?
Are food rations appropriate and delivered in a timely fashion? Is food loss minimized and properly reported?
Efficiency
Efficiency measures both qualitative and quantitative out-puts in relation to inputs. It assesses the extent to which project uses the least costly resources possible in order to achieve the desired results. Key questions to address are:
Are activities cost efficient?
Are the objectives achievable in the proposed timeline?
Is the project being implemented in the most effective way compared to alternatives?
What is facilitating or hindering the project to achieve the results and objectives in a timely manner?  What are the project’s main successes in implementation?
What are the project’s main challenges? How can the challenges be addressed?
Is the staffing structure and capacity sufficient and appropriate for effective implementation?
Is the coordination mechanism for the partnership efficient?
Does the food supply chain including transport and storage minimize loss and damages?
Impacts
Impacts are positive and negative changes produced by the intervention directly and indirectly intended or unintended. This involves changes impacts and effects on the local social economic, environmental and other development indicators.  Key questions are:
Have there been changes in children’s attendance, particularly girls, as reflected by enrollment /promotion /retention rates (actual) against projections? (see above indicator list)
To what extent have the inputs/activities (teachers trainings, infrastructures, awareness-raising, SILC, performance prizes, PD/HEARTH , report card format—questions annexed below) contribute to quality education/learning?
To what extent have capacity building activities for school management committees improved their capacities?
Are there behavioral and social changes in the form of improved understanding of the importance of education, health and nutrition, improved community participation in school development, and improved hygiene and sanitation practices in schools, etc. as a result of the project interventions?
What have the general contributions of the project been toward the educational development in Koulikoro and Mopti regions?
Are intended impacts likely to be long lasting?
What impact has CRS had on partners? Do the partners feel that their capacity has improved since they started working with CRS? Is their evidence that their capacity and ability to provide quality programming has improved?
Sustainability
Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Key questions are:
What activities and/or outcomes (both expected and unexpected) of the program are likely to be sustained? What evidence is there to suggest this?
What are the major factors which can influence the achievement or non-achievement of the sustainability of the project?
How do the government’s capacities, policies, procedures, and priorities contribute to sustainability?
What strategies should be used to obtain long lasting support from communities and local/central administration that goes beyond the time of the project?
Prospects:
What are the key obstacles for CGS becoming able to sustain the school cantine program?
What are the key issues affecting enrollment, attendance and continuation of boys and girls (separately)?
What are the key issues affecting the quality of education for boys and girls (separately)?
What are the key issues affecting the educational achievement and promotion of boys and girls (separately)?
What role do CGSs have in ensuring the quality of education?
Which local entities have the most influence on the quality of education?
Deliverables
The Survey consultant (consultancy team) will coordinate, organize and implement all aspects of the mid-term evaluation in November/December 2013.    The data collected will include, interviews, focus group discussions, and observations.  A thorough data review will be necessary to assess progress on indicators to date.  A final report including the main findings of the analysis as well as recommendations to strengthen current interventions will be expected.  The mid-term report should provide an accurate account of the program situation in the surveyed areas in order to assist with the intervention of planning and decision-making.
Roles and Responsibilities
Development and Methodologies
Review the following program key documents
Introductory Statement of FFE
Approved project documents
Baseline report
M&E plan and progress reports
Monitoring reports
SILC Methodology
Any other documents that may be requested by consultant.
Develop evaluation methodology/instruments/tools and submit them electronically to CRS for review
Review and finalize proposed evaluation methodology, tools and action plan
Meet with key CRS/ Mali staff for orientation on the program and discussion on logistics, evaluation methodology, instruments/tools, and timelines
2) Management and follow-up of the activities
Organize and supervise the evaluation taking into account the context and the different actors in the areas surveyed i.e CAP, Mayors etc.
Organize the implementation strategy for the evaluation, manage survey team and ensure that roles and responsibilities are clearly defined and attainable.
Coordinate with the logistics department to ensure a smooth implementation of the activities in the field (transport, supplies and materials and inputs needed etc…)
Evaluation and report
Provide the Food for Education Program Coordinator and CRS’s Head of Monitoring and Evaluating with all relevant information related to the implementation of the evaluation and data collected.
Produce a  final report (in both English and French) using  the following Guidelines
Executive summary
Background
Overview of Project Strategies
Project History and Operating Context
Evaluation Methodology
Evaluation purpose and objectives
Intermediate Results under all strategic objectives
Brief description of interventions
Service delivery strategies and approaches – quality, successes and challenges
Implementation progress and achievement of results
Meeting targets
Other achievements
Design and discussion of general evaluation questions
Program Quality and Crosscutting Areas
Partnership Targeting
Sustainability/ Exit Strategies
Implementation Processes
Commodity management
Monitoring and Evaluation
General Management
Recommendations
Critical Priority Recommendations
Other Recommendations
Annexes
List of acronyms
Evaluation SOW
Evaluation Plan and Schedule
Composition of the team
Evaluation Methods and  tools (topical outlines and survey questionnaire)
List of sites visited
List of key informants and communities visited
Updated Indicator performance tracking tables (IPTT)
Disclosure of conflict of interest
Statement of differences
Summary Table of Commodities
The tentative timetable is as follows: 

Tasks

Dates

Desk Review Of Program And Other Background Documents; Development Of Evaluation Methodology Instruments/Tools; And Electronic Submission To CRS For Review

3 days

Travel to Bamako

2 days

Meeting With CRS/Mali Staff In Bamako  For  Orientation/Discussion On Methodology/Logistics/Schedules

1 day

Select Evaluation Team, train and Beta testing

3 days

Travel To Koulikoro  For Field Work

1 day

Execution Evaluation Field Work (Koulikoro)

7 days

Travel Back to Bamako

1 day

Travel To Mopti  For Field Work

1 day

Execution Evaluation Field Work (Mopti)

7 days

Data Analysis

7 days

Travel Back To Bamako

1 day

Presentation Of Evaluation Findings To The CRS/Mali  Staff In Bamako /Overnight Return International Travel

1 day

Writing Up Of Draft Evaluation Report/Submission To CRS/Mali For Review And Comments

2 days

Finalization Of Final Draft (Incorporating Feedback After Final Review By CRS/SL)

5 days

Electronic Submission To CRS /Mali Of A Final Evaluation Report

1 day

Partagez : Facebook Twitter Google Plus LinkedIn Email

Autres articles