2006年-世界发展银行全球_Measuring_Results___A_Review_of_Monitoring_and_Evaluation_in_HNP_Operations_in_South_Asia_and_Some_Practical_Suggestions_for_Implementation_41页_2mb

Summary of "Measuring Results: A Review of Monitoring and Evaluation in HNP Operations in South Asia"

Core Content

This report, prepared by the South Asia Human Development Sector (SASHD-HNP) in August 2006, evaluates the Monitoring and Evaluation (M&E) practices of Health, Nutrition, and Population (HNP) operations in South Asia. It aims to improve the effectiveness of M&E by identifying strengths, weaknesses, and areas for improvement, while also offering practical recommendations.

Main Objectives

• Improve the design and implementation of M&E in Bank HNP operations in South Asia.
• Increase the number of impact evaluations of HNP innovations and policies supported by the Bank.
• Assist client countries in monitoring and evaluating the performance of their health sectors.
• Strengthen the "results" culture among sector staff.
• Establish a baseline to assess progress in improving M&E over the next 2-3 years.

Key Findings

A. Selection and Definition of Indicators

• Indicators are often selected based on opinion, with little consensus among reviewers. The agreement on whether indicators were appropriate was no better than random (kappa = 0, p-value = 50%).
• Most indicators were "upstream", focusing on inputs and processes rather than outputs and outcomes.
• Kappa values for indicator appropriateness were low, suggesting a lack of agreement among reviewers.
• Median number of indicators per project was 23, with a range from 5 to 47. Half of the projects had too many indicators.
• 80% of indicators were defined in a measurable way, but only 33% had all five key elements: measurable definition, clear data collection method, explicit schedule, target, and responsible party.

B. Design of Data Collection

• Only 33% of indicators had all five key design elements.
• Household and health facility surveys were underutilized, despite their potential to provide important data on service access, usage, and quality.
• Control/comparison groups were rarely used, even when available, and their use has declined over time.
• Data collection methods were often not specified for a significant portion of indicators, with HMIS being the most common source.

C. Implementation of Data Collection

• Baseline data was collected for only 39% of the indicators studied.
• Only 25% of projects had satisfactory baseline data in their initial PSR/ISR.
• Follow-on data was collected for 47% of the indicators, but often not in line with the PAD schedule.
• Data collection plans were rarely implemented as designed, with only 25% of projects judged to have implemented their data collection plans.

Key Recommendations

• Task teams should routinely use an M&E checklist.
• M&E should be a central part of quality enhancement reviews (QERs).
• More technical assistance should be provided to task teams on M&E.
• Focused and practical training should be given to enhance M&E capacity.
• The first two aspects of M&E (building client capacity and conducting impact evaluations) need more attention.
• Management should provide clear and consistent messages about the importance of M&E.
• Task teams and managers should regularly review M&E during implementation.
• A senior staff member should be dedicated to supporting M&E activities and implementing the recommendations.

Conclusion

The study highlights the need for more systematic and consistent M&E practices in HNP operations. While some projects demonstrated good M&E practices, many lacked clear indicators, proper data collection methods, and implementation. The report emphasizes the importance of improving the design of M&E frameworks, increasing the use of control groups, and strengthening the capacity of both the Bank and its client countries to monitor and evaluate health sector performance effectively.