2016年-世界发展银行全球_Predicting_Project_Outcomes___A_Simple_Methodology_for_Predictions_Based_on_Project_Ratings_22页_1mb

Summary of "Predicting Project Outcomes: A Simple Methodology for Predictions Based on Project Ratings"

Core Content

This paper presents a new methodology for predicting the outcome of World Bank investment projects based on project ratings during supervision. The focus is on projects that are rated moderately satisfactory or higher but may still face a downgrade upon completion. The study is conducted by the East Asia and Pacific Development Effectiveness (EAPDE) unit and aims to improve on existing risk prediction systems.

Main Viewpoints

• Decline in Outcome Ratings: Since 2006, outcome ratings for World Bank investment projects have been declining, with the East Asia and Pacific (EAP) region experiencing a steeper decline. In FY12, EAP's outcome ratings dropped below the Bank-wide average.

• Candor Gap and Net Disconnect: The candor gap, the difference between the percentage of projects with satisfactory outcomes in the active portfolio and those rated satisfactory by IEG, has been a concern. The net disconnect, the difference between IEG and regionally reported unsatisfactory outcomes, has increased and remained higher than the Bank average.

• Limitations of Existing Tools: Traditional monitoring tools like disbursement tracking and ISR ratings for implementation progress (IP) and development outcome (DO) are not effective at predicting downgrades. These tools often fail to capture the full picture due to over-optimism or lack of candor in self-reporting.

• Recent Studies: Previous studies (Denizer, Kaufmann, and Kraay 2011; Geli, Kraay, and Nobakht 2014) have identified significant predictors of project outcomes, such as CPIA scores and task team leader track records, but these models still only predict about 40% of unsatisfactory outcomes.

Key Information

Project Analysis Methodology

• The study analyzed 62 investment projects that exited the EAP portfolio in FY12 and FY13, using IEG's Independent Country Review (ICR) data.
• The analysis focused on projects with unsatisfactory outcomes and particularly those with disconnects between IEG and regional ratings.
• A convenience sample was used, including all projects with commitment amounts over $5 million.

Predictive Model Development

• A simple prediction model was developed based on the incidence of warning indicators (flags) during the project's life.
• The model identifies two groups of projects at risk of unsatisfactory outcomes:
  1. Projects currently rated moderately unsatisfactory or lower for IP or DO.
  2. Projects not currently rated unsatisfactory but with at least three of six indicators rated moderately unsatisfactory or lower at any point in their life.
• These groups form two modules in the proposed model: the "problem project module" and the "flag-based module."

Validation of the Model

• The flag-based module was validated using a sample of 62 EAP projects and 531 Bank-wide projects.
• The model achieved an ex post prediction rate of 76% for EAP projects and 68% for Bank-wide projects.
• This represents a significant improvement over existing methods and highlights the value of real-time ratings and indicators in predicting outcomes.

Indicators and Their Impact

• The six indicators analyzed include: project management, procurement, M&E, safeguards, counterpart funding, and financial management.
• These indicators are critical dimensions of project implementation and show strong correlation with unsatisfactory outcomes.
• Projects with unsatisfactory ratings in M&E and project management are particularly at risk of poor outcomes.

Key Findings

• A significant percentage of projects with disconnects had unsatisfactory ratings in at least three of the six indicators.
• Projects rated moderately unsatisfactory for DO or IP are more likely to receive an unsatisfactory rating at exit.
• The model provides a practical and effective tool for identifying projects at risk of downgrading, even if they are currently rated moderately satisfactory or higher.

Conclusion

The paper introduces a simple, yet effective, prediction model based on project ratings and real-time indicators. It addresses the limitations of existing monitoring systems and provides a better way to identify projects that may fail to meet their development objectives. The model has been validated on both EAP and Bank-wide samples and is expected to enhance the Bank's ability to manage and predict project outcomes.