IDB's Projects & Operations Evaluation: Guidelines

A. Background

In fiscal year 1426H (2004/2005), the Operations Evaluation Office (OEO) revamped and significantly enhanced the Basic Operations Evaluation Guidelines, which will be applicable to all the IDB Group windows of financing, based on the valuable experience gained by the office in the past 15 years of its existence, and the lessons learned from OEO’s interactions and continuous collaborative efforts with the other multilateral development financing institutions. As compared to the first evaluation guidelines, issued by the Bank in November 1993, the main change in the new version is its introduction of an elaborate discussion of the criteria for assessing more appropriately the performance/success of the operations or projects post-evaluated. Instead of a single overall assessment, a multi criteria-based performance rating has been adopted in order to make such an evaluation more objective, transparent and less controversial.

B. Main Purpose of the Operations Evaluation Guidelines

To be a reference book and a guide to all the staff of the Operations Evaluation (OEO) in fulfilling their duties.
To enable new staff who will join the OEO to better understand how to carry out their new work.
To give other staff of the Bank a good idea as to how operations evaluation activities are performed in IDB.

C. Need for Revising IDB’s Basic Operations Guidelines

The valuable experience gained by the office in the past 14 years of its existence (first guidelines were issued in November 1993).
Lessons learned from our close collaboration with other multilateral development financing institutions, like the World Bank, African Development Bank, Asian Development Bank, and Inter-American Development Bank.
The expanded scope of OEO’s work in accordance with both the IDB Management's directives and the recommendations made by the Audit Committee of the Board of Executive Directors

Although these Basic Guidelines focus mainly on the post-evaluation of individual operations and projects, other guidelines are currently being developed by the OEO for: (a) Country Assistance Evaluation (CAE) and (b) Technical Assistance. Guidelines for Sector and Thematic Studies will then follow

D. Salient Features of the Revised Guidelines:

(1) Rating Criteria: Elaborate discussion of the criteria for assessing more objectively the performance or success of an operation/project:

Instead of the old 3 categories, 4 broad categories have been chosen, namely: Highly Successful, Successful, Partly Successful and Unsuccessful.
To determine the category to which a particular project belongs, and to make the assessment more objective and less subjective, specific points, ranging between 6 and 20, are accorded to the project.
These specific points are distributed over 5 main criteria, namely: Relevance, Implementation, Effectiveness, Efficiency and Sustainability.

(2) Emphasis on Proper Desk Review:

Importance of making proper study of the project scheduled for post-evaluation before the Operations Evaluation Officer proceeds to his field visit to fully understand the project and find the most pertinent issues to focus on during the mission.
He is expected to make a thorough desk review by studying all the relevant documents in the Bank and by having good discussions with the relevant officers in the departments and offices concerned (i.e., the Operations Complex, Finance Department, Legal Dept,. Special Assistance Office, etc.)
Based on this thorough desk review and discussions, the Operations Evaluation Officer has to prepare a Blue-print for his coming evaluation mission to be cleared by the Head of the OEO. Otherwise, he can not proceed to his field mission. (The form of this Blue-print is one of the annexes to the Guidelines).

Consequently, four broad performance categories for completed projects, i.e., Highly Successful, Successful, Partly Successful and Unsuccessful have been retained, as mentioned earlier. After determining the most appropriate number for the main performance categories, it is suggested to accord specific points, ranging between a maximum of 20 to a minimum of 6 points, to determine the category to which the performance of the operation/project post-evaluated belongs. These points will, in turn, be distributed over five main criteria for evaluating project performance, namely, Relevance, Implementation, Effectiveness, Efficiency and Sustainability.

As indicated above, another important area which was greatly emphasized by these revised guidelines was devising an enhanced mechanism or preparatory work on the operation/project scheduled for post-evaluation before the Operations Evaluation Officer proceeds to his field visit so that he will have a full grasp of the project and will find the most pertinent issues to raise and focus on during the mission. For this reason, he is expected to make a thorough desk review before going on the post-evaluation mission by studying all the relevant documents in the Bank and to have good discussions with the relevant officers in the departments and offices concerned (i.e., the Operations Complex, Finance Department, Legal Department, Special Assistance Office, etc.). Based on this thorough desk review and discussion, the Operations Evaluation officer has to prepare a Blue-print for his coming evaluation mission to be cleared by the Head of the OEO.

Criteria for Project Performance Assessment

Overall Project Rating Category Rating Range

Category	Rating Range
1. Highly Successful (HS)	Overall maximum marks are greater than 17 points [i.e., HS =>17]
2. Successful (SU)	Maximum marks are between 13 and 16 points [13 <= S <= 16]
3. Partly Successful (PS)	Maximum marks are between 7 and 12 [7 <= PS <= 12]
4. Unsuccessful (US)	Maximum marks are less than 6 points [US <= 6]

Maximum Possible Marks for Project Performance Assessment

Criterion	Assessment	Rating Value	Maximum Possible Marks
1. Relevance	Highly Relevant Relevant Partly Relevant	3 2 1	3
2. Implementation	Highly Satisfactory Satisfactory Less Satisfactory Unsatisfactory	4 3 2 1	4
3. Effectiveness	Highly Effective Effective Less Effective Ineffective	4 3 2 1	4
4. Efficiency	Highly Efficient Efficient Less Efficient Inefficient Highly Inefficient	5 4 3 2 1	5
5. Sustainability	Most Likely Likely Less Likely Unlikely	4 3 2 1	4
Maximum Total Marks			20

Specific Points to be Accorded to Criteria for Project Rating

(a) Relevance: One point is to be accorded for each of the following items:

Project is in agreement with the member country's developmental priorities (e.g., sector given priority in the country's development plan).
Project is in consistent with IDB Group's strategic agenda.
Project has a good market (or is needed by a significant portion of the country's population or a region/province).

(b) Implementation: One point for each of the following items:

Procurement was according to IDB Groups policies & procedures (and no significant deviations from there).
The greater part of the project's physical objectives (or more than 75%) was implemented (in terms of civil works and equipment).
Performance of the contractor(s)/consultants was satisfactory.
Performance of the executing agency/IDB was satisfactory.

Design was well done and suited to the project.
All projects operational targets (or at least 75% of them) were achieved.
Project imparts marketable skills or provides affordable services to the local population.
Adequacy of costs in comparison with related benefit streams the latter should also have reached a reasonable level and grown at a sound rate.

(d) Efficiency: One point to be accorded for each of the following items:

Project was completed on time (no time over-run exceeding 1 year).
Project was completed within budget (i.e., no cost over-run exceeding 25% of original estimate).
Projects scope was implemented without major change.
Project follows good practice standards for the service it provides.
Capacity utilization of the project was at least 75% of its original design capacity.

(e) Sustainability: One point for each of the following items:

Project has enough, sufficiently trained staff to run it.
Project generates enough income (or gets enough financial support from the government of the member country).
Project has good maintenance arrangements.
Project had no adverse effect on the environment.

Back