IDB's Projects & Operations Evaluation: Guidelines


A. Background

In fiscal year 1426H (2004/2005), the Operations Evaluation Office (OEO) revamped and significantly enhanced the Basic Operations Evaluation Guidelines, which will be applicable to all the IDB Group windows of financing, based on the valuable experience gained by the office in the past 15 years of its existence, and the lessons learned from OEO’s interactions and continuous collaborative efforts with the other multilateral development financing institutions. As compared to the first evaluation guidelines, issued by the Bank in November 1993, the main change in the new version is its introduction of an elaborate discussion of the criteria for assessing more appropriately the performance/success of the operations or projects post-evaluated. Instead of a single overall assessment, a multi criteria-based performance rating has been adopted in order to make such an evaluation more objective, transparent and less controversial.

 


B. Main Purpose of the Operations Evaluation Guidelines

  1. To be a reference book and a guide to all the staff of the Operations Evaluation (OEO) in fulfilling their duties.
  2. To enable new staff who will join the OEO to better understand how to carry out their new work.
  3. To give other staff of the Bank a good idea as to how operations evaluation activities are performed in IDB.

 


C. Need for Revising IDB’s Basic Operations Guidelines

 

 

D. Salient Features of the Revised Guidelines:

(1) Rating Criteria: Elaborate discussion of the criteria for assessing more objectively the performance or success of an operation/project:

 

(2) Emphasis on Proper Desk Review:

 

Consequently, four broad performance categories for completed projects, i.e., Highly Successful, Successful, Partly Successful and Unsuccessful have been retained, as mentioned earlier. After determining the most appropriate number for the main performance categories, it is suggested to accord specific points, ranging between a maximum of 20 to a minimum of 6 points, to determine the category to which the performance of the operation/project post-evaluated belongs. These points will, in turn, be distributed over five main criteria for evaluating project performance, namely, Relevance, Implementation, Effectiveness, Efficiency and Sustainability.

 

As indicated above, another important area which was greatly emphasized by these revised guidelines was devising an enhanced mechanism or preparatory work on the operation/project scheduled for post-evaluation before the Operations Evaluation Officer proceeds to his field visit so that he will have a full grasp of the project and will find the most pertinent issues to raise and focus on during the mission. For this reason, he is expected to make a thorough desk review before going on the post-evaluation mission by studying all the relevant documents in the Bank and to have good discussions with the relevant officers in the departments and offices concerned (i.e., the Operations Complex, Finance Department, Legal Department, Special Assistance Office, etc.). Based on this thorough desk review and discussion, the Operations Evaluation officer has to prepare a Blue-print for his coming evaluation mission to be cleared by the Head of the OEO.


Criteria for Project Performance Assessment

Overall Project Rating Category Rating Range

Category Rating  Range
1.   Highly Successful (HS) Overall maximum  marks are  greater than 17 points [i.e., HS =>17]
2.   Successful (SU)      Maximum marks are between 13 and 16 points [13 <= S <= 16]
3.   Partly Successful (PS) Maximum marks are between 7 and 12 [7 <= PS <= 12]
4.   Unsuccessful (US)       Maximum marks are less than 6 points [US <= 6]

 

 

Maximum Possible Marks for Project Performance Assessment

Criterion Assessment Rating Value Maximum
Possible Marks
1.  Relevance Highly Relevant
Relevant
Partly Relevant
3
2
1
3
 2. Implementation Highly Satisfactory
Satisfactory
Less Satisfactory
Unsatisfactory
4
3
2
1
4
 3.  Effectiveness Highly Effective
Effective
Less Effective
Ineffective
4
3
2
1
4
 4.  Efficiency Highly Efficient
Efficient
Less Efficient
Inefficient
Highly Inefficient
5
4
3
2
1
5
 5.  Sustainability Most Likely
Likely
Less Likely
Unlikely
4
3
2
1
4
Maximum Total Marks 20


Specific Points to be Accorded to Criteria for Project Rating


(a) Relevance: One point is to be accorded for each of the following items:

  1. Project is in agreement with the member country's developmental priorities (e.g., sector given priority in the country's development plan).
  2. Project is in consistent with IDB Group's strategic agenda.
  3. Project has a good market (or is needed by a significant portion of the country's population or a region/province).

 

(b) Implementation: One point for each of the following items:

  1. Procurement was according to IDB Groups policies & procedures (and no significant deviations from there).
  2. The greater part of the project's physical objectives (or more than 75%) was implemented (in terms of civil works and equipment).
  3. Performance of the contractor(s)/consultants was satisfactory.
  4. Performance of the executing agency/IDB was satisfactory.

 

(c) Effectiveness: One point for each of the following items:

  1. Design was well done and suited to the project.
  2. All projects operational targets (or at least 75% of them) were achieved.
  3. Project imparts marketable skills or provides affordable services to the local population.
  4. Adequacy of costs in comparison with related benefit streams the latter should also have reached a reasonable level and grown at a sound rate.

 

(d) Efficiency: One point to be accorded for each of the following items:

  1. Project was completed on time (no time over-run exceeding 1 year).
  2. Project was completed within budget (i.e., no cost over-run exceeding 25% of original estimate).
  3. Projects scope was implemented without major change.
  4. Project follows good practice standards for the service it provides.
  5. Capacity utilization of the project was at least 75% of its original design capacity.

 

(e) Sustainability: One point for each of the following items:

  1. Project has enough, sufficiently trained staff to run it.
  2. Project generates enough income (or gets enough financial support from the government of the member country).
  3. Project has good maintenance arrangements.
  4. Project had no adverse effect on the environment.

 

Back