Monitoring and Evaluation

4.1 Scope

This section discusses the processes and procedures for the monitoring and evaluation of PIDS research studies. This project monitoring and evaluation system covers studies centrally-managed by PIDS, whether done in-house or sub-contracted to outside proponents. Monitoring and evaluation guidelines applicable to projects executed through specially created project management offices are discussed in the next chapter of this operations manual.

This section describes the information that shall be provided by the research proponents to PIDS to report on the progress of project implementation. These information are contained in progress and technical reports which are discussed below. In addition, this section describes the actions to be taken by PIDS management, particularly, the Research Services Department upon receipt of these reports. These actions include the technical review/evaluation and acceptance of the reports; remedial measures that may have to be carried out to address project implementation problems; and processing of requests for progress or final releases of project funds.

4.2 Objectives

The project monitoring and evaluation system aims at providing PIDS research managers a continuous and regular flow of information on the implementation of its research projects. The objective of the system is to help ensure that PIDS research studies are progressing or completed on a timely basis and that their outputs are of the highest quality. Relatedly, the system is designed to alert management on problems and difficulties which may impede the timely delivery of outputs at their desired qualities. Finally, the system aims to facilitate the delivery of inputs and support services from the Institute to the proponents to expedite project implementation.

4.3 Processes and Procedures

4.3.1 Submission of Progress and/or Final Reports

Technical and financial reports constitute the major instruments of the PIDS research monitoring and evaluation system. This operations manual shall focus on the technical reports. Financial reports will are covered separately by the AFD operations manual.

Technical reports shall be prepared and submitted by the proponents to PIDS once, at the very least, every six months. Submission of reports may be required quarterly depending on the nature, scope, and scale of the project.

The technical reports shall consist of:

• A narrative report (of around 2-5 pages) which shall describe the activities carried out by the project and outputs produced during the quarter under review. The report shall compare actual outputs produced to outputs expected. Divergences between acutal and expected outputs, if any, will be explained. The report shall also discuss the problems encountered in the course of implementing the project and actions undertaken (or proposed to be carried out) to resolve these problems. Special attention shall be placed on actions that may have to be done by the Institute (extension of deadlines, realignment of, or additional, budgets; technical assistance, etc.) to rectify problems encountered. Finally, the narrative report will discusses the activities to be implemented during the next period and the outputs expected from them.
• The narrative report shall include the three duly accomplished forms described in the previous chapter of this manual. These forms present in summary and tabular fashion: a) a comparison of actual versus expected outputs (PIDS Form RSD-1); b) summary of problems encountered (or expected) and remedial actions proposed (PIDS Form RSD-2); and c) list of activities proposed for the next period and expected outputs (PIDS FormRSD-3).
• In addition, the narrative report shall be accompanied by copies of actual outputs produced (e.g. literature reviews, survey designs, questionnaires, drafts of technical reports or papers, etc.) to the extent feasible.

Immediately upon the start of a research study or project, the RSD shall establish a project data base which shall include, among others, a schedule of reports to be submitted by the respective proponent. This schedule shall be based on the timing specified by the contract, in the case of contracted-out studies; or on the proposal/workplan, in the case of in-house studies.

Fifteen days prior to the scheduled date of submission of a progress or final report, the RSD shall send a pro-forma letter to the proponent reminding him/her of the required submission. If no report is received during the scheduled date, the RSD shall send one calendar week thereafter a second letter to the contractor following up the submission of the report. This may be augmented by a telephone follow-up; however, a formal letter is necessary for documentation purposes.

4.3.2 Review of Progress Reports

The RSD shall be responsible for coordinating the review of all progress reports submitted to the Institute. In carrying out this task, it shall draw upon other resources available within the Institute particularly the services of the Research Pool. It may also engage experts from outside the Institute to assist in the review process.

The review process shall involve an assessment of: (a) the substantive and technical quality of the report and (b) conformity with the terms of reference stipulated in the contract in terms of schedule and required output.

Upon receipt of the progress report, the RSD shall forward a copy to the research coordinator or fellow responsible for the area of research under which the study falls. The forwarding memo shall contain a request for the coordinator or fellow to review substantively any technical submitted report that may have been submitted with the narrative report. If necessary and in consultation with the Vice-President, the RSD may also request a policy-maker or research from outside the Institute to comment on the submitted technical reports.

The RSD shall integrate the comments and suggestions of the reviewers on the technical report submitted by the contractor together with its own assessment of the other aspects of the progress report. The RSD will prepare a memorandum to the Vice- President summarizing its findings and recommendations.

If the report is found acceptable, the RSD shall recommend Institute acceptance and release of progress payments to proponents. It shall issue a letter to the proponent informing them of acceptance of the report and consequent release of project funds. A copy of the acceptance letter will be furnished to the AFD as part of the documentation for the undertaking progress payments. Note that there are other requirements for progress releases relating to the financial and administrative aspect of the project which will be handled by the AFD.

If there are substantive changes needed on the technical report, RSD shall forward a copy of the reviewer's report to the proponents and request them to respond to the comments. In similar manner, if certain administrative or other implementational changes are required, the RSD shall inform the proponent of these changes (e.g. extension of contract, etc.) and prepare, or cause the preparation of, the necessary documentation. If the situation so requires, the RSD may meet with the proponent to discuss and agree on corrective measures to be done to put the project back on track.

Finally, the RSD shall update its project files and data base. It shall also prepare a status report of the Institutes projects for submission to the VP and President of PIDS and the other PIDS Management Committee.

4.3.3 Submission and Review of Final Reports

4.3.3.1 Review Procedures

The procedures, processes, and guidelines for reviewing and accepting final reports follow broadly those for progress reports. However, the review and acceptance processes and procedures are much more rigorous and formal.

Upon receipt of the final report, the RSD shall commission reviews of the submitted output. Three sets of reviews shall be carried out.

The first two are substantive reviews to be carried out by the research coordinator/fellow responsible for the relevant program area and by an external evaluator. These reviews shall look at the technial aspects of the reports. In general, they will determine whether the research objectives have been successfully achieved. In particular, these reviews will examine whether the analytical framework adopted has sound theoretical foundations; whether the statistical methodologies used are appropriate and valid; and whether the research findings, conclusions, and recommendations reached and put forward are adequately supported by the analyses.

The third review will be carried out by a policy-maker. This review will focus primarily on the policy aspects of the research. This review will assess whether the policy recommendations made derive logically from the research findings. In addition, it will also indicate whether the policy recommendations are feasible from an operational viewpoint.

In all cases, the reviews shall give preliminary suggestions on whether the reports are of quality suitable for publication and the appropriate publication format.

The RSD shall integrate the results of these reviews and submit its findings to the Research Review Committee.

4.3.3.2 Research Review Committee

There shall be a Research Review Committee to be chaired by the Vice-President of the Institute. The Committee shall be consist of two PIDS Research Fellows, one permanent and the other non-permanent depending on the research area; the Project Director for the PMO (for PMO managed projects), and the Director for the RSD.

The Committee shall determine acceptance or non-acceptance by the Institute of the submitted report. The Committee shall consider all the reviews prepared on the final report and determine the appropriate action to be carried out. It may recommend revision of the final report or outright acceptance of the report.

In case the Committee decides revisions and improvements to the report, the RSD shall formally notify the proponent. He shall be provided with a copy (or summary) of the reviews containing the comments/suggestions on his report. A meeting among the researcher, the Director for RSD, and the research fellow/program coordinator may be arranged to clarify the necessary revisions. The revised final report shall be re-submitted to the Committee for reconsideration.

If the final report is found acceptable by the Committee, the RSD shall prepare a memo to the President recommending PIDS acceptance of these reports. The memo shall also transmit a letter for the President's signature to the contractor informing him of the Institute's acceptance of the report.

Upon approval by the President, the RSD shall prepare project completion and contractor's performance reports and update its data bases on projects and contractors. At the same time, the RSD shall notify AFD of the approval of the report so that they can undertake the final payment to the proponent. Finally, the RSD shall forward copies of the final report to RID for possible research dissemination and utilization activities.

The above procedures and processes are shown in Chart 4.