CPD Outcome Evaluation - Poverty reduction and economic growth

Report Cover Image
Evaluation Plan:
2007-2009, Romania
Evaluation Type:
Outcome
Planned End Date:
12/2009
Completion Date:
11/2010
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document ToR_for_UNDP_ROMANIA_Country_Programme_Outcome_Evaluation.pdf tor English 186.58 KB Posted 1008
Download document UNDP RO - Outcome Evaluation Report 2005 - 2009 FINAL.pdf report English 575.20 KB Posted 1648
Download document Management Response Country Program 2005-2009 Outcome Evaluation.doc report English 73.50 KB Posted 638
Title CPD Outcome Evaluation - Poverty reduction and economic growth
Atlas Project Number:
Evaluation Plan: 2007-2009, Romania
Evaluation Type: Outcome
Status: Completed
Completion Date: 11/2010
Planned End Date: 12/2009
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
Evaluation Budget(US $): 20,000
Source of Funding:
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Barry Kolodkin, Marius Birsan Team Leader
UNDP Team Leader
GEF Evaluation: No
Key Stakeholders:
Countries: ROMANIA
Lessons
Findings
Recommendations
1 Strategic Recommendation 1. Evolve UNDP Romania to Act as a Project Management Unit. Most of the Government partners expressed a need for support to plan, develop and manage projects. UNDP was viewed as having a strong capability in this area, inexpensive, neutral and an ?international brand.? Access to funds was secondary to being able to implement the projects. Rather than become a capturer and implementer of strategic funds, UNDP would serve to assist ministries and agencies needing assistance in developing projects. The objective would be to help the entities at the national level learn how to transfer money to appropriate projects at the local level. Succinctly, UNDP adds value. This would be accomplished by formulating a strategy for UNDP to act as a project management unit. Elements of this strategy would be: a. Capture all the phases of the project development process accounting for differences in funding ? Romanian Government, EU, other donor, etc. Outcome Statement, Socioeconomic Portfolio, UNDP Romania, Country Program 2005-9, b. Decide where UNDP fits into the process, adds value and document those opportunities. c. Identify the human resources available within UNDP that can support the project efforts and their expertise. d. Create template procedures based on best practices to create successful Memoranda of Understanding, bring central and local authorities together, implement projects across different ministries and agencies, establish project steering committees and prevent funding gaps to the fullest extent possible. e. Based on the 2010-2012 Country Programme Document, Government of Romania strategy, knowledge of funds available and the requests of partners, outline potential projects that could be implemented. f. Training partners to make certain that they understand how to access funds, develop projects, and manage projects. g. Assisting with the selection of local partners and making sure the link between the central and local authorities is strong. The idea is not to hire people and build a new team rather it is to create a framework and infrastructure for the existing team to implement projects. This recommendation does not mean that UNDP should reject acting as an implementer of Structural Funds or creating new projects with other donors. Rather, the recommendation is oriented toward evolving UNDP?s identity in Romania and exploiting its core competencies.
2 Strategic Recommentation 2. Increase the Public Awareness of UNDP?s Activities. UNDP needs to make beneficiaries aware of UNDP?s contributions and social partners of UNDP?s efforts. UNDP cannot expect support from public institutions if they are unaware of UNDP?s effort expended, money spent and results achieved. In order to increase public awareness, the following steps should be taken. a. Each Project Document or Work plan should have a Public Awareness activity component that will explain how UNDP will raise awareness of the project b. Any project site where UNDP is active with a program or has contributed to the development of the site location should have a prominent sign with the UNDP logo. c. Local press releases should be issued for major project milestones. d. Non-confidential executive summaries of annual reports could be sent to stakeholders not formally involved in the project ? local authorities, related ministries, NGOs, etc. e. Face-to-face meetings with project partners to update them on progress. f. Events in the local community related to project launches or the launch of a new service. g. Limited local advertising explaining what project activities are taking place. h. Use of social media to gain interest in the project. (This was cited as effective for Beautiful Romania.) i. Build links from the project website to the websites of stakeholders. During the course of the interviews, it was apparent that the UNDP brand was still valued. Thus, it is important to sustain the visibility of that brand.
3 Strategic Recommendation 3. Be More Customer Friendly. No longer a donor itself, UNDP needs to rely on its partner and customer for financial support. During the course of the evaluations, we learned in one case that UNDP was not flexible in supplying financial reports other than the standard CDR report. On a Democratic Governance project, it was mentioned that a Memorandum of Understanding was reviewed by lawyers in New York to see if it met with UN standards. However, there is no Romanian attorney on staff to focus on the customer?s legal and regulatory needs. In the future, UNDP?s value is provided by the service it deliveres not by the funds it contributes. Thus it is incumbent upon UNDP to provide reporting in a manner that the partner requires, to understand their organizational needs and to provide strong customer support. UNDP Romania, as a part of a global organization, must continue to provide report and act in accordance with organizational standards. However, if UNDP wishes its benefactors to fund projects and act efficiently, it must be prepared to provide them information they need and understand.
4 Strategic Recommendation 4. Greater Cooperation with the Private Sector. During the evaluation discussions, the evaluators spoke with Monica Moldovan, who mentioned that the Energy and Environment portfolio was working with Coca-Cola and Cosmote6 Tactical Recommendations on projects. UNDP works extensively with private sector on projects such as the Global Compact. As UNDP?s role changes from a donor to an implementer and manager in Romania, seeking out private companies to serve as donors or co-donors in socioeconomic projects is logical. In particular, when the economy improves, most of the companies will be more focused on corporate social responsibility again. Private sector participants could be integrated into the Business Incubators program to provide expertise or donations in-kind. Perhaps tourism companies can be involved in Beautiful Romania type activities. Romanian companies operating in Moldova can assist with overseas development assistance in economic development projects.
5 Tactical Recommendation 1. Greater and More Uniform Use of the Project Document. In the documents received by the evaluators, there were Project Documents for the Beautiful Romania, Business Incubators, and Rural Women projects but not for the Hadareni, Roma Strategies and Roma Communities projects. The documents describing these projects were work plans and contracts. A UNDP-formatted project document is not necessarily the key to success. Many other donors have success with different types of formats. The emphasis on creating a project document is to ensure that the outcomes, outputs, objectives, activities and true measures of success are understood; in other words, a project business plan. Submission of a work plan or a contract in lieu of a project document is an emphasis on operations -- outputs, activities and indicators -- but fails to capture the importance, objectives and strategy. The project document formats should be relatively uniform across projects. The three project documents were similar but not in the same format. While allowing sufficient flexibility for project variances, a similar format should be used in order to establish common practices, measurements and comparability across projects. The project document should also have clear definitions of outcomes, objectives, outputs and activities. In several documents, the outcome was at a very high level such as poverty reduction or social inclusion and the outputs were build three houses or create 15 jobs. The outcomes and the outputs should be linked. Of critical importance is to be able to evaluate whether the outcomes are achieved and if the outputs are relevant.
6 Tactical Recommendation 2. Greater Use of Steering Committees. The Ministry of Development, Public Works and Housing mentioned that there were no project steering committee meetings and there were too many written reports. In order for the Ministry to participate and monitor results, they felt it was important to have steering committee meetings. Steering committee meetings also enhance coordination among the partners, contribute to evaluation and increase awareness of UNDP?s results. The idea is not to create work for project managers rather it is to ensure that objectives and outputs are in line and the partnership strategy functions smoothly. Thus, it was advisable for larger projects to form steering committees and hold 1-2 meetings per year.
7 Tactical Recommendation 3. More Consistent Monitoring and Evaluation. The process of monitoring and evaluation was uneven across projects. Consistent monitoring and evaluation is critical when external factors necessitate project changes to achieve expected outcomes, and to improve performance if beneficiaries and partners are not pleased with the project. Recommendation 2, the establishment of project steering committees is one method of improving monitoring. Monitoring would also be enhanced by more frequent visits to the project sites and stakeholders. Several times, stakeholders said that they received reports from UNDP but had not spoken with UNDP recently. In addition to complying with formal report requirements, managers should strive to speak with project participants on a quarterly basis at least. Both the National Agency for Roma and the Authority for Combating Discrimination specifically mentioned that they need UNDP?s help to monitor the government. In the same vein as the Government has limited capacity to manage projects, the capacity to monitor projects is also limited. Therefore, UNDP should be proactive in monitoring the Government?s activity. Finally, a brief project evaluation should be completed at the end of each year, and an outcome evaluation could be done at the end of the project.
8 Tactical Recommendation 4. Improved Financial and Strategic Planning. In several project instances, there were gaps in funding and project activities were delayed. Or, projects failed to continue because authorities could not furnish facilities or hire the staff necessary to operate the facilities. Clearly, UNDP cannot be expected to predict situations such as government instability or economic crises. However, UNDP has sufficient operational experience in Romania to be aware of potential problems with Memoranda of Understanding or agency budget changes. This philosophy is supported by the request by Government partners for UNDP to monitor Government project activity. While not all contingencies can be accounted, UNDP may be able to alter the project design in some cases, so the project does not stop if an agency fails to make a payment the next budget year. For example, with a cost-sharing program where UNDP refurbishes a building and a partner supplies furniture, computers, etc, these activities should be budgeted in the same fiscal year so the partner cannot excise the furnishings from the next year?s budget. The idea of a backup or contingency plan is important. When planning projects, UNDP should have a clear strategy for handing off responsibility for the project activity to a local partner, and a contingency plan should the local partner fail in its obligations. The point where UNDP exits the process should not be at a critical juncture where services or operations cease if an unavoidable failure occurs. Incorporated into the output/activity planning should be a monitoring of the Government or other partners? funding and capacity to assume any follow-on duties.
1. Recommendation: Strategic Recommendation 1. Evolve UNDP Romania to Act as a Project Management Unit. Most of the Government partners expressed a need for support to plan, develop and manage projects. UNDP was viewed as having a strong capability in this area, inexpensive, neutral and an ?international brand.? Access to funds was secondary to being able to implement the projects. Rather than become a capturer and implementer of strategic funds, UNDP would serve to assist ministries and agencies needing assistance in developing projects. The objective would be to help the entities at the national level learn how to transfer money to appropriate projects at the local level. Succinctly, UNDP adds value. This would be accomplished by formulating a strategy for UNDP to act as a project management unit. Elements of this strategy would be: a. Capture all the phases of the project development process accounting for differences in funding ? Romanian Government, EU, other donor, etc. Outcome Statement, Socioeconomic Portfolio, UNDP Romania, Country Program 2005-9, b. Decide where UNDP fits into the process, adds value and document those opportunities. c. Identify the human resources available within UNDP that can support the project efforts and their expertise. d. Create template procedures based on best practices to create successful Memoranda of Understanding, bring central and local authorities together, implement projects across different ministries and agencies, establish project steering committees and prevent funding gaps to the fullest extent possible. e. Based on the 2010-2012 Country Programme Document, Government of Romania strategy, knowledge of funds available and the requests of partners, outline potential projects that could be implemented. f. Training partners to make certain that they understand how to access funds, develop projects, and manage projects. g. Assisting with the selection of local partners and making sure the link between the central and local authorities is strong. The idea is not to hire people and build a new team rather it is to create a framework and infrastructure for the existing team to implement projects. This recommendation does not mean that UNDP should reject acting as an implementer of Structural Funds or creating new projects with other donors. Rather, the recommendation is oriented toward evolving UNDP?s identity in Romania and exploiting its core competencies.
Management Response: [Added: 2010/02/10] [Last Updated: 2010/06/16]

This has been the rationale of our current pipeline development based on EU funds. It is important to underline, in this context, that the UNDP role is that of facilitator of access to European funding. Even when UNDP leads grant application, this role is intended to support government and non-governmental institutions? access to structural and other relevant development funds. UNDP will work directly with eligible beneficiary institutions and support their project preparation and management capacities by joint project formulation for consideration under applicable operational programmes.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Based on the 2010-2012 Country Programme Document, Government of Romania strategy, knowledge of funds available and the requests of partners, outline potential projects that could be implemented.
[Added: 2011/04/08]
RR, Program Unit 2010/12 Completed
2. Recommendation: Strategic Recommentation 2. Increase the Public Awareness of UNDP?s Activities. UNDP needs to make beneficiaries aware of UNDP?s contributions and social partners of UNDP?s efforts. UNDP cannot expect support from public institutions if they are unaware of UNDP?s effort expended, money spent and results achieved. In order to increase public awareness, the following steps should be taken. a. Each Project Document or Work plan should have a Public Awareness activity component that will explain how UNDP will raise awareness of the project b. Any project site where UNDP is active with a program or has contributed to the development of the site location should have a prominent sign with the UNDP logo. c. Local press releases should be issued for major project milestones. d. Non-confidential executive summaries of annual reports could be sent to stakeholders not formally involved in the project ? local authorities, related ministries, NGOs, etc. e. Face-to-face meetings with project partners to update them on progress. f. Events in the local community related to project launches or the launch of a new service. g. Limited local advertising explaining what project activities are taking place. h. Use of social media to gain interest in the project. (This was cited as effective for Beautiful Romania.) i. Build links from the project website to the websites of stakeholders. During the course of the interviews, it was apparent that the UNDP brand was still valued. Thus, it is important to sustain the visibility of that brand.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

The Management agrees that the Country Office has not been able to leverage well the good reputation and brand recognition in Romania of UNDP. The consultative processes for the formulation and consequent approval of the 2010-2012 Country Program have created structured mechanisms for UNDP to communicate its mandate and capacities to stakeholders and partners. Upon CPAP approval, there will be renewed efforts at dissemination and knowledge management under new projects.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Updating of the UNDP CO Website
[Added: 2011/04/08]
Program Unit 2011/12 Completed
2.2. Developing a CO and Project communication strategy
[Added: 2011/04/08] [Last Updated: 2012/05/02]
Program Unit 2011/12 Completed The Program Unit has focused on updating UNDP CO website, making sure that it has all pertinent information so that the CO can capitalize on the brand name. Furthermore the Unit has also incorporated PR/visibility into projects (notably European Social Funded projects where it was actually required).
3. Recommendation: Strategic Recommendation 3. Be More Customer Friendly. No longer a donor itself, UNDP needs to rely on its partner and customer for financial support. During the course of the evaluations, we learned in one case that UNDP was not flexible in supplying financial reports other than the standard CDR report. On a Democratic Governance project, it was mentioned that a Memorandum of Understanding was reviewed by lawyers in New York to see if it met with UN standards. However, there is no Romanian attorney on staff to focus on the customer?s legal and regulatory needs. In the future, UNDP?s value is provided by the service it deliveres not by the funds it contributes. Thus it is incumbent upon UNDP to provide reporting in a manner that the partner requires, to understand their organizational needs and to provide strong customer support. UNDP Romania, as a part of a global organization, must continue to provide report and act in accordance with organizational standards. However, if UNDP wishes its benefactors to fund projects and act efficiently, it must be prepared to provide them information they need and understand.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

While the Management is understanding of this Evaluation Recommendation, the CO experience demonstrates that non-standard financial reporting to partners is extremely risky and exposes the organization to questions that we cannot answer. Therefore to the best extent possible financial reporting is limited to the CDR and detailed expenditure reports are used for explanatory purposes. Management has sought instead for partners to better understand UNDP?s own financial management and reporting tools and procedures instead of focusing on the often changing national rules and regulations of the clients. This recommendation is being interpreted by the Management as an area of improvement whereby the CO has more structured sessions with its ?clients? to explain the UNDP?s rules and regulations and to ensure corporate compliance of our reporting.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Structured presentations to Ministry of Finance and other key partners on UNDP?s project, financial and procurement management; and continued efforts to eliminate non-standard agreements
[Added: 2011/04/08]
RR, OM, Program Unit 2010/12 Completed
4. Recommendation: Strategic Recommendation 4. Greater Cooperation with the Private Sector. During the evaluation discussions, the evaluators spoke with Monica Moldovan, who mentioned that the Energy and Environment portfolio was working with Coca-Cola and Cosmote6 Tactical Recommendations on projects. UNDP works extensively with private sector on projects such as the Global Compact. As UNDP?s role changes from a donor to an implementer and manager in Romania, seeking out private companies to serve as donors or co-donors in socioeconomic projects is logical. In particular, when the economy improves, most of the companies will be more focused on corporate social responsibility again. Private sector participants could be integrated into the Business Incubators program to provide expertise or donations in-kind. Perhaps tourism companies can be involved in Beautiful Romania type activities. Romanian companies operating in Moldova can assist with overseas development assistance in economic development projects.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

The CO will strengthen private sector partnership both in terms of benefiting from private sector perspective in understanding the overall development context in the country and in terms of promoting private sector participation in project activities, notably through CSR capacity development

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Introduce CSR into projects pipeline
[Added: 2011/04/08]
Socio-Economic/Poverty Section 2010/12 Completed
5. Recommendation: Tactical Recommendation 1. Greater and More Uniform Use of the Project Document. In the documents received by the evaluators, there were Project Documents for the Beautiful Romania, Business Incubators, and Rural Women projects but not for the Hadareni, Roma Strategies and Roma Communities projects. The documents describing these projects were work plans and contracts. A UNDP-formatted project document is not necessarily the key to success. Many other donors have success with different types of formats. The emphasis on creating a project document is to ensure that the outcomes, outputs, objectives, activities and true measures of success are understood; in other words, a project business plan. Submission of a work plan or a contract in lieu of a project document is an emphasis on operations -- outputs, activities and indicators -- but fails to capture the importance, objectives and strategy. The project document formats should be relatively uniform across projects. The three project documents were similar but not in the same format. While allowing sufficient flexibility for project variances, a similar format should be used in order to establish common practices, measurements and comparability across projects. The project document should also have clear definitions of outcomes, objectives, outputs and activities. In several documents, the outcome was at a very high level such as poverty reduction or social inclusion and the outputs were build three houses or create 15 jobs. The outcomes and the outputs should be linked. Of critical importance is to be able to evaluate whether the outcomes are achieved and if the outputs are relevant.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

This evaluation finding had been noted by the CO management at end 2008 and action has been rolled out in the form of a Program Management Action Plan to ensure elimination of non-standard and non-uniform project documents, project cost-sharing agreements and other programming tools. The CO entered into a new programming cycle with the CPD 2010-2012 and the CPAP which is to be approved in 2010 provides for a strategic planning and monitoring document with relevant baselines, indicators and targets that then get incorporated into individual project documents

Key Actions:

6. Recommendation: Tactical Recommendation 2. Greater Use of Steering Committees. The Ministry of Development, Public Works and Housing mentioned that there were no project steering committee meetings and there were too many written reports. In order for the Ministry to participate and monitor results, they felt it was important to have steering committee meetings. Steering committee meetings also enhance coordination among the partners, contribute to evaluation and increase awareness of UNDP?s results. The idea is not to create work for project managers rather it is to ensure that objectives and outputs are in line and the partnership strategy functions smoothly. Thus, it was advisable for larger projects to form steering committees and hold 1-2 meetings per year.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

For newly approved projects under the CPAP 2010-2012, Management has designated Outcome and Output level review committees as well as project based steering committees have been established building on the LPAC process.

Key Actions:

7. Recommendation: Tactical Recommendation 3. More Consistent Monitoring and Evaluation. The process of monitoring and evaluation was uneven across projects. Consistent monitoring and evaluation is critical when external factors necessitate project changes to achieve expected outcomes, and to improve performance if beneficiaries and partners are not pleased with the project. Recommendation 2, the establishment of project steering committees is one method of improving monitoring. Monitoring would also be enhanced by more frequent visits to the project sites and stakeholders. Several times, stakeholders said that they received reports from UNDP but had not spoken with UNDP recently. In addition to complying with formal report requirements, managers should strive to speak with project participants on a quarterly basis at least. Both the National Agency for Roma and the Authority for Combating Discrimination specifically mentioned that they need UNDP?s help to monitor the government. In the same vein as the Government has limited capacity to manage projects, the capacity to monitor projects is also limited. Therefore, UNDP should be proactive in monitoring the Government?s activity. Finally, a brief project evaluation should be completed at the end of each year, and an outcome evaluation could be done at the end of the project.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

The CO Management has specified CP level and project level monitoring and evaluation needs as per its Monitoring and Evaluation Plan for 2010-2012. Outcome boards have been identified and established to support overall CP monitoring.

Key Actions:

8. Recommendation: Tactical Recommendation 4. Improved Financial and Strategic Planning. In several project instances, there were gaps in funding and project activities were delayed. Or, projects failed to continue because authorities could not furnish facilities or hire the staff necessary to operate the facilities. Clearly, UNDP cannot be expected to predict situations such as government instability or economic crises. However, UNDP has sufficient operational experience in Romania to be aware of potential problems with Memoranda of Understanding or agency budget changes. This philosophy is supported by the request by Government partners for UNDP to monitor Government project activity. While not all contingencies can be accounted, UNDP may be able to alter the project design in some cases, so the project does not stop if an agency fails to make a payment the next budget year. For example, with a cost-sharing program where UNDP refurbishes a building and a partner supplies furniture, computers, etc, these activities should be budgeted in the same fiscal year so the partner cannot excise the furnishings from the next year?s budget. The idea of a backup or contingency plan is important. When planning projects, UNDP should have a clear strategy for handing off responsibility for the project activity to a local partner, and a contingency plan should the local partner fail in its obligations. The point where UNDP exits the process should not be at a critical juncture where services or operations cease if an unavoidable failure occurs. Incorporated into the output/activity planning should be a monitoring of the Government or other partners? funding and capacity to assume any follow-on duties.
Management Response: [Added: 2010/02/10] [Last Updated: 2011/04/08]

In standard government cost-shared projects, contribution schedules have been aligned to AWP requirements. Management agrees with the Evaluation finding that better exit strategies will have to be put in place per project. The approval of new projects under the new CPD will allow for this.

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org