Integrating Rio Conventions Provisions into Ukraine’s National Policy Framework

Report Cover Image
Evaluation Plan:
2018-2022, Ukraine
Evaluation Type:
Final Project
Planned End Date:
03/2018
Completion Date:
03/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
12,000

Share

Document Type Language Size Status Downloads
Download document ToR_ TE Rio project Ukraine.docx tor English 69.04 KB Posted 304
Download document PIMS 4478 Evaluation of RIO Conventions Project_FINAL1.docx report English 2150.25 KB Posted 579
Title Integrating Rio Conventions Provisions into Ukraine’s National Policy Framework
Atlas Project Number: 00074532
Evaluation Plan: 2018-2022, Ukraine
Evaluation Type: Final Project
Status: Completed
Completion Date: 03/2018
Planned End Date: 03/2018
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.4. Scaled up action on climate change adaptation and mitigation across sectors which is funded and implemented
SDG Goal
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 13.2 Integrate climate change measures into national policies, strategies and planning
Evaluation Budget(US $): 12,000
Source of Funding: Project budget
Evaluation Expenditure(US $): 9,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Elinor Bajraktari
Viktor Karamushka
GEF Evaluation: Yes
GEF Project Title: Capacity Development: Integrating Rio Convention Provisions into Ukraine’s National Environmental policy Framework
Evaluation Type: Terminal Evaluation
Focal Area: Multifocal Areas
Project Type: MSP
GEF Phase: GEF-5
GEF Project ID: 4913
PIMS Number: 4478
Key Stakeholders:
Countries: UKRAINE
Lessons
Findings
1.

3. FINDINGS

While the amount of information generated by this evaluation was enormous, the findings presented in this chapter cover only the most essential aspects of the project and are to some extent focused on those issues that require improvement and the attention of the project team and country office management. The findings of this evaluation are organized in the following sections: i) Project Design; ii) Project Implementation; and, iii) Project Results.

3.1. Project Design

This section examines the project's logic and design features by focusing on the adequacy of elements like the results framework, management arrangements, identification of risks and assumptions, use of lessons derived from other projects, linkages with relevant UNDP or donor projects, UNDP's comparative advantage in the area, planned stakeholder engagement, replication approach and exit strategies, etc. The main questions that have driven the analysis presented in this section are shown in Box 2 below.


Tag: Communication Human and Financial resources Knowledge management Operational Efficiency Programme/Project Design Theory of Change Institutional Strengthening Data and Statistics

2.

3.1. Project Design - 3.1.1. Analysis of the Project Document and Planning Matrix(continuation)

While the shortcomings in the area of environmental protection are clearly identified, designing a feasible course of actions that address them in an effective manner is difficult. The Rio Project was conceived to contribute precisely to this. The project intervention was organized in three components, summarized as follows:

- The first component focused on strengthening the country's policy and institutional framework by integrating Rio Convention provisions into sectoral policies that serve to meet national socio-economic development priorities. This mainstreaming exercise was to inform the development of a National Sustainable Development Strategy (NSDS) that would reinforce the legitimacy of sectoral policies, programmes, plans and legislations. -The second component focused on strengthening individual and organizational capacities by targeted implementation of policy and programme recommendations made under the first component. Guidelines and training manuals would be prepared to help institutionalize the learning-by-doing to implement integrated global environmental and sustainable development strategies through the SDS. Targeted training would be provided on focal area and sectoral aspects of sustainable development in order to reinforce the validity and legitimacy of each ministry and their respective agencies and departments' actions towards congruent implementation of the policy and programme recommendations. -The third component focused on developing and implementing a broad-based public awareness campaign to raise the level of public support to the goal and objectives of the project, focusing on promoting an understanding of Rio Convention issues. 


Tag: Policies & Procedures Programme/Project Design Advocacy Awareness raising Institutional Strengthening Policy Advisory Data and Statistics

3.

3.1. Project Design - 3.1.1. Analysis of the Project Document and Planning Matrix(continuation)

In addition to the logical framework, there are a number of other design aspects which could have been framed more adequately. The following is a short summary of the most important ones. It is important to emphasize here the following discussion does not pertain to how the project was implemented, but only to how it was designed. 

Insufficient Focus on the Process Not only project outcomes, but the Project Document in general is largely focused on outputs and specific products such as the NSDS, the Roadmap, sectoral analyses, surveys, trainings, etc. Overall, it provides limited guidance on the process through which these outputs will be realized. Let's take, for example, the development of the NSDS, which is one of the most important aspects of the project. The Project Document does not outline in any detail any processes or governance structures that would guide the development of the NSDS. The focus of the document is not on the “how” of the formulation, but on the “what” of the NSDS. The Project Document understates the complexity inherent in the process of formulating a national development strategy. There is ample evidence out there, shown by other countries' experiences, that the benefits of developing a national strategy are just as much, if not more, related to the process through which it is developed than the document (end product) itself. The formulation of a national development strategy is a massive undertaking that involves a large number of actors and activities and requires a significant amount of resources. To be sustainable and fully institutionalized, it also requires the full support of the countrys leadership and a wide political coalition (including political parties and civil society). The way the NSDS process unfolded supports this point – with hindsight, we know that the project had a hard time obtaining the right level of commitment from the government's leadership for the adoption of the strategy.


Tag: Partnership Programme/Project Design Project and Programme management Results-Based Management Risk Management Theory of Change Coordination Institutional Strengthening

4.

3.1. Project Design - 3.1.1. Analysis of the Project Document and Planning Matrix(continuation)

Intervention Centered on Environmental Institutions Although the project?s stated objective was to promote cross-sectoral integration of environmental concerns, the real focus of activities as presented in the Project Document is the environmental sector and MENR. For all the linkages between environmental concerns and development policy highlighted at the conceptual level, practical-level activities involve primarily environment sector specialists and institutions. For example, we know that work on the NSDS comprises crucial social and economic issues related to all sectors. Yet, the Project Document envisages the process largely within the purview of MENR, which does not allow for sufficient buy-in from central ministries and sectors (especially the key Ministries of Finance, Transport, Energy, Agriculture, etc.). Such heavy focus on MENR is misplaced, given the inability of MENR's policy units to mobilize action of the magnitude required for the NSDS across the government and beyond. With hindsight, we know that the consequence of this set up was insufficient interest and uptake from key institutions – especially, in the case of NSDS which remains unapproved in the Presidential Administration. 

The following issues could have received more attention in the Project Document: o Analysis of the political economy of the mainstreaming of environmental concerns and sustainable development, with a view to understanding more carefully the interests and positions of the key players in the government. o Devising strategies and tactics for building stronger political will and support for the project at the highest levels of government. o Establishing stronger coordination mechanisms within the government and between the government and civil society for the discussion and agreement of policy and strategy options (i.e. inter-departmental committees to improve environmental integration). Key here is the institutionalization of these structures which in the course of the project turned out to be a real challenge to accomplish. o Addressing more effectively the lack of knowledge and understanding of environmental matters in key ministries and regional governments.


Tag: Efficiency Resource mobilization Operational Efficiency Policies & Procedures Results-Based Management Country Government Civil Societies and NGOs National Institutions

5.

3.1. Project Design - 3.1.1. Analysis of the Project Document and Planning Matrix(continuation)

Insufficient of Focus on the Sub-national Level and Decentralization The Project Document has a limited focus on the sub-national level (this received more attention later thanks to the project's adaptive management). In the project document, activities are not conceptually integrated with community development interests and the decentralization process. When the project was designed it was inconceivable that Ukraine would experience a revolution, armed conflict in Donbass, major reform initiatives and, more importantly, increased impetus for decentralization. Yet, the sub-national level was always supposed to be crucial for the promotion of sustainable development. Improving the integration of environmental and development objectives requires greater decentralization of power and resources and the participation of people at grass-roots level.

A careful reading of the Project Document does not reveal why the Project Document did not establish a more solid line of work at the sub-national level – for example, in the context of NSDS, providing support to sub-national governments to develop their own development strategies linked to national strategies. No discussion of this type of work is provided in the Project Document. Furthermore, the Project Document could have focused more on the implementation of the Conventions on the ground, by supporting activities of local authorities, organizations and communities to implement the conventions in their local context, and improving feedback mechanisms from local to national and international levels. Also, the Project Document could have designed more effective instruments to building the capacities of local governments, organizations and communities to develop their own initiatives and translate the NSDS into community-level resource management strategies. It could also have envisaged the establishment and strengthening of governance mechanisms that allow local communities to feed back into the national agenda (i.e. committees with community and civil society representatives).


Tag: Effectiveness Efficiency Local Governance Policies & Procedures Project and Programme management Results-Based Management Coordination Private Sector

6.

3.1. Project Design (continuation)

3.1.1. Analysis of the Project Document and Planning Matrix(continuation)

Weak Linkages with the MDGs (and subsequently SDGs)The design of the project was an opportunity to integrate environmental concerns into policy frameworks through the MDG12 process, and subsequently the SDGs (which at the time of the development of the project document were still under discussion). However, the Project Document leaves the MDGs mostly out of the picture. As the SDG process got started when the project was half-way into implementation, the importance of sustainability and the need for strong government commitment to meet agreed global targets by 2030 came the fore of public debate internationally and in the country. As will be discussed further in this report, the project team reacted to this opportunity by crafting a role for the project in supporting GoU adapt SDG targets and indicators to national conditions and report results nationally and internationally. Furthermore, the NSDS process was closely linked to the SDGs activities. This turned out to be a positive modification which added to the relevance of the project and the quality of its results. Overall, based on the examples provided here, it can be argued that design shortcomings in the Project Document represent had a constraining effect on project activities and results (as will be seen further in this report). As discussed in more detail in the section on adaptive management, the team was able to mitigate the consequences of some of these challenges, but nevertheless the roots of the problem were such that they could be neutralized entirely. 


Tag: Effectiveness Efficiency Policies & Procedures Results-Based Management Risk Management Theory of Change Data and Statistics SDG Integration SDG monitoring and reporting

7.

3.1.2. Assumptions and Risks (continuation)

A major risk that occurred during the lifetime of the project and which the Project Document could not have foreseen was the 2014 Maydan revolution, the ousting of the previous administration, Russia's takeover of Crimea, the conflict in Donbass and the ensuing political turbulence and major reform agenda. Beyond this, the four risks described above have been and remain serious limitations of Ukraine's context and have therefore been correctly identified in the Project Document. However, the assumptions underlying the factors that would mitigate these risks could have been analyzed more adequately

1. The lack of government commitment to prioritize environmental concerns due to other governance priorities turned out to be a major factor that had a considerable effect on the project. The reform agenda that emerged after the 2014 uprising was something that could not have been predicted when the project was developed. While this created opportunities for new avenues of cooperation, it also further relegated the environment to a less prominent position in the list of priorities, given GoU' heavy focus on anti-corruption, justice, restructuring of State-Owned Enterprises, health, energy security, etc. As will be discussed in the “adaptive management” section, the project team tried to address this risk by linking NSDS to the SDGs and the government's commitment to implement them.

2. Also, resistance from politically and institutionally entrenched sectors that have traditionally governed Ukraine has been and remains a powerful force to reckon with in Ukraine's current political environment. The focus of the design on environmental institutions and MENR was not helpful in mobilizing the support of other sectors and ministries.


Tag: Efficiency Parliament Integration Operational Efficiency Policies & Procedures Results-Based Management Risk Management Country Government

8.

3.1. Project Design(continuation)

3.1.3. Lessons from Other Relevant Projects Incorporated into the Project Design

As described earlier in this report, between 2004 and 2007 UNDP supported the implementation of Ukraine?s GEF-funded National Capacity Self-Assessment project. The NCSA was implemented in three phases. The first consisted of taking stock of the country's national implementation of the three conventions, with particular attention paid to analyzing the framework underlying convention implementation. The second phase focused on an assessment of the cross-cutting interactions of convention implementation, including an assessment of stakeholder capacities. The final phase identified priority actions for capacity development. The NCSA was prepared through a collaborative process, involving multiple stakeholders which strengthened the legitimacy of its outcomes. One key result was the National Environmental Policy (NEP) of 2007 which incorporated the principles and provisions of the Rio Conventions. Except for NCSA, the Rio Project Document does not explore lessons from other relevant projects by UNDP other donors. For example, Ukraine does have a National Sustainable Development Strategy (till 2020) which was adopted by the authorities in 2015, but had been in the making since 2012. 13 It would have been useful if the Project Document had examined the experience of that earlier initiative and identified from it relevant lessons. While the project document links the project to ongoing UNDP initiatives on the mainstreaming of the Rio Conventions, it does not refer to the experience and lessons learned from many other UNDP/GEF projects on the mainstreaming of the Rio Conventions implemented throughout the world


Tag: Climate Change Adaptation Biodiversity Communication Knowledge management Programme Synergy Results-Based Management

9.

1.3 Project Desgn (continuation) 

3.1.4. Linkages to other Relevant Projects in the UNDP Portfolio (continuation) As has been discussed, although the potential for interlinkages within the E&E cluster and the broader UNDP portfolio is significant, the design of the Rio Project did not consider them. The Project Document did not foresee any formal mechanisms for close coordination, active sharing of experiences or joint activities. The analysis of the information collected in the course of this evaluation indicates that the degree of cooperation between the Rio Project and the other projects within the cluster and in the broader CO programme was limited and this may be attributed to some extent to the shortcomings of the project design. One positive example is the cooperation that took place with the Small Grant Programme (SGP) which is a GEF-funded project. The regional pilots were administered through the SGP which ensured compliance with GEF regulations and procedures. Importantly, all three projects contribute to implementation of the globally adopted Sustainable Development Goals (SDGs) in Ukraine.


Tag: Partnership Programme Synergy Strategic Positioning Country Government

10.

3.1.6. Planned Stakeholder Participation

A broad-based stakeholder participation process was inherent in the design of the project, given that it was intended to facilitate partnerships with a broad-spectrum of stakeholders in different areas related to sustainable development. Public sector organizations were identified as key partners of the project – in particular, central level ministries and local governments. At the central level, the main project partner was designated to be MENR, given its role as the focal point for the Rio Conventions and its responsibility for reporting on their national implementation. For all the positive aspects related to MENR's central role in the implementation of the Rio Conventions, this approach also had its downsides discussed in the previous section – most importantly, MENR's lack of authority to mobilize other line ministries around the NSDS and the concept of sustainable development. The Project Document also highlighted the role of the Ministry of Agrarian Policy and Food (MAPF), State Environmental Investment Agency (SEIA) and a number of other key state agencies directed and coordinated by the Cabinet of Ministers through MENR. At the regional level, the Project Document identified local governments (at the oblast, district and municipal levels) and state departments for environmental protection which implement environmental policy in oblasts and select cities. Although the latter authorities are subordinated to the MENR, their work is coordinated with regional administrations. The Project Document also identified a number of civil society organizations as project partners on a range of activities, especially the development of the NSDS. Academia, in particular Ukraine's National Academy of Sciences, were given a particular role in the conduct of analytical work, drafting of multiple reports and delivery of training. Also, the media was designed to play a special role in the project in support of the awareness raising activities. 


Tag: Effectiveness Efficiency Donor relations Partnership Project and Programme management Results-Based Management Country Government International Financial Institutions

11.

3.1 Project Design (continuation)

3.1.7. Replication Approach

As for the replication approach, the Project Document is not very explicit about what exactly is meant by replication, what aspects of the project are intended to be replicated and how replication is supposed to occur. The replication approach and the broader issue of sustainability would have benefited from better definitions and a more in-depth analysis. The following is a brief overview of three main issues closely related to the project's capacity for replication. The Project Document recognizes that high-level political commitment is critical for the sustainability of outcomes and their further replication, but does not outline any strategy for how this commitment could be obtained. Although it recognizes the need to institutionalize the project's capacities development structures, it does not discuss how institutionalization will be achieved and does not provide any guidance for that process. It leaves this contingency open by stating that “the best set of institutional structures and mechanisms for NSDS implementation will be validated by stakeholders and incorporated into the NSDS Roadmap”. In practice, however, we know that by the time the Roadmap was developed there was little time left for the project to pursue the institutionalization of governance structures. With hindsight, it is clear that the form of institutionalization should have been designed right from the beginning of the project on the basis of a well-outlined plan. Similarly, the Project Document considers the establishment of an inter-agency working group on sustainable development under the Cabinet of Ministries or the Parliament, but does not provide any guidance and does not foresee any activities for doing this under the “Activities and Results Framework”. Overall, the lack of detail on the institutionalization of these structures represents another limitation of the project design.


Tag: Sustainability Parliament Results-Based Management Country Government Data and Statistics SDG Integration

12.

3.1 Project Design (continuation)

3.1.8. Management arrangements The project was designed to be implemented under the national implementation (NIM) modality, but soon after its launch implementation was switched to direct implementation (DIM). The organogram provided in the Project Document for the project implementation arrangements is shown in Figure 7 below.

The Project Document foresaw the following organizational arrangements: - MENR was designated the Executing Agency. It would assign a National Project Director (NPD) and provide its staff and network of experts as support to Project Management Unit (as part of government co-financing). - The Project Board would be established by the project to provide management oversight of project activities and was to be chaired by the MENR (Focal Point for GEF). The Board would review progress and evaluation reports, and approve programmatic modifications to project execution, as appropriate and in accordance to UNDP procedures. In addition to MENR, government membership of the Project Board would include the SEIA (Focal Point for the FCCC), as well as representatives from relevant line ministries and their respective state agencies. Non-state stakeholders would also be represented on the Project Board, namely from the private sector, academic and research institutions and civil society organizations. The Project Board was expected to meet four times per year. - A senior government official was to be designated at the National Project Director, and would be responsible for management oversight of the project. The NPD would devote a significant part of his/her working time on the project. In the fulfillment of his/her responsibilities, the NPD would be supported by the Project Board and a full-time National Project Manager (NPM).


Tag: Efficiency Human and Financial resources Implementation Modality Oversight Project and Programme management Technical Support

13.

3.2. Project Implementation

During the four years of its lifetime, the project went through a number of important stages. The following is the chronology of key events that marked the project's conceptualization and implementation phases that spanned the 2012-2017 period.

· The Project Identification Form was approved by GEF on 21 September 2012. · The project was endorsed by GEF on 22 October 2013. · The Local Project Appraisal Committee meeting took place on 26 November 2013. · The Project Document was signed by GoU on 9 December 2013, thus marking the official start of the project. · The Inception Workshop was organized on 29 April 2014. · The mode of implementation was switched from National Implementation Modality (NIM) to Direct Implementation Modality (DIM) shortly after its launch in April 2014 (the following section on Adaptive Management provides more details on the reasons for the change). · The Project Board was not established formally. Over the four years of its implementation, the Project Board met three times – in April 2014 (constitutive meeting), December 2015 and March 2017. · In 2016 the project's closing date was extended at the request of the project from 9 December 2016 to 31 December 2017. · The project closed on 31 December 2017.


Tag: Effectiveness Efficiency Global Environment Facility fund Human and Financial resources Oversight Project and Programme management

14.

3.2. Project Implementation (continuation)

Furthermore, in 2015 the new global development agenda was launched and the project approach was revised to accommodate GoU's need for support in adapting the SDGs to the local context. In particular, the project team made a crucial decision to pursue a number of SDG-related activities under the scope of this project and link the development of the NSDS to GoU's work on the adaptation and implementation of SDGs (this will be described in more detail in the following section). This meant that the bulk of activities on the NSDS had to wait until the government had completed the SDG adaptation process. The production of other outputs related to the NSDS, such as the Roadmap, had also to be delayed. Overall, this affected a number of project activities directly related to the SDGs, which had to be conducted in a rush closer to the end of the project, after the National Report on SDGs was finally released on 15 September 2017. Only then could the project release products such as the RIA analysis and the report on SDGs statistics, convene regional trainings on the integration of SDGs into local development strategies, etc. 

As a result of this rapidly changing political context and in response to the need to align certain activities to GoU's SDG agenda, the project requested a one-year no-cost extension, which was granted by GEF in 2016. Despite these challenges, the project managed to complete on time manner all tasks it set out to complete.


Tag: Oversight Project and Programme management Quality Assurance Results-Based Management Agenda 2030 Data and Statistics SDG Integration SDG monitoring and reporting

15.

3.2.1. Adaptive Management (continuation)

Another positive adaptive change pursued by the project was the conduct of a number of pilot initiatives at the local level. As has been stated, the Project Document did not provide details about the pilots, but the project team was able to develop a system for supporting community-led initiatives at the sub-national level and identify good opportunities. The cooperation with the Small Grants Programme (SGP), a long-standing GEF-funded project, played an important role as it not only ensured compliance with GEF regulations and procedures, but also allowed the project to take advantage of SGP's tested delivery mechanisms at the grassroots level.


Tag: Global Environment Facility fund Local Governance Parliament Public administration reform Oversight Project and Programme management Country Government

16.

3.2.1. Adaptive Management (continuation)

With support from the UNDP CO, the project team could have tried to link some of its high-level policy work more closely to the post-Maydan reform agenda led by the National Reform Council. This is certainly difficult because the SDGs are voluntary and do not entail budgetary resources, as do the agreements with the IMF and the EU which represent binding commitments for GoU. Nevertheless, more efforts could have been made, especially from the CO management, to align UNDP's SDG activities to the reform efforts spearheaded by the National Reform Council. This would have required that the CO and project staff understand and navigate the reform infrastructure and agenda more effectively in order to be able to position itself more advantageously vis-à-vis the other donors. Doing this would have required a deeper understanding of and engagement with ongoing reform initiatives and a careful identification of opportunities for synergies. By connecting its activities more effectively to the reform agenda, UNDP would have been able to achieve better cooperation and synergies with the other donors who are also connected to and actively supporting the government?s reform agenda. Also, stronger links could have been established with the process of decentralization which came to the fore of the government?s agenda after the start of the conflict in the Donbass region. Certainly, this was the focus of other UNDP projects, especially CBA, but still work around the SDG process at the sub-national level could have been connected more effectively to those activities. 


Tag: Public administration reform Oversight Partnership Programme Synergy Project and Programme management Country Government Coordination SDG Integration Civil Societies and NGOs

17.

3.2.2. Partnership Arrangements (continuation)

Also, members of parliamentary committees were invited to participate in consultations about the NSDS and received drafts of the strategy. The project translated into Ukrainian UNDP?s toolkit “Parliament’s Role in Implementing the Sustainable Development Goals: A Parliamentary Handbook”. Oblast and district assemblies were extensively involved in regional mainstreaming activities (pilots).The project also built strong partnerships with civil society organizations. A large number of NGOs and community groups were involved in the consultations and public awareness activities organized by the project. Furthermore, the project relied on the expertise, capabilities and networks of NGOs for the conduct of a number of studies and pilot initiatives. Table 5 below shows the eleven NGOs that were contracted directly by the project to deliver specific activities. 

The project involved 44 local experts (the complete list of experts engaged by the project can be found in Table 12), especially from the academic institutions like the Natural Sciences Faculty of Kyiv Mohyla Academy, National Academy of Sciences, National Academy of Public Management under the Presidential Administration, etc.). The project created an effective platform for academics, experts, civil society representatives to discuss important issues with government officials (primarily, from MENR and MEDT). For all these achievements in engaging a wide variety of partners in project activities, there was certainly room for better engagement of certain partners. The case for more active engagement of actors such as the Ministry of Finance has already been made in previous sections. Overall, the engagement of government agencies could have been more effective in terms of both breadth and depth. This is obviously challenging and goes beyond the purview of this project. Weak government engagement in these types of initiatives is an endemic phenomenon reported across the board by other UNDP projects, as well as other development agencies. This challenge is related to long-standing structural weaknesses and inadequate incentives internal to the government which cannot be reviewed thoroughly in this report because they fall outside the scope of this evaluation. This issue has been explored to some extent in the outcome evaluation of UNDP?s E&E portfolio and a short summary of the challenges that are most pertinent to the Rio Project is provided in Box 5 below. 


Tag: Local Governance Parliament Oversight Partnership Project and Programme management SDG Integration Civil Societies and NGOs

18.

3.2.3. Feedback from M&E Activities Used for Adaptive Management

As noted in the previous sections, adaptive management was crucial for the project team?s response to Ukraine?s rapidly changing political and institutional environment. This adaptive reaction resulted in a number of new activities such as the scaling up activities on the SDGs, piloting of initiatives at the local level, etc. (these were described in detail in the section on adaptive management). To some extent, the ability of the project team to react was enabled by feedback received through the M&E system which consisted of a number of mechanisms (i.e. planning, monitoring, risk management, etc.). A primary tool of M&E were the Project Implementation Reviews (PIRs) conducted at the end of each year. The evaluation team reviewed all four PIRs produced in the course of project implementation (2014 to 2017) and found them to have been used adequately by the project team to inform the CO management, government partners, GEF and other stakeholders. PIRs provided the project team and board with the opportunity to take stock of the situation in the middle of each year and engage relevant actors into discussion. 


Tag: Monitoring and Evaluation Project and Programme management Quality Assurance Results-Based Management Risk Management

19.

3.2.4. Project Finance

The project had a budget of US$ 900,000, provided by GEF. The Project Document indicates that co-financing amounting to US$ 980,000 was expected from GoU and project partners. Cofinancing was expected both in kind and in financial terms. The overall project budget had the structure shown in Table 6 below, with resources spread out evenly across all three project components and US$ 90,000 for project management.

The total amount of co-financing that materialized in the course of the project was hard to estimate, despite attempts made by the evaluation team to quantify the information provided by the project team. The government and project partners did provide in-kind contributions in the form of man-hours spent by government officials and staf on project activities – such as participating in SDG working groups, discussing the NSDS and Roadmap with civil society, participating in project events, reviewing analytical reports, etc. But converting this contribution into a financial figure was impossible not only because it is hard to price such things appropriately, but also because the project did not have records of the total amount of time spent by government officials on activities strictly related to the project. One source of co-financing for which there was an estimate was the financial contribution provided by local partners in the pilot initiatives implemented at the regional level (a description of these initiatives can be found in the effectiveness section of this report). The total amount of co-financing reported for three pilots was about US$ 35,000. However, due to lack of records, the evaluation team was not able to validate this claim.


Tag: Efficiency Global Environment Facility fund Government Cost-sharing Country Government SDG Integration

20.

3.2.5. Monitoring and Evaluation Design at entry

The design of the Monitoring and Evaluation provided in the Project Document comprises the standard tools used in most UNDP projects in accordance with established UNDP and GEF procedures. Box 6 summarizes (below) these tools, as outlined in the Project Document. The logical framework matrix (attached in Annex V of this report) provides the logical structure for monitoring the project's performance and delivery using a set of indicators and targets. At the output level the indicators are adequate, although we know with hindsight that the project changed its scope to include additional outputs, especially in the area of SDGs. However, when it comes to the outcomes, the framework is quite weak (see discussion on the quality of indicators under the section on the Analysis of the Project Document and Planning Matrix). The focus of the framework is on specific outputs like the NSDS or the SWOT and Gap Analyses, and it is not clear how by just having these outputs the project will ensure the mainstreaming of Rio principles and obligations into sectoral policies. Certainly, just having some strategic documents per se is not sufficient for the mainstreaming of certain principles into policies, let alone their implementation (and the fact that the NSDS currently remains unapproved proves this point). The Project Document would have benefitted from a more explicit exposition of the Theory of Change underlying the logic of the project which would have linked the multiple outputs and activities descried in the document to the overarching goal of mainstreaming and further to sustainable development. Furthermore, the Project Document does not outline or prescribe mechanisms for the explicit tracking of a number of parameters which would have been crucial to project implementation – such as co-financing, demonstration activities, capacity building effects of the various training events, etc. Given the challenges described above, the rating of the Monitoring and Evaluation design at entry point is “Moderately Unsatisfactory”. 


Tag: Challenges Effectiveness Monitoring and Evaluation Project and Programme management Quality Assurance Results-Based Management Risk Management Data and Statistics SDG Integration SDG monitoring and reporting

21.

3.2.5. Monitoring and Evaluation Design at entry (continuation)

A Lessons Learned Log was not used formally as a separate tool, but lessons learn were analyzed when annual PIRs were prepared, as well as when the project conducted Quality Assurance each year in December. Lessons were discussed in Project Board meetings and analyzed in annual Progress Reports. Tools that were not used but could have been considered are: Issues Log and GEF's Capacity Development Scorecard. No external audits were organized for this project. The project could have tracked more effectively a number of crucial parameters. The following are the most important.


Tag: Global Environment Facility fund Green Climate Fund Monitoring and Evaluation Project and Programme management Quality Assurance Results-Based Management Risk Management Data and Statistics SDG Integration

22.

3.2.6. Execution and Implementation Performance of the Executing Agency (MENR)

Because the project was implemented by UNDP through the DIM modality, MENR's role was that of a key national partner.41 Ministry representatives attended project events, suggested comments on draft analytical reports, participated in Project Board meetings and provided recommendations, etc. The Project Board met twice, but communications with board members were maintained regularly and they were consulted on major issues. MENR did not assign a Project National Director for this project. 

However, as has been mentioned in the previous sections, government organizations, including MENR, could have demonstrated more intensive engagement, ownership and leadership in relation to the project. A number of challenges internal to the government did not allow this to happen: - Restructuring of the MENR, high turnover of staff, frequent change of personnel, including Minister (3 different ministers during the lifetime of the project), GEF's operational and political focal points and national focal points on UNCBD, UNCCD and UNFCC. - Lack of clarity on departments and representatives assigned to project implementation. - Lack of leadership among other ministries and lack of inter-governmental coordination. Overall, the rating of MENR?s engagement with the project is “Moderately Satisfactory”. After the implementation modality has been changed to DIM, the UNDP assumed a role of Executing Agency. 


Tag: Public administration reform Implementation Modality Project and Programme management Results-Based Management Country Government

23.

3.3. Project Results

This section of the report is organized along the four standard dimensions of UNDP evaluations: i) relevance - the extent to which the project was relevant to the country?s priorities and needs; ii) effectiveness - whether the project was effective in achieving the desired and planned results; iii) efficiency - whether the process of achieving results was efficient; iv) sustainability - the extent to which the benefits of the project are likely to be sustained; and, v) mainstreaming – the extent to which considerations related to gender, vulnerable groups, conflict, poverty, etc., were incorporated into project activities.


Tag: Effectiveness Relevance Project and Programme management Results-Based Management Country Government Policy Advisory Technical Support Data and Statistics SDG Integration Civil Societies and NGOs

24.

3.3 Overall Results - 3.3.1. Overall Results (continuation)

The project produced a considerable number of analytical reports – policy analyses and reviews, strategies, guidelines, courses on many topics, templates, articles, videos, presentations, etc. The extent to which this body of research will be incorporated into public policy and programmes and will influence government actions is unknown for two reasons: first, the project did not keep track of change generated by them; and, second, it is too early for the effects of this research to have fully played out. Nevertheless, the amount of knowledge and anaysis that was created represents a good research base which reserachers and policymakers can tap into to develop good policy. The project also stimulated the engagement of local academics and researchers with sustainable development matters – 44 local experts were hired by the project to conduct different pieces of research (see Table 12 for the complete list of local experts engaged by the project).


Tag: Communication Knowledge management Project and Programme management Results-Based Management Awareness raising Institutional Strengthening National Institutions

25.

3.3.2. Relevance

This section provides an assessment of the relevance of the project. While there may be many criteria for assessing relevance, here it will be assessed along the following dimensions: i) relevance to the country's needs and priorities; ii) relevance to UNDP?s global corporate strategies & means of implementation; and, iii) relevance to UN Country Priorities and UNDP?s Country Mandate and Strategy.

Relevance to the country’s needs and priorities - The feedback received from national stakeholders, including government officials, NGOs and research institutions participating in project activities, donors, and the UNDP CO staff was unambiguously positive. It was stated that the project was fulfilling an important role among development partners, in particular, providing analytical evidence on the importance of sustainable development, helping improve inter-sectoral coordination, and contributing to policy changes. The project is also relevant to the national processes of mainstreaming climate change and environmental concerns into national development policies.


Tag: Relevance Policies & Procedures Project and Programme management Strategic Positioning Country Government Civil Societies and NGOs

26.

3.3.3. Effectiveness

Effectiveness in the context of this assessment means the extent to which the project achieved what it planned to achieve at the outset. This section provides a brief overview of the project's main achievements. One of the project's main contributions was the production of a wide range of analytical reports related to various dimensions of sustainable development. Table 8 (below) lists the main analytical reports that were produced in the course of the project, whereas the rest of this section provides a more detailed description of some of them. As has been discussed in previous sections, one of the main results of this project was the development of the National Sustainable Development Strategy – the so-called NSDS-2030.  Its overarching aim is to harmonize Ukraine's economic, social and ecologic development in the medium term. The strategy includes goals related to rule of law and inclusive governance, high quality education, healthy lifestyle and well-being, to list a few. It also serves as a tool for the broad-based implementation of priority actions resulting from the Rio Conventions (which is the core objective of the project).


Tag: Climate Change Adaptation Biodiversity Effectiveness Policies & Procedures Coordination Institutional Strengthening Technical Support Data and Statistics SDG monitoring and reporting

27.

3.3.3. Effectiveness (continuation)

Another important process that underpinned the development of the NSDS was the Rapid Integrated Assessment (RIA) of SDGs. The project conducted RIA to determine the SDG's relevance to the country context, both at the national and subnational level, and identify interlinkages across targets. The assessment included national, sectoral and regional development plans, strategies and programmes and provided an overview of their alignment with SDG targets. It also assessed the need for multi-sectoral coordination around the achievement of specific SDG targets and recommended policy improvements and revisions of targets. The development of the NSDS was also informed by eight sectoral analyses (the so-called SWOT and Gap analyses) conducted in 2015. They were discussed intensively by representatives of eight line ministries, academia, CSOs and the private sector and were peerreviewed by more than 80 national experts. These sectoral analyses provided policy makers with recommendations on how to strengthen the implementation of the Rio Conventions through improvements in the legislative and institutional framework. At the end of 2017, the project conducted assessments of the implementation status of the recommendations provided in the SWOT and Gap analyses.


Tag: Climate Change Adaptation Biodiversity Effectiveness Civic Engagement Public administration reform Institutional Strengthening Data and Statistics SDG Integration SDG monitoring and reporting Private Sector

28.

3.3.3. Effectiveness (continuation)

To demonstrate in practical terms the benefits of mainstreaming the Rio Conventions and implementing the SDGs, the project supported three regional pilots in 2016. These were shortterm initiatives with a focus on demonstrating practical results. Two initiatives were aimed at establishing centers of best practices for sustainable development - one in Eastern Ukraine and the other in Central Ukraine. The third initiative targeted the restoration of land affected by the illegal extraction of amber in Olevsk District, Zhytomyr Region. The average duration of the pilots was 8 months, and the budget of each did not exceed US$ 50,000. They were administered through the GEF Small Grants Programme, which ensured efficiencies and compliance with GEF regulations and procedures. Box 8 below provides a brief description of the three pilot initiatives.


Tag: Effectiveness Communication Knowledge management Policies & Procedures Institutional Strengthening SDG Integration SDG monitoring and reporting

29.

3.3.4. Efficiency

This section provides an assessment of the project's efficiency. As the term efficiency is typically used to indicate the cheapest way of achieving a particular result, best assessments of efficiency are standard cost-benefit analyses which quantify the benefits and costs of an intervention and compare them to certain benchmarks. Given the lack of project level data, especially the inability to link broad outcomes to specific project activities, this type of estimation was not possible for this evaluation. Instead, to assess efficiency, the report focuses on a number of parameters which are closely associated with efficient project management. These parameters are categorized into the following categories: i) budget execution rates; ii) cost structure; iii) timeliness of project activities; and, iv) synergies and linkages. 

Budget Execution Rates Budget execution rates can be an adequate indicator of the project's efficiency because inefficient projects usually have delays in expenditure which results in higher amounts of spending occurring at accelerated rates closer to project end dates. This typically leads to hurried decisions and hastened implementation which is rarely efficient. Table 10 shows the project's execution rates based on planned expenditure as per Project Document. Clearly, the project had a slow start in the first year with just 50% of the planned budget executed. Execution was slow in the second year too, but this time only for the second component. In the third year and in the year of extension (fourth year), the project was able to ratchet up the execution tempo and managed to spend all the allocated resources.

Cost Structure Another indicator of project efficiencies is the composition of expenditures. In particular, administrative costs are an important factor to examine because unusually high administrative costs are a sign of inefficient management. Table 11 (below) shows the composition of project expenditure for all years of project implementation.


Tag: Climate Change Adaptation Efficiency Operational Efficiency Programme Synergy SDG Integration SDG monitoring and reporting

30.

3.3.4. Efficiency(continuation)

Synergies and Linkages As has been mentioned, the project?s pilot initiatives were administered under the Small Grants Programme which allowed for fast and efficient execution, given SGP?s established systems and many years of experience with implementing such initiatives. On the other hand, as discussed in the “linkages” section, the project had significant potential for linkages and synergies with other UNDP projects which were not fully exploited. As has been argued already, the Project Document did not explore these linkages and did not foresee the establishment of any systems of collaboration. Neither was adaptive management used to take advantage of them. 

As for cooperation with other international development organizations, one good example is the collaboration with GIZ's “Programme to Support the Green Modernization of the Ukrainian Economy” on the consultations on the development of the SDG?s baseline and benchmark indicators conducted in Dnipro and Lutsk. This joint work, and subsequent regular exchange on results and planned activities, led to the signing at the end of 2017 of a 70,000 Euro contribution agreement between UNDP and GIZ on further collaboration on the implementation of SDGs. Efforts were made by the project team to connect more closely with other donors operating in the area of sustainable development, but no joint activities materialized. In spite of the implementation delays resulting from the political crisis and the conflict in the East, as well as the modification of the project scope to include the SDG process, the project team has for the most part made efficient use of resources. More effort should have been invested in fostering partnerships and linkages with other UNDP projects and similar interventions of international donors. Overall, the efficiency dimension of the project is rated as “Moderately Satisfactory”.


Tag: Climate Change Adaptation Efficiency Partnership Programme Synergy SDG Integration SDG monitoring and reporting

31.

3.3.5. Sustainability

While the sustainability of project outcomes is shaped by a number of factors, the focus of this section is on risks related to financial, sociopolitical, institutional, and environmental sustainability of project outcomes. Financial resources. The main objectives pursued by this project – mainstreaming of environmental concerns into policy and legal frameworks and the adaptation of the SDGs – involve high-level policy work that does not entail significant financial resources. These activities require foremost national ownership and the commitment of the highest levels of government. Financing is more relevant for the continuity of the results of the pilot initiatives involving communities and local governments at the sub-national level. At this level, continued financing is important because it is an indication of commitment and ownership from the partners, and as such an important aspect of sustainability. As indicated in the “project financing” section, the project reported that local communities and governments provided about US$ 35,000 in cofinancing for the local pilot initiatives. It is expected that financing will continue to be provided by the local NGOs and governments involved, but it is still too early to say how far into the future they will be able to sustained the structures (i.e. sustainable development centers) established by the project.As far as support from international organizations on aspects related to capacity development is concerned, there is a sufficient degree of interest for this kind of work to ensure that the necessary amount of financing will be available. This will also depend on UNDP's continued engagement in this area and its ability to position itself as a natural leader for this type of work. A good step in this direction was the securing of 70,000 Euro of financing from GIZ to continue some of the SDG-related activities initiated through this project. Given these positive examples, the likelihood of sustainability of the project's outcomes from a financial perspective is rated as “Moderately Likely


Tag: Climate Change Adaptation Climate change governance Environmental impact assessment Sustainability Ownership Inclusive economic growth Data and Statistics

32.

3.3.5. Sustainability (continuation)

Institutional framework and governance(continuation) At the local level, the pilots were good mechanisms for identifying opportunities for quick wins (although not clearly outlined in the Project Document63). For example, the establishment of the pilot Center for Best Practices for Sustainable Development in Vyhrayiv, Cherkasy region, was a way of showing quick results with a strong demonstration effect. The village committee is reported to have taken full ownership of the center and has allocated finances for its operation. The grant recipient reported that about 15 new communities have now expressed interest in establishing similar centers in their communities. Similarly, the Varvarivka community in the Luhansk region became interested in the concept of sustainable development and decided to develop its own sustainable development strategy with support from the project. Another example of institutionalization is the approval of the Conception of Land Protection against Deterioration and Desertification (2015) and the National Action Plan on Prevention Land Degradation and Desertification (2016) by the Cabinet of Ministers. These instruments were developed on the basis of activities conducted by the project and incorporated the provisions of the Convention to Combat Desertification. Weaker aspects of sustainability were the lack of adoption of the NSDS by the government (components of this document are implementing now in some ways and formats). This is linked to the need for national leadership at the highest level. Another factor discussed earlier in the report is the need to institutionalize sustainability concerns in budget allocations. Also, widespread corruption, especially at the institutional level, presents a risk for the sustainability of the mainstreaming results. Given this, the likelihood of sustainability from the governance perspective is rated as “Moderately Likely”.

 


Tag: Climate Change Adaptation Environmental impact assessment Sustainability Ownership Policies & Procedures Risk Management

33.

3.3.6. Mainstreaming

The evaluation found that the project had mainstreamed reasonably well cross-cutting programming principles such as capacity gender equality, the rights of vulnerable groups, etc. Abundant evidence indicated that women were involved in all stages of the project cycle from planning to implementation, monitoring and evaluation. They participated in large numbers in the expert groups working on the various analysis documents and reports, as well as other project activities. Many expert teams were led by distinguished Ukrainian female academics. They also participated in project activities at the sub-national level, with female representatives of local governments, CSOs and business organizations playing a crucial role. The pilots demonstrated best practices in the promotion of gender equality. In particular, the creation of the center forsustainable development in Eastern Ukraine promoted women's leadership qualities in cooperation with the “Businesswomen of Donbass Club” in Slaviansk.


Tag: Challenges Effectiveness Gender Equality Gender Mainstreaming Women's Empowerment Human rights Communication Integration Knowledge management Monitoring and Evaluation Civil Societies and NGOs Vulnerable Women and gilrs Youth

Recommendations
1

Evaluation Recommendation 1: Strengthen Engagement with SDGs at the Sub-national Level

The Rio Project has established good foundations for further UNDP engagement with the SDGs, which the CO has already begun to build on. It has also contributed to UNDP’s partnership with local communities and governments in the pilot areas. The project has showcased that the combination of SDGs and local development carries significant potential for UNDP, as has been argued in more detail in Chapter 4 of the report. UNDP should continue to strengthen its support for public institutions, particularly at the sub-national level, to adapt SDG targets and indicators to local circumstances, establish monitoring systems and create databases for monitoring progress, and report results nationally and internationally. It should also continue to explore ways of getting international donors more closely engaged with the SDGs. Furthermore, efforts should be made to link all this work more closely with the broader reform agenda led by the National Reform Council.

2

Evaluation Recommendation 2: Strengthen Synergies and Linkages between Projects

Drawing on lessons from the experience of the Rio Project, UNDP should further strengthen collaboration between projects, and where feasible establish integrated frameworks for project planning and implementation. The latter is certainly difficult, given the projects’ short timeframes and varying funding modalities, but it might be possible at the sub-national level if a sufficient number of projects are up and running. This will require the application of development of principles and methods for integrated ecosystem-based management. Using the example of collaboration between the Rio and SGP projects, UNDP could explore the establishment of a common platform for managing aspects of projects that share similar objectives. Such platform may combine not only elements related to information sharing, data systems, monitoring and evaluation, but also implementation tools such as systems for procurement, recruitment, awareness raising, etc. If such a platform is established across projects, it should be fully integrated with the CO’s results-based management system.

3

Evaluation Recommendation 3: Using the M&E System to Track Important Parameters

UNDP should examine how projects use their M&E systems to track important aspects of their work with a view to improving the availability of information for management purposes. Measuring some of these dimensions was a challenge in this evaluation. The following are a few worth considering.

  • Uptake of project outputs (studies, training, etc.) and the degree to which they serve their intended purpose – Projects should monitor the extent to which research and analytical documents they produce get incorporated into national policies and programmes.
  • Capacity of beneficiaries – Projects should also try to track the degree to which the capacity of participants taking part in the various training programmes organized by the project has improved.
  • Replication effects of pilot initiatives, the lessons they generate during the piloting stage and the extent to which the get scaled up – One key characteristic of pilots is that they serve to produce lessons which when shared lead to replication. They are key vehicles for transmitting experience and play a crucial role for upscaling and replication. However, it is not clear how the lessons are collected, analyzed, synthesized and shared. UNDP should develop a tracking mechanism for pilot initiatives, including documenting results, lessons, experiences and good practices. The tracking of pilots should be fully integrated into the CO’s results-based management systems and lessons learned and best practices should be managed as an important component of the CO’s knowledge management strategy.
  • Co-financing – The CO should strive for a more standardized definition of co-financing and monitor it more effectively by developing a tracking system at the project level.
1. Recommendation:

Evaluation Recommendation 1: Strengthen Engagement with SDGs at the Sub-national Level

The Rio Project has established good foundations for further UNDP engagement with the SDGs, which the CO has already begun to build on. It has also contributed to UNDP’s partnership with local communities and governments in the pilot areas. The project has showcased that the combination of SDGs and local development carries significant potential for UNDP, as has been argued in more detail in Chapter 4 of the report. UNDP should continue to strengthen its support for public institutions, particularly at the sub-national level, to adapt SDG targets and indicators to local circumstances, establish monitoring systems and create databases for monitoring progress, and report results nationally and internationally. It should also continue to explore ways of getting international donors more closely engaged with the SDGs. Furthermore, efforts should be made to link all this work more closely with the broader reform agenda led by the National Reform Council.

Management Response: [Added: 2018/07/13] [Last Updated: 2021/01/20]

UNDP fully agrees with these recommendations and certain actions have already been made.

Specifically, UNDP has an agreement with two oblast administrations to prepare oblast-specific reports with the list of SDGs targets and indicators identified in the process of their adaptation the context. UNDP is developing its strategy on SDGs localization in the rest of oblasts of Ukraine, learning from experience in the first two oblasts. The local presence in Ukraine that UNDP has due to its recent projects (e.g., CBA), is used as an asset – namely, the contacts with local administrations, the newly signed Memoranda of Cooperation with many oblast administrations, other partner networks and resource centers established with support from UNDP in the past.

Furthermore, UNDP will explore cooperation with other donor-funded projects operating at the oblast level, especially those on decentralization.

Specific actions proposed below.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Develop a strategy for SDGs localization using the experience of localization in Volyn and Dnipro oblasts
[Added: 2018/07/13]
SDGs DST, ENV portfolio 2018/05 Completed Completed
1.2 Support the Government (MEDT and Statistics Service of Ukraine) in identifying best platform and mechanisms to organize regular monitoring of SDGs implementation at the national and oblast levels.
[Added: 2018/07/13] [Last Updated: 2020/08/07]
UN SDG task force, UNDP CD, SDGs DST, SDG advisor 2019/11 Completed In 2019 UNDP started cooperation with State Statistics Service on the development of a platform for SDGs monitoring at the national level. History
1.3 Provide to the Government (MEDT and Statistics Service of Ukraine) expert support on development on SDGs indicators which can be used at oblast level as an alternative to those national SDGs indicators which are not applicable at the oblast level
[Added: 2018/07/13]
GIZ/UNDP project, SDGs DST 2018/05 Completed Through the GIZ/UNDP
1.4 Intensify talks with donors who can provide technical assistance to support national SDGs process. Two-fold task: resource mobilization or integration of SDGs into current donor-funded projects
[Added: 2018/07/13] [Last Updated: 2018/12/18]
UNDP CD, Strategic Unit, SDGs DST 2018/11 Completed Per 2018/11, over $1.5 million mobilized from Sida, GIZ and SlovakAid on e-learning on SDGs, local SDG implementation, sharing of international practices on SDG implementation, and strengthening the work of Parliament of SDGs. History
1.5 Intensify advocacy for inclusion of SDGs into the agenda of Ukraine’s development priorities
[Added: 2018/07/13] [Last Updated: 2018/10/26]
UNDP CD, SDGs DST 2018/10 Completed MAPS mission conducted in March – August 2018 gave a significant burst to this process. The cabinet of Ministers instructed in July 2018 the line ministries to align their annual plans to SDGs. The Ministry of Regional Development has requested inclusion of UNDP in working group on regional development indicators, with an intention to align regional development priorities (and monitoring indicators) to SDGs. History
1.6 Strengthen cooperation with development partners by going beyond information sharing and forging collaboration at the level of project activities
[Added: 2018/07/13] [Last Updated: 2020/08/07]
CO, SDGs DST 2019/11 Completed In 2019-2020, UNDP hold SDGs advocacy days aimed to strengthen its partnership network by identifying new development partners. Additionally, in 2020 at the oblast level, UNDP made an analysis on development interventions (areas) implemented by other development agencies. History
2. Recommendation:

Evaluation Recommendation 2: Strengthen Synergies and Linkages between Projects

Drawing on lessons from the experience of the Rio Project, UNDP should further strengthen collaboration between projects, and where feasible establish integrated frameworks for project planning and implementation. The latter is certainly difficult, given the projects’ short timeframes and varying funding modalities, but it might be possible at the sub-national level if a sufficient number of projects are up and running. This will require the application of development of principles and methods for integrated ecosystem-based management. Using the example of collaboration between the Rio and SGP projects, UNDP could explore the establishment of a common platform for managing aspects of projects that share similar objectives. Such platform may combine not only elements related to information sharing, data systems, monitoring and evaluation, but also implementation tools such as systems for procurement, recruitment, awareness raising, etc. If such a platform is established across projects, it should be fully integrated with the CO’s results-based management system.

Management Response: [Added: 2018/07/13] [Last Updated: 2021/01/20]

UNDP mostly agrees with this recommendation, however establishing this platform and its effective management is a challenging. The CO will consider what are the opportunities.

In order to avoid silos and move beyond a projectized, ad hoc modality, UNDP has gradually moved towards a programme approach in its support to countries worldwide, including Ukraine. The next step in this regard is the switch towards a full area-based approach in supporting the roll-out of the sustainable development agenda at the subnational level where relevant. Specific actions are listed below.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Undertake a review of the operational processes in the UNDP Ukraine portfolios where common platforms and mechanisms can be applied. Assess which processes are efficient and which need improvement (procurement, grant management, interns engagement etc.). Clarify how human resources (project staff and consultants) in projects can be used more effectively, where they can contribute to multiple projects and or co-funded from peer projects.
[Added: 2018/07/13] [Last Updated: 2019/05/02]
UNDP CD, SDGs DST 2019/04 Completed In the follow-up to this recommendation and CO Audit in 2018, the office implemented a new SOP on grant management that is coherent across all portfolios. The capacities of the procurement and HR units have been strengthened with additional personnel to avoid operational bottlenecks. The portfolio has undergone significant restructuring and managerial and assistant positions are now shared between the projects, increasing overall efficiency of the implementation. History
2.2 Ensure regular review of the list of projects partners – to see where different projects work with the same beneficiaries and how to improve coherence of UNDP programmes
[Added: 2018/07/13] [Last Updated: 2018/10/26]
UNDP CD, SDGs DST, All portfolios 2018/09 Completed UNDP Ukraine has established the new SDG area based portfolio with a solid network of UNDP Oblast SDGs Coordinators, aimed at planning and implementing UNDP Ukraine activities at oblast-level. The proposed new way of operating together with existing area based networks with government, civil society, and other partners will allow UNDP Ukraine to act as a unified force, as a coordinated advocacy and implementation structure, as a central convening agency. A mapping of different UNDP projects and their implementation geography has been conducted and can form the basis for further area-based planning and implementation. History
2.3 Review the process through which new projects are designed and negotiated with partners to ensure that new projects built on experience, expertise and partnerships of earlier and ongoing UNDP projects in Ukraine
[Added: 2018/07/13]
Strategic unit 2018/02 Completed Regularly as part of a project concept development
2.4 Use SDGs as a cross cutting topic to be integrated into results frameworks for all UNDP projects. This will allow building synergies between the projects.
[Added: 2018/07/13] [Last Updated: 2018/10/26]
M&E and Strategic unit 2018/09 Completed UNDP Ukraine has established the new SDG area based portfolio with a solid network of UNDP Oblast SDGs Coordinators, aimed at planning and implementing UNDP Ukraine activities at oblast-level. The proposed new way of operating together with existing area based networks with government, civil society, and other partners will allow UNDP Ukraine to act as a unified force, as a coordinated advocacy and implementation structure, as a central convening agency. A mapping of different UNDP projects and their implementation geography has been conducted and can form the basis for further area-based planning and implementation. History
2.5 Improve communications activities by more frequent engagement with mass media and posting articles and op-eds about UNDP activities where UNDP’s activities are presented in integrated manner (as ‘delivering as one’ instead of delivering fragmented activities). SDGs, climate change, human rights, gender equality etc. can be a common underpinning topic which consolidates all activities under one logically consistent umbrella
[Added: 2018/07/13] [Last Updated: 2018/10/26]
Communications Unit, all projects 2018/09 Completed UNDP media presence has increased with more article and op-eds are published, covering many cross-functional topics at once. UNDP CO moved all projects websites to one UNDP Ukraine site. History
3. Recommendation:

Evaluation Recommendation 3: Using the M&E System to Track Important Parameters

UNDP should examine how projects use their M&E systems to track important aspects of their work with a view to improving the availability of information for management purposes. Measuring some of these dimensions was a challenge in this evaluation. The following are a few worth considering.

  • Uptake of project outputs (studies, training, etc.) and the degree to which they serve their intended purpose – Projects should monitor the extent to which research and analytical documents they produce get incorporated into national policies and programmes.
  • Capacity of beneficiaries – Projects should also try to track the degree to which the capacity of participants taking part in the various training programmes organized by the project has improved.
  • Replication effects of pilot initiatives, the lessons they generate during the piloting stage and the extent to which the get scaled up – One key characteristic of pilots is that they serve to produce lessons which when shared lead to replication. They are key vehicles for transmitting experience and play a crucial role for upscaling and replication. However, it is not clear how the lessons are collected, analyzed, synthesized and shared. UNDP should develop a tracking mechanism for pilot initiatives, including documenting results, lessons, experiences and good practices. The tracking of pilots should be fully integrated into the CO’s results-based management systems and lessons learned and best practices should be managed as an important component of the CO’s knowledge management strategy.
  • Co-financing – The CO should strive for a more standardized definition of co-financing and monitor it more effectively by developing a tracking system at the project level.
Management Response: [Added: 2018/07/13] [Last Updated: 2021/01/20]

This recommendation is fully accepted as it is fully in line with what is required for quality assurance and ROAR reviews. RBM approach should be used to a larger extent - at the stage of project formulation and, with adjustments, at the stage of implementation and monitoring. Specific additional actions are proposed below.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 To provide guidelines, advice and assistance on M&E frameworks and reporting to project staff on a regular basis
[Added: 2018/07/13] [Last Updated: 2018/12/13]
M&E unit, ENV Portfolio 2018/11 Completed Full-time M&E Officer (UNV) has been deployed to assist the projects within the portfolio with developing M&E frameworks, collecting outcome and output level data, reviewing reports and proposal to ensure they are result-oriented and comply with UNDP RBM standards. A comprehensive RBM training for the new portfolio team was conducted in November 2018. History
3.2 To monitor the extent to which research and analytical documents produced get incorporated into national policies and programmes
[Added: 2018/07/13] [Last Updated: 2019/10/25]
ENV Portfolio, M&E unit 2019/07 Completed UNDP CO has reinforced internal controls and accountability to the government of Ukraine for all research conducted. History
3.3 Reinforce a tracking system for assessing capacity of participants taking part in the training programmes
[Added: 2018/07/13] [Last Updated: 2020/08/07]
M&E unit, ENV Portfolio 2019/10 Completed The new Integrated Monitoring and Reporting System (IMRS) is underway to be implemented in CO tracking the project’s activities in a more precise, accurate and timely manner. History
3.4 Develop a mechanism to analyse and disseminate lessons learnt at all UNDP projects based on analysis of M&E indicators, projects results, grant initiatives etc.
[Added: 2018/07/13] [Last Updated: 2018/12/13]
M&E unit, ENV Portfolio 2018/11 Completed Portfolio carries biannual workplan reviews, that include discussion of lessons learnt, progress on indicators and project results, grant initiatives and results of pilots, Meetings are held for the entire portfolio to allow for exchange of information between the projects and facilitation the discussion on building stronger synergies and linkages between the projects. In addition reinforced annual (at minimum) board meetings also provide a platform for discussion and lessons learnt dissemination with external stakeholders and final beneficiaries. History
3.5 Develop guidance to the projects on monitoring co-financing from beneficiaries (against co-financing set in the Project Documents; and against co-financing indicated for individual initiatives, e.g., which UNDP receives from applicants for grants)
[Added: 2018/07/13] [Last Updated: 2018/07/27]
M&E unit, ENV Portfolio 2018/06 Completed History
3.6 Improve M&E monitoring system by introducing M&E indicators which are available from external sources, not those produced by the projects implementing the activities. This will allow to get more reliable and less subjective data for analysis of projects effectiveness
[Added: 2018/07/13] [Last Updated: 2018/12/13]
M&E unit, ENV Portfolio 2019/11 Completed All ongoing projects within the environment portfolio now have M&E frameworks that are in line with UNDP RBM standards. The indicators rely on a mix of sources, including project data, as well as national and regional external sources. New UNDP Strategic Plan, Country Programme Document and UNDAF frameworks for 2018-2022 ensure regular collection of data that demonstrated progress at output and outcome levels. The indicators enable management to make evidence-based decisions. History
3.7 Develop a tracking mechanism for grant and pilot initiatives, including documenting results, lessons, experiences and good practices. The tracking of grants and pilots should be fully integrated into the CO’s RBM systems.
[Added: 2018/07/13] [Last Updated: 2020/08/07]
CO, M&E unit, ENV Portfolio 2020/12 Completed UNDP Ukraine has introduced Integrated Monitoring and Reporting Platform to track grants and pilots in one system, introduced across all projects History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org