EVALUATION OF THE COUNTRY PROGRAMME OUTCOME 1.3 “Ecosystems and natural resources are protected and sustainably used, and human settlements are resilient to natural and human-induced disasters and climate change”

Report Cover Image
Evaluation Plan:
2016-2020, Kazakhstan
Evaluation Type:
Outcome
Planned End Date:
12/2018
Completion Date:
12/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
30,000

Share

Title EVALUATION OF THE COUNTRY PROGRAMME OUTCOME 1.3 “Ecosystems and natural resources are protected and sustainably used, and human settlements are resilient to natural and human-induced disasters and climate change”
Atlas Project Number: 97247,88403,82364,81141,86627,89800,81775,60598
Evaluation Plan: 2016-2020, Kazakhstan
Evaluation Type: Outcome
Status: Completed
Completion Date: 12/2018
Planned End Date: 12/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.1.1 Low emission and climate resilient objectives addressed in national, sub-national and sectoral development plans and policies to promote economic diversification and green growth
  • 2. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation
Evaluation Budget(US $): 30,000
Source of Funding: UNDP
Evaluation Expenditure(US $): 18,600
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
N/A
GEF Evaluation: No
Key Stakeholders: Ministry of National Economy, Ministry of Energy, others
Countries: KAZAKHSTAN
Lessons
Findings
Recommendations
1

Results-Based Management at the Sectoral Level

In preparation for the development of the new CPD, the CO should strengthen the RBM system at the programme and project level.

  • While it is hard to develop an RRF at the CPD level that will stay realistic and useful till the end of the programme cycle (given the uncertainty of programme funding), it is important for the CO to strengthen the system through which it tracks programme and project results on an ongoing basis. Information on some basic indicators that are driven by the nature of projects undertaken by the CO should be quickly available at any time and should be used by the CO management and sector teams to monitor and manage activities.
  • The CO should strengthen the quality criteria for the development of project documents. This is an area where there is high predictability and having a well-structured project RRF is essential for the monitoring and management of project activities.
  • The sector will also benefit from the development of a Theory of Change that connects all the specific pieces (projects). This is not just a theoretical exercise, but has practical value in that it will provide the team with insights into how these individual projects could be tied more effectively together.
  • Also, the CO should strengthen quality criteria for evaluations and the way it manages the learning that is derived from them. The CO should develop minimum quality criteria for project evaluations and should establish a tracking system to closely monitor their quality.
2

Positioning and Resource Mobilization

In terms of positioning, the CO should continue to consolidate its position in the areas of biodiversity and natural resources, climate change and energy efficiency, and waste and water management. These are areas where UNDP Kazakhstan has already positioned itself well and is quite competitive. The DRR sector could offer opportunities for further work, if funding will be available. Furthermore, one cross-cutting issue with which the sector should be engaged more actively is the Sustainable Development Goals. Work on the SDGs should be coordinated closely between the sectors, but the E&E sector can play a much bigger role. Potential work the E&E sector could engage in includes the incorporation of SDGs in strategic documents and policies, establishing national targets and baselines to measure progress, supporting the distribution of responsibilities among government institutions, establishing data and monitoring systems that support SDGs, and assisting with reporting nationally and internationally. Overall, the E&E sector is well positioned to support through the SDG process the mainstreaming of the environmental concerns into the country’s legal and policy framework and assist the government in further implementing the concept of Green Economy.

The highest priority for the CO in the area of E&E now is the mobilization of funding. Ongoing efforts with traditional donors which as GEF and the EU should continue, although the amount of financing expected from these sources will continue to decline. With regards to the EU, the CO should undertake a more systematic assessment and identification of opportunities by researching what the EU is planning for Kazakhstan.  In this context, the CO could launch a more organized process of exploration concerning all sectors and involving UNDP’s representation in Brussels and New York. As far as non-traditional donors are concerned, UNDP should continue its current efforts at identifying new modes of engagement. The IFIs, in particular, present good potential which the CO should explore by developing innovative mechanisms – such as the management of technical assistance components in the framework of loan agreements. 

3

Programme Integration

The CO should strengthen integration between sectors by establishing integrated frameworks for project planning and implementation. Governance and diversification activities, especially at the sub-national level, provide the E&E sector with a platform on which to embed environmental and energy efficiency initiatives. In the upcoming CPD, the CO could consider merging E&E and diversification activities, given their complementary nature (depending on whether diversification activities will be active by that time). The CO should also explore the feasibility of integrated work plans elaborated at the regional/local level and matched with the CO’s plan at the national level. Such an area-based approach will enable UNDP to weave more effectively cross-cutting issues (such as energy efficiency, citizen engagement, transparency and accountability, gender equality) into other thematic activities (i.e. community development, service delivery, etc.). Stronger synergies may also be forged with international organizations at the sub-national level, which may also provide increased funding opportunities. UNDP can also support local authorities to facilitate more effectively donor coordination at the sub-national level. A crucial step in achieving a higher level of programme integration and consolidation at that level could be the development of a clearer strategy for how UNDP should structure itself and operate at the sub-national level.

4

Policy implementation

The CO should further strengthen its focus on implementation, by thinking beyond just the passing of laws and strategies, and considering measures that consolidate organizational structures that will implement those laws and strategies. This includes actions like the creation of organizational structures, staffing organizations and allocating funding for their operations, training management and staff to implement policies, etc. The sector team has already been doing a lot of this, but the point here is to promote a mentality shift in the programme and within the government away from “form” (how a piece of law looks like) to functionality (how a law is implemented and what effects it produces).  From this perspective, it is important that the team consider how the capability of government organizations is built and changes. For this, the CO should develop RBM systems that track implementation parameters linked to functionality and outcomes rather than form and inputs/outputs and assess more rigorously the sustainability of achievements. Project documents should contain clear criteria related to performance based on a strategy for achieving and demonstrating results. Achieving this focus on functionality and outcomes is difficult when considering the short timeframes of UNDP projects, but it is not impossible. What is important is the mentality shift which implies that UNDP staff start designing and implementing projects with these implementation considerations in mind.

5

Co-financing

The CO has had good results when it comes to co-financing. For projects that involve infrastructure investments it is essential to keep pushing for stronger competitive/market mechanisms to ensure the sustainability and scale of initiatives. Overall, the recommendation here is to stay on the same path and not backtrack, because market-based solutions to infrastructure problems are essential for their sustainability. Instead of providing grants, UNDP should keep strengthening incentives that promote access to international financial institutions and banks for finance.

6

Awareness Raising

In the area of awareness raising and information sharing, the CO should reassess its approaches, methods and results more strategically. This is an area where there have been significant shifts in research and practice recently and it is time for UNDP to upgrade its approach. First, the CP should recognize that information sharing and awareness raising are done for a simple reason – to change  behavior. So, when designing  information campaigns and events, it is important to ask what behavior and whose behavior the programme or project is seeking to change. This requires careful thinking about the behavior the programme/project seeks to promote and the agents whose behavior it wants to change. As a next step, it is also important to understand what type of information and what channel of information has the potential to change the identified behavior in the target group. The way the information is packaged matters a lot, but who carries the information and how that person is perceived by the target group matters even more. In this sense, it is important to understand whose opinion matters for the target group and how that opinion can be constructed and used to influence behavior. It is also important to recognize that individuals operate in a social environment and that human behavior is largely influenced by social norms set by the community in which an individual embedded. So, to change an individual’s behavior, it is important to understand the prevailing social norms in his/her community and the factors that shape those social norms. This is something that the CO could examine a bit more closely in the context of the development of the new CPD and new projects.

1. Recommendation:

Results-Based Management at the Sectoral Level

In preparation for the development of the new CPD, the CO should strengthen the RBM system at the programme and project level.

  • While it is hard to develop an RRF at the CPD level that will stay realistic and useful till the end of the programme cycle (given the uncertainty of programme funding), it is important for the CO to strengthen the system through which it tracks programme and project results on an ongoing basis. Information on some basic indicators that are driven by the nature of projects undertaken by the CO should be quickly available at any time and should be used by the CO management and sector teams to monitor and manage activities.
  • The CO should strengthen the quality criteria for the development of project documents. This is an area where there is high predictability and having a well-structured project RRF is essential for the monitoring and management of project activities.
  • The sector will also benefit from the development of a Theory of Change that connects all the specific pieces (projects). This is not just a theoretical exercise, but has practical value in that it will provide the team with insights into how these individual projects could be tied more effectively together.
  • Also, the CO should strengthen quality criteria for evaluations and the way it manages the learning that is derived from them. The CO should develop minimum quality criteria for project evaluations and should establish a tracking system to closely monitor their quality.
Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The country office accepts the recommendation.

UNDP Kazakhstan has been continuously working to enhance the monitoring system and RBM and will ensure the improvement of planning and reporting (internally and outside UNDP) both on the programme and projects outcome level. Programme and projects staff are duly informed and trained on the reporting tools and how to make best use of it. Anyhow the country office sees the place for the improvement of M&E system  and RBM in the office.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Reinforcement of M&E and RBM in the country office Standard Operating Procedures for the project management through the design, implementation and closure stages
[Added: 2019/01/03] [Last Updated: 2019/09/15]
M&E Associate, SDU Unit 2019/11 Initiated History
Maintenance of the practice of M&E trainings for the projects staff for M&E programming and implementation in the projects.
[Added: 2019/01/03] [Last Updated: 2019/09/15]
M&E Associate, SDU Unit 2019/11 Initiated History
Introduce and maintain the exchange of the best M&E and RBM practices among the programme and projects i.e.project document development, report writing, presentation of the project results to the national partners.
[Added: 2019/01/03] [Last Updated: 2019/10/02]
M&E Associate, SDU Unit 2019/12 Initiated History
2. Recommendation:

Positioning and Resource Mobilization

In terms of positioning, the CO should continue to consolidate its position in the areas of biodiversity and natural resources, climate change and energy efficiency, and waste and water management. These are areas where UNDP Kazakhstan has already positioned itself well and is quite competitive. The DRR sector could offer opportunities for further work, if funding will be available. Furthermore, one cross-cutting issue with which the sector should be engaged more actively is the Sustainable Development Goals. Work on the SDGs should be coordinated closely between the sectors, but the E&E sector can play a much bigger role. Potential work the E&E sector could engage in includes the incorporation of SDGs in strategic documents and policies, establishing national targets and baselines to measure progress, supporting the distribution of responsibilities among government institutions, establishing data and monitoring systems that support SDGs, and assisting with reporting nationally and internationally. Overall, the E&E sector is well positioned to support through the SDG process the mainstreaming of the environmental concerns into the country’s legal and policy framework and assist the government in further implementing the concept of Green Economy.

The highest priority for the CO in the area of E&E now is the mobilization of funding. Ongoing efforts with traditional donors which as GEF and the EU should continue, although the amount of financing expected from these sources will continue to decline. With regards to the EU, the CO should undertake a more systematic assessment and identification of opportunities by researching what the EU is planning for Kazakhstan.  In this context, the CO could launch a more organized process of exploration concerning all sectors and involving UNDP’s representation in Brussels and New York. As far as non-traditional donors are concerned, UNDP should continue its current efforts at identifying new modes of engagement. The IFIs, in particular, present good potential which the CO should explore by developing innovative mechanisms – such as the management of technical assistance components in the framework of loan agreements. 

Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The county office accepts the recommendation.

UNDP, in particular SDU unit is included in two working groups (Planet and Prosperity), established by the Prime Minister for attaining SDGs. UNDP is closely working in this area and actively engaging in the work of the working groups. The SDU unit is focusing on diversifying the resource mobilization and partnering with financial institutions such as – IFIs, Government of Kazakhstan, Quazi-State Enterprises, Private Sector (Coca-Cola, Shell), Chambers of Commerce, Korea Forest Service, Governments of Switzerland, Germany, China, Russian Trust Fund, GCF, EU, etc. The most perspective for the SDU today is IFIs such as – World Bank, IsDB, ADB, EBRD, EIB etc. The CO fully agrees with the recommendation on developing innovative financing mechanisms and use UNDP advantages in management of technical assistance components in the framework of loan agreements.

 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Participation in at least 5 meetings of the working groups – Planet and Prosperity
[Added: 2019/01/03]
SDU Unit 2019/12 Initiated
Mobilization of GEF resources – submit at least 3 proposals for GEF financing
[Added: 2019/01/03]
SDU Unit 2019/12 Initiated
Mobilization of GCF resources – submit at least 1 full funding proposal for GCF financing
[Added: 2019/01/03]
SDU Unit 2019/12 Initiated
Mobilization of resources from Government of Kazakhstan – submit at least 1 Government cost-sharing proposal Mobilization of resources from alternative sources of financing – engage with IFIs for potential implementation of TA components and private sector
[Added: 2019/01/03]
SDU Unit 2019/12 Initiated
3. Recommendation:

Programme Integration

The CO should strengthen integration between sectors by establishing integrated frameworks for project planning and implementation. Governance and diversification activities, especially at the sub-national level, provide the E&E sector with a platform on which to embed environmental and energy efficiency initiatives. In the upcoming CPD, the CO could consider merging E&E and diversification activities, given their complementary nature (depending on whether diversification activities will be active by that time). The CO should also explore the feasibility of integrated work plans elaborated at the regional/local level and matched with the CO’s plan at the national level. Such an area-based approach will enable UNDP to weave more effectively cross-cutting issues (such as energy efficiency, citizen engagement, transparency and accountability, gender equality) into other thematic activities (i.e. community development, service delivery, etc.). Stronger synergies may also be forged with international organizations at the sub-national level, which may also provide increased funding opportunities. UNDP can also support local authorities to facilitate more effectively donor coordination at the sub-national level. A crucial step in achieving a higher level of programme integration and consolidation at that level could be the development of a clearer strategy for how UNDP should structure itself and operate at the sub-national level.

Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The country office accepts the recommendation.

The experience of Joint Programmes will be used in upcoming CPD planning with focus on integrated approaches on the national level. SDU unit is now exploring new opportunities for regional projects, such as DRR with China, Kyrgyzstan and Tajikistan, replication of Climate Box with Uzbekistan, Tajikistan, Kyrgyzstan, Armenia and Russia. UNDP Kazakhstan also participates in 2 global projects – 6th National Communication of Biodiversity and BIOFIN. In upcoming new cycle of PFD, CPD and CPAP development the CO will focus on programme integration and consolidation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Development of synergetic programme activities as part of new PFD, CPD cycles
[Added: 2019/01/03]
SDU and GLD Units, senior management 2020/12 Initiated
4. Recommendation:

Policy implementation

The CO should further strengthen its focus on implementation, by thinking beyond just the passing of laws and strategies, and considering measures that consolidate organizational structures that will implement those laws and strategies. This includes actions like the creation of organizational structures, staffing organizations and allocating funding for their operations, training management and staff to implement policies, etc. The sector team has already been doing a lot of this, but the point here is to promote a mentality shift in the programme and within the government away from “form” (how a piece of law looks like) to functionality (how a law is implemented and what effects it produces).  From this perspective, it is important that the team consider how the capability of government organizations is built and changes. For this, the CO should develop RBM systems that track implementation parameters linked to functionality and outcomes rather than form and inputs/outputs and assess more rigorously the sustainability of achievements. Project documents should contain clear criteria related to performance based on a strategy for achieving and demonstrating results. Achieving this focus on functionality and outcomes is difficult when considering the short timeframes of UNDP projects, but it is not impossible. What is important is the mentality shift which implies that UNDP staff start designing and implementing projects with these implementation considerations in mind.

Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The country office accepts the recommendation.

UNDP Kazakhstan recognizes the importance the sustainability of the project results and ensures it through the corporate mechanisms of the project progress monitoring through the entire project cycle such as Quality Assurance Assessment (QAA) of the project progress. Through all the stages the project, the country office places evidenced records of the assessment criteria: strategic direction, relevance, effectiveness, sustainability, exit strategy.

With this the country office will work to strengthen RBM parameters in the projects making putting the functionality of the results and their sustainability at the center of the project design and ensuring the tools for their monitoring and assessement throughout the projects’ cycle.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Reinforcement of M&E and RBM in the country office Standard Operating Procedures for the project management through the design, implementation and closure stages
[Added: 2019/01/03] [Last Updated: 2019/09/15]
M&E Associate, SDU Unit 2019/11 Initiated History
Develop and introduce the project exit strategy focused on the sustainability of the project outcomes as a part of the project document
[Added: 2019/01/03]
M&E Associate, SDU Unit 2019/12 Initiated
5. Recommendation:

Co-financing

The CO has had good results when it comes to co-financing. For projects that involve infrastructure investments it is essential to keep pushing for stronger competitive/market mechanisms to ensure the sustainability and scale of initiatives. Overall, the recommendation here is to stay on the same path and not backtrack, because market-based solutions to infrastructure problems are essential for their sustainability. Instead of providing grants, UNDP should keep strengthening incentives that promote access to international financial institutions and banks for finance.

Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The country office accepts the recommendation.

The CO will further focus on strengthening incentives and continue introduction of innovative financial mechanisms and building enabling environment for promoting access to finance.
 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Analyze the implemented financial mechanisms for replicability and potential scale-up and improvement in other sectors (green finance development)
[Added: 2019/01/03]
SDU Unit 2019/12 Initiated
Development of new outcome indicators in the area of market-based solutions for sustainability of UNDP interventions in the new PFD, CPD cycles.
[Added: 2019/01/03]
SDU Unit and the senior management 2019/12 Initiated
6. Recommendation:

Awareness Raising

In the area of awareness raising and information sharing, the CO should reassess its approaches, methods and results more strategically. This is an area where there have been significant shifts in research and practice recently and it is time for UNDP to upgrade its approach. First, the CP should recognize that information sharing and awareness raising are done for a simple reason – to change  behavior. So, when designing  information campaigns and events, it is important to ask what behavior and whose behavior the programme or project is seeking to change. This requires careful thinking about the behavior the programme/project seeks to promote and the agents whose behavior it wants to change. As a next step, it is also important to understand what type of information and what channel of information has the potential to change the identified behavior in the target group. The way the information is packaged matters a lot, but who carries the information and how that person is perceived by the target group matters even more. In this sense, it is important to understand whose opinion matters for the target group and how that opinion can be constructed and used to influence behavior. It is also important to recognize that individuals operate in a social environment and that human behavior is largely influenced by social norms set by the community in which an individual embedded. So, to change an individual’s behavior, it is important to understand the prevailing social norms in his/her community and the factors that shape those social norms. This is something that the CO could examine a bit more closely in the context of the development of the new CPD and new projects.

Management Response: [Added: 2018/12/27] [Last Updated: 2019/01/03]

The country office accepts the recommendation.

 

UNDP Kazakhstan has been actively communicating the project success stories inside and outside UNDP using various mass media tools.

Anyhow the country office sees the need to strengthen communications and advocacy for development initiatives putting in the center raising the awareness and information sharing to trigger the behavioral change.

UNDP will take the best use of new PPM which puts partnerships, resource mobilization, and communications at the center of UNDP programming efforts  to work with new partners and mobilize financing.

 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
To elaborate and implement Partnerships and Communications Strategy and Action Plan (PCAP) for the new CPD cycle
[Added: 2019/01/03]
Communications Officer, SDU Unit 2020/12 Initiated

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org