Final Evaluation for the Strengthening the Governance of Climate Change Finance to benefit the poor and vulnerable

Report Cover Image
Evaluation Plan:
2014-2017, RBAP
Evaluation Type:
Final Project
Planned End Date:
12/2015
Completion Date:
02/2017
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Document Type Language Size Status Downloads
Download document Final Review UNDP_SIDA Governance of CCF_ Final (25 June 2016).doc report English 679.50 KB Posted 584
Title Final Evaluation for the Strengthening the Governance of Climate Change Finance to benefit the poor and vulnerable
Atlas Project Number:
Evaluation Plan: 2014-2017, RBAP
Evaluation Type: Final Project
Status: Completed
Completion Date: 02/2017
Planned End Date: 12/2015
Management Response: Yes
Focus Area:
  • 1. Environment & Sustainable Development
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 7.4. Countries enabled to gain equitable access to, and manage, ODA and other sources of global development financing
Evaluation Budget(US $): 50,000
Source of Funding: SIDA
Evaluation Expenditure(US $): 50,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Jonathan Hampshire Independent Consultant
GEF Evaluation: No
Key Stakeholders:
Comments:

Completed 

Lessons
Findings
1.

3 Findings

3.1 Relevance and quality of design

Findings with respect to relevance and quality of design are provided in response to the three guiding questions below (drawn from the PFR matrix shown at Annex 3). Question 1: Are programme objectives appropriately aligned with global, regional, national and/or donor policy priorities and strategies on strengthening the governance of climate change financing to benefit the poor and vulnerable? 

Global policy context

n the global context, the programme's objectives appear to be drawn primarily from discussions held, and principles agreed, at the Fourth High-Level Forum on Aid Effectiveness that took place in Busan, Korea in November 2011. At the forum, the Busan Building Block (BBB) on Climate Finance and Development Effectiveness was established as a voluntary partnership to promote coherence and collaboration across climate and development communities. Its aims are to: (i) strengthen linkages between climate finance and countries' planning, budgeting and public financial management systems; (ii) support regional platforms that promote lesson-learning across countries and policy areas; and (iii) share lessons across diverse international policy processes, and pursue coherent approaches to the effective delivery of international finance based on common principles. The programme objectives are directly aligned with meeting these aims. They also give particular emphasis to the need for the use of climate finance to be led and owned by recipient countries (aligned with their policies and plans), and to be channelled (at least in significant part) through existing public finance management systems. This is the aid effectiveness agenda being applied to climate finance.  


Tag: Climate Change Adaptation Climate change governance Environment Policy Relevance Global Climate Fund Global Environment Facility fund Knowledge management Oversight Programme/Project Design Country Government SDG Integration SDG monitoring and reporting

2.

3 Findings

3.1 Relevance and quality of design

Regional policy and institutional context At the regional level, the ASEAN Working Group on Climate Change (AWGCC) was established in 2009 to oversee the implementation of the relevant action lines in the ASEAN Social and Cultural Community Blueprint. An Action Plan on Joint Response to Climate Change was also developed in 2012 to provide a more detailed reference in implementing the Blueprint. The design notes that the programme „will contribute to and assist in building closer cooperation with ASEAN' and „contribute to strengthening the position of ASEAN as a regional actors. The design also talks about supporting „regional capacity development alongside policy advocacy through regional institutions', and (in addition to ASEAN) makes mention of institutions such as the Lee Kuan Yew School in Singapore and the Energy and Resources Institute in India . However, the design is not particularly clear about how this regional engagement would, in practice, be achieved. The regional nature of the project, and how effective it has been in strengthening regional institutions, is discussed further later in the report. Regional relevance, at least as articulated in the design, is considered largely theoretical. 


Tag: Climate Change Adaptation Climate change governance Effectiveness Efficiency Regional Donor relations Partnership Capacity Building

3.

3 Findings

3.1 Relevance and quality of design

Question 2 : Has the original design provided a clear and robust framework to support ongoing programme implementation and monitoring (e.g. clear theory of change, clear indicators and targets, and sound governance / management arrangements)?

The original design has provided a reasonably robust framework for guiding programme implementation over the past 3.5 years. In particular, the overall implementation strategy of working with and through partner government institutions and systems, and supporting existing / emerging UNDP country office supported programmes/projects, has helped ensure a flexible and responsive approach to the provision of regional programme support. Using the CPEIRs as the entry point in each target country has been a sound and successful approach. The relative simplicity and generality of the output structure (only 3 programme outputs) and corresponding indicators and targets, has also allowed for flexibility in annual work planning, a learning by doing approach and ease of quantitative reporting. The fact that the regional programme has not had an overarching steering / coordinating committee (only annual review meetings with the primary donor SIDA) has kept governance arrangements simple and easy to manage. However, while the broad scope of the design is fairly clear, there are some notable weaknesses (at least in hindsight). These include:  


Tag: Climate Change Adaptation Climate change governance Efficiency Relevance Regional Monitoring and Evaluation Oversight Partnership Programme/Project Design Results-Based Management Theory of Change Vulnerable

4.

3 Findings

3.1 Relevance and quality of design

Question 3 : Does the design scope remain relevant to need?

In a broad sense the answer is a clear „yes?. With an increased incidence of extreme weather events occurring, and the negative impacts being felt most acutely by the poor and vulnerable, helping countries to better link their climate change and poverty policies and plans to the allocation of the necessary budgetary resources to implement them remains a very clear need, and will remain so for the foreseeable future. There is also a valuable role to be played by a regional programme of this nature with respect to information and knowledge sharing. Nevertheless, a number of lessons have been learned through the process of programme implementation that can be factored in to improving the design of future programme phases. Recommendations in this regard are provided in the final section of this report. 


Tag: Climate Change Adaptation Climate change governance Relevance Regional Knowledge management Programme/Project Design

5.

3.2 Effectiveness

3.2.1 Overview of achievements and challenges

This section of the report presents the consultant's findings under four main headings, namely the programme's outcome and three output statements. Where relevant, the corresponding indicators and targets are referenced. However, given some overlap between the outcome and output statements, and weaknesses in the formulation of some indicators (as noted in the previous section on relevance), it is not always useful to focus only on these indicators as measures of programme achievement. Further details of specific country level achievements and implementation issues are provided in Annex 5, which should be referenced to get a full picture of review findings. 

Bridging Objective / Outcome : Equitable use of climate finance widely recognized as a national policy priority within Asia and the Pacific, with specific measures put in place to channel climate resources to the poor and vulnerable

Achievements

The programme, working with and through UNDP Country Offices (COs) as well as other development partners, has effectively engaged with senior government officials in all target countries to advocate for strengthening the management of climate change finance. Awareness has been raised through the generation of evidence / information (including the initial CPEIRs, Climate Change Fiscal Frameworks, analytical tools such as CCBA and budget tagging, publications and regional workshops/conferences) and triggered changes in either policy, systems, thinking and/or practice in all countries. The programme's focus on advocating for Ministries of Finance to be more actively engaged in the climate change financing policy discourse has also borne fruit, particularly in Bangladesh, Cambodia and Indonesia (as further profiled under Output 1 findings below). The programme has also had influence beyond the four target countries, with CPEIRs reported to have been undertaken (as of August 2015) in 11 countries in the Asia-Pacific region, and 5 countries in Africa. CPEIRs are also reported to be underway in other countries in Africa and South America. Other evidence of the programme?s broader influence is reported to include: (i) the World Bank's publication of its „Climate Change Public Expenditure and Institutional Review Sourcebook in 2014, drawing significantly on the CPEIR experience; (ii) recognition by the UNFCCC that the CPEIR represents the most thoroughly documented approach to assessing and reporting on domestic climate finance flows; and (iii) OECD's recognition of CPEIRs as a useful tool in contributing to a better understanding of climate resiliency planning. 


Tag: Climate Change Adaptation Climate change governance Effectiveness Gender Equality Gender Mainstreaming Regional Communication Knowledge management Partnership Country Government Data and Statistics SDG monitoring and reporting Vulnerable

6.

3.2 Effectiveness

3.2.1 Overview of achievements and challenges 

Challenges

While the programme has clearly influenced the debate on policy and practice regarding the equitable use of climate finance, this remains early days in developing better understanding of climate finance governance issues in many developing countries. In both international and domestic climate finance debate, there is also still a stronger focus on managing „newâ international climate finance (e.g. access to and use of Green Climate Fund resources) rather than improving allocation and use of resources from domestic budgets. It is clear that the programme has initially struggled to deliver on its stated poverty and vulnerability objectives, given that it: (i) has not been specifically resourced to tackle these issues, (ii) has been working primarily at the national level on budget analysis and management issues (somewhat remote from the poor and vulnerable); (iii) did not have much in the way of other related work to draw experience from; and (iv) did not have a clearly articulated poverty / vulnerability strategy from the outset. Also, while the link between climate change vulnerability and poverty is clear enough to understand, the links to gender inequality are conceptually more complex. It is also important to note that the original design did not have a specific focus on trying to address / integrate gender equality objectives. There is also, as yet, no evidence of programme supported measures actually delivering increased funding flows to the poor and vulnerable, although the IBRCR and LOGIC projects in Bangladesh promise to do so in the near future. However, it is important to question whether delivery of increased climate finance to the poor should be an indicator of the programmeâ??s success, given the limited resources at its disposal, its broad regional coverage, and its primary focus on policy and systems change. 

At the outcome level, the programme initially had three indicators of progress / performance, namely: 1. Number of countries that have established new policies to effectively govern climate finance 2. Number of poor and vulnerable groups accessing climate finance in selected countries 3. Number of regional institutions that are prioritising South-South collaboration on managing climate finance Indicator 2 was at some point dropped and never reported on (for good reason, as this was clearly beyond the scope of the regional programme itself to directly influence or measure). The other two indicators are covered / duplicated by output level indicators, so are assessed below under output level findings. 


Tag: Climate Change Adaptation Climate change governance Challenges Effectiveness Gender Equality Gender Mainstreaming Regional Policies & Procedures Results-Based Management Vulnerable

7.

REFORM PROGRAMMES

Achievements

Indicator: 3 countries develop and reference climate change specific policy measures within medium and annual fiscal reform measures at national level

In each country, the conduct of CPEIRs has been the critical, and effective, entry point. Good progress has subsequently been made in Cambodia, Indonesia and Bangladesh with follow-up work on supporting the development of national climate fiscal / financing frameworks and other related measures. In Cambodia this has included: (i) the development of the national Climate Change Financing Framework (CCFF) which was formally approved in November 2014; (ii) supporting cost benefit analysis of sector budgets and preparation of costed sectoral action plans (within selected pilot Ministries); and (v) skills training for public sector officials from central and line Ministries (primarily on Climate Cost Benefit Analysis (CCBA)), through a partnership between UNITAR and the Economy and Finance Institute (EFI). 

In Indonesia this has included: (i) preparation of the National Mitigation Fiscal Framework (MFF) and its publication by Government of Indonesia (GoI) in May 2013; (ii) conduct of two subnational level CPEIR's during 2014/15 (in NTT and Babel provinces), designed primarily to provide analysis which would help the provincial governments improve the allocation of finance in support of their emission reduction and climate change adaptation policies and commitments; (iii) ongoing support for a review of existing performance-based budgeting (PBB) systems in the context of national mitigation, adaptation and biodiversity strategies (under the ongoing UNDP Country Office's Sustainable Development Financing (SDF) Programme). In Bangladesh this has included: (i) development of a Climate Fiscal Framework (CFF), which was published in June 2014 by the Finance Division of the Ministry of Finance (MoF); (ii) development of the Inclusive Budgeting and Financing for Climate Resilience (IBFCR) project, which is designed to support implementation of a number of the CCF policy recommendations; and (iii) supporting an Adaptive and Social Protection (ASP) study, aimed primary at helping inform policy and practice with respect to how the Government's National Social Security Security Strategy (and associated financing) might be made more effective in strengthening resilience of vulnerable groups (including women) to covariate shocks, including those being exacerbated by climate change. The ASP concept is particularly interesting in terms of bringing together work on poverty / vulnerability, disaster resilience and climate change adaptation.  


Tag: Climate Change Adaptation Climate change governance Effectiveness Gender Equality Gender Mainstreaming National Regional Communication Knowledge management Policies & Procedures Advocacy Institutional Strengthening SDG Integration SDG monitoring and reporting National Institutions Vulnerable Women and gilrs

8.

REFORM PROGRAMMES

Challenges

Indicator: 3 countries develop and reference climate change specific policy measures within medium and annual fiscal reform measures at national level

Notwithstanding these significant achievements, the following challenges are noted: While the he CPEIRs and CFFs have clearly been effective in generating useful information, prompting new ideas and follow-up action, there remain methodological limitations, primarily related to the availability of data and reliance on expert opinion. The recommendations emanating from CCFFs, as noted also by the MTR, might also benefit from clearer prioritisation to help in sequencing their implementation. The CCBA approach makes technical sense in trying to help line agencies analyse and justify their budget proposals in the context of addressing climate change impacts, however the Guidelines are a relatively complex technical product, and the ability (and incentives) to apply any such economic cost benefit analysis within many of the targeted government agencies is limited. While the programme has supported some training in CCBA application (in Thailand and Cambodia), it remains to be seen whether or not this will produce a sustainable (institutional strengthening) benefit. Supporting PBB improvements in Indonesia is ongoing. Making systemic improvements to PBB at country level will be a formidable capacity building challenge, and it raises questions about the extent to which a programme like this should itself „drill down? in to such complex areas of capacity building work.


Tag: Climate Change Adaptation Climate change governance Challenges Effectiveness National Regional Communication Knowledge management Policies & Procedures Results-Based Management Institutional Strengthening Data and Statistics SDG monitoring and reporting National Institutions Vulnerable

9.

TRACKING SYSTEMS

Achievements

Indicator: National budgetary processes in at least 3 countries are tracking and analysing climate expenditures by 2016

In Indonesia, following production of the CPEIR / MFF in 2012, UNDP and UNEP subsequently supported the conduct of a national level „Low emission budget marking and scoring system? (LESS) study, which was presented at a workshop in September 2013. The GoI has since developed and issued a Ministerial Decree (launched July 2014) which includes the „Budget Tagging for Climate Change Mitigation' system, which is mandatory for seven key line Ministries. The DG budget has developed an online application and thematic budget coding system (embedded within the budget application template) that allows Ministries to tag the budget for mitigation, adaptation and biodiversity activities. The SDF programme has since supported the conduct of training and awareness raising on the implementation of the LESS system, which became operational in 2015. The programme has also supported LESS tagging pilots in three Provinces and the generation of subsequent recommendations for further discussion with MoF and the Ministry of Home Affairs (MoHA). In Cambodia, the programme has supported establishment of a system to track international climate change financing in the Council for the Development of Cambodia?s ODA database. CDC is now able to produce Climate Change Finance reports which include a breakdown on ODA support for climate change relevant initiatives, as well as for other cross-cutting issues such as promoting gender equality. Climate finance budget tagging of the national budget is being considered. In Bangladesh, plans are in place to introduce climate finance budget tagging, which will be progressed under the recently approved IBFCR project. Based on the programme?s indicator and target for this area of work, it is therefore likely that this will be significantly achieved, with some qualification. 


Tag: Climate Change Adaptation Climate change governance Challenges Efficiency National Regional Knowledge management Operational Efficiency Partnership UN Agencies Awareness raising Institutional Strengthening Technical Support National Institutions Vulnerable

10.

CROSS MINISTERAL COORDINATION MECHANISMS

Achievements

Indicator : Clearly defined inter-ministerial mechanism with a mandate to report on the delivery of climate finance to the poor’

In all countries, key respondents noted the valuable role that UNDP can play in promoting inter and intra agency consultation and cooperation. As a trusted „middle-man', UNDP can raise issues and facilitate consultations that help break down institutional „silos? and bring agencies together to discuss and resolve issues of common concern. This is one of UNDP's core strengths. In Thailand, there is an Interagency Task Force on Climate Change which the programme has been working with on the development of the CCBA Guidelines. There is also a National Climate Change Committee chaired by the Prime Minister, which has established a Sub-Committee on Climate Change Planning and under which it is anticipated a Technical Working Group on Climate Finance will be established. The Office of National Energy Policy (ONEP) is the Secretariat to this Sub-Committee, and one of UNDP's partner implementing agencies. In Bangladesh, UNDP is supporting the Climate Fiscal Framework Committee which is chaired within the Budget Department of the Ministry of Finance, and is also engaging with MOF's Economic Relations Division in its new role as National Designated Authority for the Green Climate Fund. In Cambodia, the National Council for Sustainable Development (NCSD) was established in May 2015, which takes on, among other things, the mandate of the previous National Climate Change Committee. UNDP continues to engage with the work of the NCSD. 


Tag: Climate Change Adaptation Climate change governance Challenges Efficiency National Regional Knowledge management Oversight Partnership Programme Synergy Strategic Positioning Institutional Strengthening Technical Support

11.

Output 2 : Government budgets delivering more climate change programmes that reach the poor and vulnerable.

There is some duplication of the achievements reported under this output with those of Outputs 2 and 3. This is not in itself of any significant concern, given the necessary linkages between the work streams under each output. However, it does suggest that improvements could be made in output definition and indicator selection / specification. 

Achievements

There are three main indicators used to track progress under this output. 

1. Percentage change in the climate change related investments captured in the budget - In Thailand, the programme reports that the target (35% of the budget proposal from MOAC will have integrated Climate Change Analysis) will be achieved in 2017 for the 2018 budget proposal. - In Cambodia, the target is that 15% of climate related expenditure be reflected in the budget / ODA tracking system. The programme has only supported climate finance tracking in the ODA system, where 100% of climate related donor funding is now tracked. - In Bangladesh, the target of 100% of on-budget climate change expenditures being tracked has not yet been met due to longer than expected time-lines in getting the IBFCR project approved (which has the resources to do this work). Nevertheless, this target could yet be achieved over the next 12 months. - In Indonesia, the target of 100% of direct mitigation expenditures being tracked in the budget has been partially met, with 7 key line ministries now tagging their mitigation expenditures. 

2. Official reports analysing the relationship between climate, poverty and gender related expenditures at national or sub-national levels - In Cambodia, it is reported that an Adaptive Social Protection (ASP) Strategic Paper was developed that provides a 10 year road map for taking forward ASP programming. However, no reference was made to this during the review consultant?s mission to Cambodia. - In Bangladesh, an ASP review is being undertaken, with results expected by end of August 2016. Two government committees have been established to review the study results, which it is hoped will lead to official endorsement of the final publication. There is clearly significant interest in this study, which is (among other things) collecting and distilling the life stories of more than 150 poor women (eligible for social welfare payments) in climate stressed areas. - In Indonesia, two sub-national CPEIRs have been conducted which incorporate poverty and gender analysis, although the final reports have not yet been completed or formally endorsed. The conduct of these reviews has clearly stimulated interest and discussion within the provincial planning agencies concerned, however they probably raise more questions than answers about how to implement changes. ? In Thailand, a study of the indirect social benefits that would accrue to climate change vulnerable communities living along the Chao-Phraya river has been conducted (in terms of reduced stress and anxiety if flood diversion projects were successfully implemented) using contingent valuation methods. The collaboration between MOAC and research / academic institutions in undertaking this work appears to have been mutually beneficial as a learning exercise. 


Tag: Climate Change Adaptation Climate change governance Challenges Effectiveness Efficiency Gender Equality Gender Mainstreaming Women's Empowerment National Regional Knowledge management Project and Programme management Results-Based Management Country Government Data and Statistics SDG monitoring and reporting

12.

Output 3 : Capacity of regional institutions strengthened to provide products, services and skills that better meet the climate finance needs of Asia and the Pacific

Achievements

There are three indicators used to track achievements under this output. 

1. At least 2 global climate finance processes (namely UNFCCC and HLF on Aid Effectiveness) are informed by the preferences of country representatives, mediated through regional institutions - The programme is reported to have met this target, having provided input to: (i) the UNFCCC Standing Committee on Finance in 2014, with specific reference made to programme generated data; (ii) the Global Partnership on Effective Development Cooperation High Level Forum in Mexico, raising awareness of the importance of using country systems for managing international climate finance; (iii) further engagement with the UNFCCC in 2015, with the tabling of 2 submissions (on MRV and climate change finance reporting); and (iv) a presentation made to the 3 rd International Conference on Finance for Development in Jakarta in April 2015. - The programme has also started advocating at regional and global levels on aligning the UNFCCC's National Adaptation Plan (NAP) process with country level planning and budgeting systems, and is promoting synergies of effort between NAP and UNDP led processes through joint work planning for TA delivery.

2. At least 3 south-south meetings supported in response to demand from country policy makers and practitioners for climate finance support and at least 4 knowledge products developed - The programme is reported to have met this target having supported: (i) the Global Forum on Using Country Systems to Manage Climate Change Finance (Korea, December 2014); (ii) a Regional Technical Workshop on Climate Responsive Budgeting (Bangkok, November 2014); and (iii) a Regional Forum on Climate Change Finance and Sustainable Development (Jakarta, September 2015). ? The programme has also supported what it terms a „regional skills building initiative? in partnership with UNITAR. Training has been delivered in Cambodia (in partnership with the EFI), however the planned programme of training in Thailand has not progressed as planned. The programme has also contributed to developing the idea of establishing a Regional Peer Learning Network (RPLN), which was discussed during a Climate Finance Training Programme developed and delivered in collaboration with the International Institute for Environment and Development (IIED), International Centre for Climate Change and Development (ICCCAD) and UNITAR. The training took place in Dhaka, Bangladesh from 31 January – 04 February 2016. ? The knowledge product target has also been met, including production of: (i) CPEIR Methodological Guidelines (updated 2015); (ii) The Climate Budget Tagging Paper (July 2015); (iii) Budgeting for Climate Change – How Governments have used National Budgets to articulate a response to climate change; (iv) Climate Change Fiscal Frameworks Note – Cambodia Case Study; and (v) Incorporating Gender and Poverty Analysis in the CPEIR : A Methodological Note (October 2014). The programme has also developed a number of communication tools and materials, including policy briefs, a web-based information portal, and a video on climate finance. ? Feedback from country level respondents during the review indicates the useful role the programme has played in generating and sharing knowledge between countries and thus facilitating learning. 

3. At least 3 donor signatories to the BBB have undertaken specific measures to improve coordination with country governments (in at least 3 countries) over the management of climate finance. ? This target was reported as having been met in 2013-14, with Switzerland, Korea and Mexico having taken specific measures to better coordinate with country governments on the management of climate finance. 


Tag: Climate Change Adaptation Climate change governance Effectiveness National Regional Communication Knowledge management Programme Synergy Service delivery Capacity Building SDG monitoring and reporting National Institutions Vulnerable

13.

3.2.2 Reflections on the programme’s theory of change

Both the MTR and this Final Review note that the programme design was overly ambitious and did not realistically account for many of the political-economy challenges that would be faced in trying to advance the broad and ambitious scope of work initially envisaged. Clarification of the programme?s „theory of change? (ToC) has been suggested as a way to help address this problem. However, discussions with the programme team, as well as UNDP staff in Country Offices and the lead donor, indicate there is no clear consensus as to what constitutes a ToC. In the reviewer's opinion, the basic elements of the programme's current ToC are as follows:

1. By working in support of key aid effectiveness principles / commitments, the programme will support government ownership, alignment with national systems, mutual responsibility for results and better donor coordination. This will help promote relevance, effectiveness and sustainability of benefits, whatever the exact scope of support provided. 2. Because climate change is a development issue (not just the concern of environmental agencies), it requires the coordinated action of multiple agencies, including both central and line Ministries / Departments. 3. In order to implement their climate change policies and priorities, governments / senior officials need to better understand what they are spending on climate change related programmes and projects, as well as the institutional arrangements for managing this money. Because delivering benefits to the poor and vulnerable is an overarching programme objective, better understanding is also required as to how climate finance can more effectively targeted at meeting their needs. 4. If governments / senior officials are to better understand the climate relevance of their budgets and the institutional mechanisms required to improve climate finance delivery (also taking into account poverty and vulnerability targeting), they need the tools by which to collect and use this information (e.g. CPEIRs, CCFFs, Budget tagging, CCBA, climate vulnerability data, poverty data, gender inequality data) 5. If this information is effectively generated and then used to strengthen policy measures and systems, then plans and budgets will be more effectively targeted at meeting climate change mitigation and adaptation priorities 6. If the money is effectively spent and managed in line with improved plans and budgets, then benefits will accrue to the economy, environment and target groups. 


Tag: Climate Change Adaptation Climate change governance Effectiveness Efficiency National Regional Communication Knowledge management Programme Synergy Results-Based Management Risk Management Theory of Change Institutional Strengthening Technical Support National Institutions Vulnerable

14.

3.2.2 Reflections on the programme’s theory of change (continuation)

Institutional strengthening / capacity building. The programme's objectives (at outcome and output levels) imply a significant focus on strengthening institutional capacities. Governmental agency institutional capacities are dependent on a number of things, including : (i) leadership, political support and influence; (ii) quality of policies, plans and budgets; (iii) effectiveness of operational procedures and systems, including monitoring and accountability mechanisms; (iv) budget / financial resources available; (v) quality and number of staff; and (vi) stock of physical assets and equipment. However, the specific role of the regional programme in strengthening such elements of institutional capacity, as well as the scope of the institutional strengthening support it could realistically provide, is not clearly defined. In the reviewer's opinion, to the extent that a regional programme such as this can influence institutional capacities, it should focus primarily on (ii) above, and be extremely cautious in directly venturing in to any of the other areas. Nevertheless, given the way that the programme works with and through UNDP Country Office supported programmes, as well as with other global and donor funded initiatives, it does have the capacity to indirectly influence / support these other elements of institutional capacity. 


Tag: Climate Change Adaptation Climate change governance Effectiveness Efficiency Impact Gender Equality Gender Mainstreaming Women's Empowerment Human and Financial resources Knowledge management Results-Based Management Theory of Change Institutional Strengthening Technical Support National Institutions Vulnerable Women and gilrs

15.

3.2.2 Reflections on the programme’s theory of change (continuation)

Impact on vulnerability/gender (continuation)

- on balance, climate change adaptation expenditures are more easily linked to addressing poverty / vulnerability than mitigation expenditures. Focusing primarily on increased allocation of adaptation expenditures to benefiting the poor and vulnerable is therefore where most attention should be focused, particularly with respect to agriculture, land/water resources management, disaster risk reduction and adaptive social protection programmes; ? impacting poverty and vulnerability can clearly only happen in practice at the local level, primarily through implementation of sub-national programmes / projects. While the programme has the capacity to contribute to / influence the development of such programmes (as it has already done in Bangladesh, for example), it cannot realistically play any direct role in their implementation; and finally? given that the programme is working far away from the coal face (primarily on upstream / policy and systems issues), the extent to which it can or cannot influence actual delivery of benefits to the poor and vulnerable needs to be more carefully and realistically defined. 


Tag: Climate Change Adaptation Climate change governance Effectiveness Efficiency Impact Gender Equality Gender Mainstreaming Women's Empowerment Regional Civic Engagement Parliament Knowledge management Results-Based Management Theory of Change Country Government Institutional Strengthening SDG monitoring and reporting

16.

3.3 Programme management / efficiency

Overall, the programme has been well managed and achieved a great deal over the past 3.5 years with limited resources. SIDA?s flexibility and streamlined financing and management arrangements have been instrumental in supporting efficient programme management. Specific achievements and challenges are profiled under the sub-headings below:

Governance / management arrangements

The programme has benefited from not having any complex multi-stakeholder governance arrangements put in place. Decisions on programme strategy, plans and budgets have been made primarily through ongoing consultation with UNDP Country Offices, and an annual review process involving the programme team (namely the Programme Manager) and SIDA alone. This has proved efficient and effective. The programme also appears to have effectively avoided (or at least managed around) much of the complex bureaucracy that can be associated with UN programmes, such as the need to plan within, and report against, multiple layers of UN strategies and results frameworks. The programme has been flexible and responsive. The strategy of working with and through UNDP Country Office programmes (and with other development partners) has also been well managed, allowing the programme to have influence well beyond its own modest budget. The two donor-financed elements of the programme (SIDA and DFID) appear to have collaborated well and worked in synergy. However, there are transaction costs for the programme in having two different results frameworks, timelines and reporting requirements to meet. Ideally, the two programmes would be merged under one set of programme planning and reporting arrangements. A key challenge for the programme moving forward (and for UNDP in general), is UNDP's financial situation. With continuing contraction in core funding, UNDP is increasingly dependent on specific short-term project financing from individual donors. This impacts negatively on UNDP?s ability to be programmatic in its approach, and means that it must continually seek cost recovery opportunities. UNDP is also increasingly in competition with other UN agencies for a limited pool of donor money, and indeed different parts of UNDP are increasingly in competition with each other. While there may be some benefits to this (e.g. increased pressure to demonstrate results and value for money), there are also significant transaction costs in chasing money, and UNDP is not currently well set up to compete in this way. 


Tag: Climate Change Adaptation Climate change governance Efficiency Donor relations Human and Financial resources Policies & Procedures Project and Programme management Results-Based Management

17.

3.3 Programme management / efficiency 

Finance and personnel management (continuation)

With respect to personnel management, the programme team appears to be technically competent and well respected by their peers. The team has a good mix of technical, process management and crosscultural communication skills. They work well together as a team, are motivated to do a good job and achieve results. Core team member positions have been filled in good time, and there has been no problem with untimely staff turnover. Nevertheless, given the over ambitious nature of the original design, the team has been stretched to try and deal with all the demands on their time. As previously noted, some areas of work have consequently been under-resourced, such as with respect to progressing the poverty and gender agenda and implementing the communications (including the K2P) strategy. The issue of basing some of the programme?s international TA outside Bangkok (e.g. within other „target? countries) was raised during the review. The DFID financed element of the programme already does this. However, in the reviewer?s opinion, there is a risk that this could compromise regional team work / ongoing communications as well as „capture? of the posted TA by the specific country in which they are located. With the current suite of countries being supported (all within a relatively short and convenient flight from Bangkok), there does not seem to be a compelling reason to „break up? the regional team. Nevertheless, if future plans include any targeted support in the Pacific, then a dedicated TA to cover that work would certainly be best based in that region (e.g. Fiji). 

Work planning, monitoring and reporting

ork planning has been undertaken on either an annual or 18-monthly basis. Work plans are prepared at both the regional and country levels. In most cases the country level work plans have not been fully implemented as planned, as also reflected in under expenditure against budget. This might be attributed to inadequate appreciation of the local political economy in each country, as well as a natural tendency to be somewhat over-optimistic when formulating plans of this nature. Nevertheless, in reviewing some of the country-level work plans (particularly for the current planning period), they do appear at times to have been compiled in some haste, could contain a bit more detail, and may have benefited from greater quality assurance. It also appears that the work plans are not necessarily being reviewed and updated in a regular and consistent way. Programme monitoring is primarily undertaken in an informal way, through regular personal communication within the regional team and between the regional team and country-level partners, particularly UNDP CO staff. There appear to be few structured monitoring tools being used by the regional team, apart from making some reference back to the annual work plans / budgets / indicators and targets, and the use of feedback forms at workshops / training events. As noted earlier in this report, there are some weaknesses in the formulation of programme indicators, namely that they don?t necessarily measure the real intent of the objective statement as written, and focus only on quantitative targets with no clear measures of quality. 


Tag: Climate Change Adaptation Climate change governance Donor relations Human and Financial resources Monitoring and Evaluation Policies & Procedures Project and Programme management Results-Based Management Theory of Change Data and Statistics

18.

3.3 Programme management / efficiency 

Publications and communications

The programme has produced, and contributed to, a number of important publications which appear to have been generally well-received. Key publications / knowledge products have already been listed in this report in Section 3.2.2 under Output 3. As noted in the MTR, the quality of some of these publications could nevertheless have been improved through more rigorous quality assurance processes, such as through peer review and final editing by a professional editor or communications specialist. While the programme has indeed started using a process of peer review for key publications, there is considered to be scope for some ongoing improvement. Engaging more peer reviewers from outside the UN system might be useful in this regard. The programme has established and maintains a useful information portal on the web, where all key documents are made available. A CPEIR data base is also available for interrogation, although the reviewer found this a little cumbersome to use. By comparison the World Bank's Climate Change Knowledge Portal is easier to navigate and interrogate, even though it does not include the CPEIR specific data. 

The programme developed a Communications Strategy document in 2013. On paper this looks basically sound (from a non-specialist?s perspective), however only some elements have been implemented in practice. The main constraint is simply one of resources. The programme staff member specifically tasked with overseeing and directing the programme?s communications strategy has many other duties, and estimates she has only been able to spend about 10% of her time specifically on communications issues. While she has had some additional support for the past six months or so from the programme?s NAP/Policy Specialist, if the programme's communication ambitions are to be met going forward (a new Communications Strategy has recently been developed), more dedicated specialist resources will be required. In the reviewer's opinion, the key area for greater focus is in distilling and delivering key messages (based on credible evidence), that support the programme?s K2P objective. 


Tag: Efficiency Resource mobilization Communication Donor relations Knowledge management

19.

3.4 Sustainability

Given that the programme is ongoing, it is only possible to comment on the likely prospects for sustainability should the regional programme cease to operate.

At the country level, many of the regional programme supported initiatives would likely continue in some form, as long as the UNDP CO's have the resources to continue to provide related support. Some initiatives have also now been (at least partly) institutionalised within government / local systems and would likely continue (e.g. climate change budget tagging in Indonesia and for the ODA database in Cambodia; ongoing reference to / use of CPEIR and CCFF data and recommendations in Cambodia, Indonesia and Bangladesh; and, possibly, the use of CCBA ideas / analysis in Thailand and Cambodia).


Tag: Climate Change Adaptation Climate change governance Sustainability Government Cost-sharing National Regional Communication Knowledge management Institutional Strengthening National Institutions

Recommendations
1

5 Recommendations for forward programming

The Final Review Terms of Reference require the consultant to help „make recommendations regarding specific actions or new approaches that might be useful and „identify potential areas to be continued and strengthened or to be excluded in the future programme. In this light, the following recommendations are provided with respect to the proposed next phase of support: 

Recommendation 1: Continue to pursue the core strategy of supporting aid effectiveness principles and synergies of effort with UNDP COs.

This includes:

  • Continuing to work in support of national planning and budget systems so that they can more effectively identify and track climate relevant finance (primarily from the domestic budget)
  • Working with and through existing or new UNDP CO programmes
  • Being flexible and responsive to local needs and priorities, including those of UNDP COs; and 

The programme should also be in a position to provide some core financial support to each CO, e.g. for a dedicated locally-engaged staff member to help interface between the regional programme and the CO.

2

Recommendation 2:  Re-design the programme logic and results framework, including key elements of the overall theory of change, so that objectives are more realistic and the limited resources available to the programme are more effectively targeted.

This includes:

  • Re-defining programme impact, outcome, and output statements, which are more logically linked in a hierarchy of means to an end.  Output statements should be more clearly within the influence of the programme to deliver. 
  • Improving outcome and output indicators to more directly measure achievement of outputs and contribution to outcome, including an assessment of process and product quality.
  • Discontinuing the objective of trying to directly strengthen the capacity of other regional institutions, while continuing to share knowledge and undertake collaborative actions with institutions in the region. 
  • Continuing to strengthening UNDP BRH’s own role as a regional platform for knowledge sharing; and 

More clearly defining (and limiting) the country level ‘capacity development’ role that the programme is expected to directly play and support. 

3

Recommendation 3 : Continue to give clear focus to helping countries better target domestic climate change finance to benefit the poor and vulnerable, but more clearly define what the programme can realistically influence in this regard, including with respect to promoting gender equality.

This includes:

  • Not trying to integrate poverty and gender analysis directly in to all climate financing analytical tools, but rather using other complementary data sets /information to overlay on to the financing analysis / information.
  • Focusing particularly on improved analysis of CC adaptation budgets / expenditures (rather than mitigation), primarily for agriculture, land / water resource management, disaster risk reduction and adaptive social protection programmes. 
  • Continuing (albeit cautiously) to support ongoing country-level assessment / analysis of the role that Adaptive Social Protection approaches and programmes can play in helping deliver climate finance benefits to the poor and vulnerable, at least in Bangladesh and Cambodia. 

Seeking additional dedicated programme resource for analysing and integrating poverty / vulnerability and gender dimensions into climate finance work, and helping produce relevant knowledge products

4

Recommendation 4 : Prioritise the programme’s role in generation of empirical evidence / data and knowledge products to influence the allocation of budgetary resources to meet climate change policy priorities.

This includes:

  • Building on and continuing to support the review, updating and improvement of CPEIRs and/or CFFs, given the demonstrated interest in the work undertaken so far, ongoing scope for improvements, and the useful role this can play in helping analyse changes over time.
  • Pursuing opportunities to implement, test and refine the Climate Change Integration Index methodology already developed.  
  • Giving greater focus to the programme’s role in supporting ‘knowledge to policy’ processes.  This would require, among other things, enhanced attention to and resourcing of, key elements of the programme’s (updated) Communications Strategy. 
  • More clearly limiting programme activities in other areas, such as for national capacity development and providing any direct support for implementation of national level projects, given overall programme budget limitations. 
5

Recommendation 5 : Should the programme seek to expand in to the Pacific, ensure that this is appropriately resourced within the region.

This includes:

  • Funding a full time international TA position, probably best based in Suva Fiji, ideally within an established regional agency such as the Pacific Islands Forum Secretariat (assuming the programme would wish to provide support to more than just one Pacific Island country); and
  • Ensuring adequate administrative and technical support was also provided for working in this geographically wide / dispersed region
6

Recommendation 6 : Continue to actively seek the merging all donor financing for the regional programme into one mutually agreed / unified programme.

This includes:

•          Seeking to establish a unified programme results framework, time line, budget and reporting requirements that is jointly financed by all interested donors.

1. Recommendation:

5 Recommendations for forward programming

The Final Review Terms of Reference require the consultant to help „make recommendations regarding specific actions or new approaches that might be useful and „identify potential areas to be continued and strengthened or to be excluded in the future programme. In this light, the following recommendations are provided with respect to the proposed next phase of support: 

Recommendation 1: Continue to pursue the core strategy of supporting aid effectiveness principles and synergies of effort with UNDP COs.

This includes:

  • Continuing to work in support of national planning and budget systems so that they can more effectively identify and track climate relevant finance (primarily from the domestic budget)
  • Working with and through existing or new UNDP CO programmes
  • Being flexible and responsive to local needs and priorities, including those of UNDP COs; and 

The programme should also be in a position to provide some core financial support to each CO, e.g. for a dedicated locally-engaged staff member to help interface between the regional programme and the CO.

Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

UNDP Response in the new Proposal Design

  • Country-level theories of change were developed with intensive consultation with UNDP Country Offices in order to:
    1. Identify existing and new national programmes to work with and
    2. Be responsive to local needs and priorities
  • Programme design continues with existing modalities of providing financial support to each CO (for a dedicated locally engaged staff) to help implement the programme activities at the country level and provide the interface with the regional programme.

Programme design (Outputs 1 & 2 activities) continue to support the implementation of reforms to national planning and budget systems, including tracking gender responsive climate change public expenditures.

Key Actions:

2. Recommendation:

Recommendation 2:  Re-design the programme logic and results framework, including key elements of the overall theory of change, so that objectives are more realistic and the limited resources available to the programme are more effectively targeted.

This includes:

  • Re-defining programme impact, outcome, and output statements, which are more logically linked in a hierarchy of means to an end.  Output statements should be more clearly within the influence of the programme to deliver. 
  • Improving outcome and output indicators to more directly measure achievement of outputs and contribution to outcome, including an assessment of process and product quality.
  • Discontinuing the objective of trying to directly strengthen the capacity of other regional institutions, while continuing to share knowledge and undertake collaborative actions with institutions in the region. 
  • Continuing to strengthening UNDP BRH’s own role as a regional platform for knowledge sharing; and 

More clearly defining (and limiting) the country level ‘capacity development’ role that the programme is expected to directly play and support. 

Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

  • Overall theory of change and results framework are strengthened, particularly based on experience from Phase 1 and the Dfid-funded part of UNDP’s regional programme on governance of climate finance.
  • The proposal will include a regional theory of change as well as theory of change developed for each country of the programme.
  • The Programme design explicitly allocate additional resources (compared to Phase 1) to deliver the gender equality objective of the programme.
  • The programme design positions UNDP BRH as the regional platform to facilitate South South knowledge exchange and regional cooperation.

The programme maintains the activities with regional institutions, however shifting more focus to the Pacific where a regional approach is merited.

Key Actions:

3. Recommendation:

Recommendation 3 : Continue to give clear focus to helping countries better target domestic climate change finance to benefit the poor and vulnerable, but more clearly define what the programme can realistically influence in this regard, including with respect to promoting gender equality.

This includes:

  • Not trying to integrate poverty and gender analysis directly in to all climate financing analytical tools, but rather using other complementary data sets /information to overlay on to the financing analysis / information.
  • Focusing particularly on improved analysis of CC adaptation budgets / expenditures (rather than mitigation), primarily for agriculture, land / water resource management, disaster risk reduction and adaptive social protection programmes. 
  • Continuing (albeit cautiously) to support ongoing country-level assessment / analysis of the role that Adaptive Social Protection approaches and programmes can play in helping deliver climate finance benefits to the poor and vulnerable, at least in Bangladesh and Cambodia. 

Seeking additional dedicated programme resource for analysing and integrating poverty / vulnerability and gender dimensions into climate finance work, and helping produce relevant knowledge products

Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

  • The programme has taken a sectoral approach to budget reform implementation, focusing on supporting ministry of agriculture with view to scale up to other climate related sectors.
  • The programme toolkits are now integrated with gender dimensions (see Annex V for TORs of Toolkits).

The programme continues the work on Adaptive Social Protection under Output 2. 

Key Actions:

4. Recommendation:

Recommendation 4 : Prioritise the programme’s role in generation of empirical evidence / data and knowledge products to influence the allocation of budgetary resources to meet climate change policy priorities.

This includes:

  • Building on and continuing to support the review, updating and improvement of CPEIRs and/or CFFs, given the demonstrated interest in the work undertaken so far, ongoing scope for improvements, and the useful role this can play in helping analyse changes over time.
  • Pursuing opportunities to implement, test and refine the Climate Change Integration Index methodology already developed.  
  • Giving greater focus to the programme’s role in supporting ‘knowledge to policy’ processes.  This would require, among other things, enhanced attention to and resourcing of, key elements of the programme’s (updated) Communications Strategy. 
  • More clearly limiting programme activities in other areas, such as for national capacity development and providing any direct support for implementation of national level projects, given overall programme budget limitations. 
Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

  • The programme continues to pursue and strengthened CPEIR, CCFF, CCBII with expanded implementation and integrated gender dimensions.
  • The programme also includes resources to implement the communications strategy, in particular to support the ‘knowledge to policy’ processes.

The programme still pursue capacity building at the national level as part of implementation of budget reforms. Without capacity building, the programme risks sustainability and ownership by the governments. 

Key Actions:

5. Recommendation:

Recommendation 5 : Should the programme seek to expand in to the Pacific, ensure that this is appropriately resourced within the region.

This includes:

  • Funding a full time international TA position, probably best based in Suva Fiji, ideally within an established regional agency such as the Pacific Islands Forum Secretariat (assuming the programme would wish to provide support to more than just one Pacific Island country); and
  • Ensuring adequate administrative and technical support was also provided for working in this geographically wide / dispersed region
Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

The programme allocates additional resources to the Pacific recognising the importance of the region but also the demand given its geographical characteristics.

Key Actions:

6. Recommendation:

Recommendation 6 : Continue to actively seek the merging all donor financing for the regional programme into one mutually agreed / unified programme.

This includes:

•          Seeking to establish a unified programme results framework, time line, budget and reporting requirements that is jointly financed by all interested donors.

Management Response: [Added: 2017/04/16] [Last Updated: 2021/02/28]

The programme manager continues to request Dfid to consider increasing joint-programming with this programme where possible (beyond Output 3).

Key Actions:

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org