Ten Island Challenge - Terminal Evaluation

Report Cover Image
Evaluation Plan:
2017-2021, Barbados
Evaluation Type:
Final Project
Planned End Date:
08/2019
Completion Date:
12/2019
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
20,000

Evaluation Ratings:

 

 

 

1. Monitoring and Evaluation

Rating

2. IA & EA Execution

Rating

M&E design at entry

MU

Quality of UNDP Implementation - Implementing Agency

S

M&E Plan Implementation

MS

Quality of Execution - Executing Agency 

S

Overall quality of M&E

MU

Overall quality of Implementation / Execution

S

3. Assessment of Outcomes 

Rating

4. Sustainability

Rating

Relevance 

R

Financial resources

ML

Effectiveness

MS

Socio-political

ML

Efficiency 

S

Institutional framework and governance

ML

Overall Project Outcome Rating

MS

Environmental 

ML

 

 

Overall likelihood of sustainability

ML

Share

Title Ten Island Challenge - Terminal Evaluation
Atlas Project Number: 00089334
Evaluation Plan: 2017-2021, Barbados
Evaluation Type: Final Project
Status: Completed
Completion Date: 12/2019
Planned End Date: 08/2019
Management Response: Yes
UNDP Signature Solution:
  • 1. Energy
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.5.1 Solutions adopted to achieve universal access to clean, affordable and sustainable energy
SDG Goal
  • Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all
SDG Target
  • 7.1 By 2030, ensure universal access to affordable, reliable and modern energy services
  • 7.2 By 2030, increase substantially the share of renewable energy in the global energy mix
  • 7.b By 2030, expand infrastructure and upgrade technology for supplying modern and sustainable energy services for all in developing countries, in particular least developed countries, small island developing States and landlocked developing countries, in accordance with their respective programmes of support
Evaluation Budget(US $): 20,000
Source of Funding: Project
Evaluation Expenditure(US $): 27,199
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Maria Onestini
GEF Evaluation: Yes
GEF Project Title: The Ten Island Challenge: De-risking the Transition of the Caribbean from Fossil Fuels to Renewables
Evaluation Type: Terminal Evaluation
Focal Area: Climate Change
Project Type: MSP
GEF Phase: GEF-6
GEF Project ID: 9112
PIMS Number: 5526
Key Stakeholders:
Countries: BARBADOS
Lessons
1.

There are a series of learned lessons that can be assimilated in the future for enhanced project planning and implementation.  These lessons are listed below, and they are linked to proactive recommendations in the next section of this report.

  • Design robustness, or lack of, has an indelible impact on implementation and monitoring.
  • If indicators are not robust, applicable and achievable, then implementation and monitoring could be negatively affected.
  • Throughout implementation, opportunities for adjustments are essential (adjustments to indicators for adaptive management, etc.) in order to properly reflect changes in exogenous conditions or to adjust for over ambitiousness.
  • When innovative implementing modalities are applied, these should be accompanied by proper architecture and guidelines on how this implementation would take place from design onward, how reporting is to take place (to whom, when, etc.) and how the decision-making process and assurance are carried out.
  • Gender equality promotion does not occur unless a specific gender approach that addresses fully the different needs of men or women is developed early on in design and in the implementation process.
  • The usefulness of an exit and sustainability strategy is related not only to content but also to when it is drawn and carried out. In order for an exit strategy to be appropriated by the relevant parties which will implement all or some of the strategy’s options, an exit/sustainability strategy needs to be drawn and carried out at a proper time and not at the very end of a project.

Findings
1.

3.1     Project Design/Formulation

Analysis of LFA/Results Framework (Project logic /strategy; Indicators)

As all projects of this sort, a key aspect of its design is the inception log frame/results framework which includes project strategy and the intervention’s logic as well as baseline and target indicators.  The TIC initiative’s logic and strategy at the design and formulation level was fitting.  The formulation documents effectively identify the key issues, threats, and other matters that hinder adequate transition to renewable energy in the Caribbean. The results framework, therefore, bases its logic and strategy upon identified threats and barriers.


Tag: Energy Programme/Project Design

2.

Threats, Barriers, and underlying Causes sustaining project logic/strategy

The threats as well as underlying causes that hinder an adequate transition to renewable energy in the Caribbean were properly identified at design. The Project Document identifies three types of barriers (1) regulatory policy/legal  (such as limited capacity to generate and enforce nation-wide clean energy policies and regulations); (2) institutional/technical (for instance, lack of coordination and expertise for the adoption of nation-wide clean energy technologies; as well as (3) market/financial (such as operational risk perception affecting credit, market and other nation-wide clean energy financing).

 The Project’s logic and strategy therefore was to confront these issues through specific outputs and expected outcomes that would, plausibly, deal with identified threats and barriers. Therefore, in terms of overall logic and strategy the design responded to an adequate rationale and it was designed as a strategic intervention.


Tag: Energy Challenges Policies & Procedures

3.

LOG FRAME and indicators

Although the overall logic and strategy is adequate, design suitability weakens -however- when analysing the log frame, target indicators, as well as context in general.  First of all, the log frame (a full copy of which is found in Annexes — Annex  5:  Project Results Framework) goes far beyond the GEF-funded UNDP-Implemented project.  For instance, its achievement indicators encompass far more than the five countries where the GEF-funded UNDP-Implemented project is to take place and focus on (i.e. Bahamas, Belize, Grenada, Saint Lucia, as well as Saint Vincent and the Grenadines). Therefore, many of the targets are not specific to the project itself, overambitious, and are consequently not achievable and unrealistic. This, in turn, created misunderstandings vis-à-vis reporting.

While IP attempted a reformulation of the log frame target indicators, this was not accepted by UNDP-GEF since they were connected to the Project’s main objective. It was recommended, as an alternative, that the indicators be maintained and, instead, reporting should be disaggregated to identify the incremental impact of GEF resources. Therefore, the Project had to reconcile the misconstruing of indicators at design with differential reporting of indicators. Although, it was indicated that there should be reporting of desegregated data for targets this arrangement did not fully incorporate accurate indicator reporting given the complexity inherent in the intervention.  Furthermore, due to the fact that the intervention was much broader than the Project, a number of funders were involved creating a very intricate architecture for the initiative. Therefore, reporting achieved indicators as a direct result of the project proved to be complex and problematic at times.

 


Tag: Challenges Programme/Project Design

4.

Design formulation

Design formulation was not as robust as desirable for a project of this type. Key stakeholders have indicated that the inception/design phase was not robust, which is an analysis that this evaluation shares.  Issues such as the relevancy of indicators as seen above, and others which will be expanded upon further in this report, attest to this issue. 

Although a potential stakeholder list was drawn up, the design does not thoroughly include an analysis and mapping of engagement. Albeit a list of groups and stakeholders were identified at design, their real capacities, characteristics, commitments to the Project, abilities or willingness to participate in the project, the relation and/or conflict between different sorts of stakeholders, as well other such issues were not analysed, nor included as thoroughly or as deliberately as needed in the stakeholder analysis. 


Tag: Programme/Project Design

5.

Being that climate change and renewable energies are such significant topics in the target countries, as expected there are a myriad of development activities dealing with renewable energies in the five target countries. Some were acknowledged at design. For instance, there are specific remarks at design indicating that the 10 Island Challenge project will complement or build upon existing activities across the region, such as the GEF-funded UNDP-implemented PACES project in St. Vincent and the Grenadines, as well as the regional renewable energy and energy efficiency activities under the Caribbean Energy Efficiency Lighting Project and Japan-Caribbean Climate Change Partnership. However, there is no full mapping of initiatives, nor a partnership framework drawn to integrate parallel initiatives and identify complementarities or conflicts between and among these and other similar development projects. 

The design documents explicitly indicate that lessons learned have informed the planning process.  For instance, it is indicated that “Ongoing work builds on the lessons learned and progress made in the regional space by identifying upscaling and complementary opportunities with projects such as the Promoting Access to Clean Energy Systems (PACES) and the Disaster Risk and Energy Access Management (DREAM)”. 

Moreover, project design goes further into indicating potential collaborating arrangements with other actors in the region (besides CWR/RMI and its own partners). For instance, it is indicated that the TIC project would liaise with GIZ, through BMUB (Germany), in Grenada, given that UNDP is already linked to a Programme on Integrated Climate Change Adaptation Strategies (ICCAS) in that country.  Also, in Grenada, government was — at the time of design — working with UNDP’s SRO for its National Appropriate Mitigation Action plans for support to “convert government buildings to solar”, potentially indicating a probable collaboration between projects. 

Furthermore, UNDP supported and supports energy access as well as climate change mitigation and adaptation work at the regional level and at the country level in the Caribbean.   These projects and processes set the base for potential integration and to avoid duplication with activities supported by other agencies within the UN system as well as with other donors.  Although the potential for collaboration is indicated, what specific lessons learned from these interventions have been incorporated is not explicitly identified nor are they integrated into all aspects of the Project Document.


Tag: Energy Partnership Programme/Project Design

6.

Regarding planning process and the resulting design, several key stakeholders pointed out a series of issues.  First of all, it was deemed that the design did not accurately consider political and policy issues in several of the target countries.  This in turn had impacts on delaying in the achievement of outputs (for instance in Bahamas) or in failing to carry-out any national activities at all at the national level (for instance in Grenada).

Also, stakeholders indicated that the design concentrated upon environmental issues deviating from an integrated analysis of development issues that are closely related not only to climate change mitigation but also to development per se, energy access, etc., in the target countries.


Tag: Challenges Programme/Project Design Theory of Change

7.

Assumptions And Risks

At the design stage, a series of assumptions were identified in the Project Log Frame.

In retrospect, the assumptions were correct in most cases for most countries (as will be seen in the sections strictly dealing with implementation).  Therefore, in the cases where the assumptions unfolded correctly, this impacted positively on the overall achievement of targets.

The design also included a risk analysis.  The analysis (see table below) included not only identifying the risk per se, it also included an assessment as to its likelihood of occurring and possible mitigation actions.

 


Tag: Natural Disaster Challenges Programme/Project Design

8.

Lessons from other relevant projects (same focal area) incorporated into project design

Project design indicates that lessons from other relevant projects in the same focal area would be applied (for instance “applying best practices and lessons learned underpins the focus of this execution strategy, with the intention of providing replicable models for other islands and isolated economies”).  Nevertheless, these are not specified at design level.

Also, it is indicated that “The 10 Island Challenge will complement or build on existing activities across the region, including the GEF-funded PACES project in St. Vincent and the Grenadines, as well as the regional RE and EE activities under the Caribbean Energy Efficiency Lighting Project and Japan-Caribbean Climate Change Partnership.” Overall, therefore, the Project had a series of experiences and projects to draw upon for its design and implementation and the aim to build upon lessons is expressed, albeit it is not made explicit in the project design which or what lessons would be incorporated in the Project.


Tag: Programme/Project Design Sustainability

9.

Planned stakeholder participation

At the design stage a general list of potential stakeholders was drawn. The design document indicates that stakeholders (agglutinated in what the design regards as communities) would be involved in the different stages of project implementation and application. It is indicated that, once project starts, a community stakeholder mapping and engagement exercise would be developed in order to identify, map and engage with relevant audiences.


Tag: Civil Societies and NGOs

10.

Replication approach

At the design level, the replication approach has been very much a continual and solid aim of the project, both tacitly and explicitly.  Explicitly even the objective of the Project itself establishes, from inception, that a replication approach should and would be interweaved in project implementation and as a demonstration of achievements (“…Ten Island Challenge (TIC) is to accelerate the transition of Caribbean island economies from heavy dependence on fossil fuels to a diverse platform of renewables and energy efficiency, thereby establishing a blueprint for other isolated economies.”). Although no explicit replication strategy is presented at design, there is a stated aim to upscale, replicate or expand outcomes and outputs, both during project implementation as well as in follow-up.


Tag: Programme/Project Design

11.

UNDP comparative advantage

In particular as it relates to GEF – funded projects, UNDP’s comparative advantage is associated to the agency’s global network of country offices, its experience in integrated policy development, human resources development, institutional strengthening, and non-governmental and community participation.  UNDP’s comparative advantage as a GEF implementing agency also relates to the subject of the Ten Island Challenge Project.  For instance, UNDP’s focus on access to sustainable energy services as a priority area brings together energy access and climate change concerns.  Priorities within this area include, explicitly and as related to TIC’s overt and implicit aims, strengthening national policy frameworks for poverty reduction and sustainable development; promoting clean energy technology (that is, low GHG emitting, renewable energy and energy efficient technologies) for sustainable development; as well as increased access to financing for sustainable energy.[1] However, design did not specifically indicate what UNDP’s comparative advantage would be in the context of the Project.

 


Tag: Policy Advisory Clean Energy Strategic Positioning Institutional Strengthening

12.

Management arrangements

The management arrangements for this project were fairly unique. The project was executed under UNDP’s NGO implementation modality and implemented by Rocky Mountain Institute (RMI)/Carbon War Room (CWR), in partnership with the U.S. Department of Energy (US DOE).  Therefore, this was not a typical DIM nor NIM implemented project since it was executed through a non-government entity.  Furthermore, the organization – although it had links to the region — was based outside the Caribbean.


Tag: Implementation Modality Oversight Partnership Programme/Project Design Civil Societies and NGOs

13.

Adaptive management (changes to the project design and project outputs during implementation)

Adaptive management is defined as the project’s ability to adapt to changes to the project design (project objective, outcomes, or outputs) during implementation resulting from: (a) original objectives that were not sufficiently articulated; (b) exogenous conditions that changed, due to which change was needed; (c) the project’s restructuring because the original expectations were overambitious; or (d) the project’s restructuring because of a lack of progress.  

If this definition is followed for the TIC project, then it can be said that adaptive management was not carried out in full force. This in particular when it was understood that the expected output and outcome indicators were overly ambitious and unachievable in the context of the intervention.  That is, there were no changes in log frame, reforming the expected output indicators to induce more achievable results.


Tag: Programme/Project Design Project and Programme management

14.

Partnership arrangements (with relevant stakeholders involved in the country/region)

As established in the Project Document and at inception, a broad framework for stakeholder analysis was carried out at Project design.  The main partnership arrangements with relevant stakeholders to be involved was established.  The implementation of project activities engaged with several key actors, fairly following the planned framework for stakeholder analysis.


Tag: Partnership National Institutions Regional Institutions

15.

Feedback from monitoring and evaluation activities used for adaptive management

Imbedded in design there was a Monitoring and Evaluation (M&E) plan, this included a series of standard activities. Although the M&E plan did include a final evaluation (i.e. the present assessment) it did not include a mid-term review.  This has proven to be a defect in design given that there was no clear mid-point opportunity to provide feedback on activities and design (including metrics) nor to enable conditions for adaptive management. Audits were carried out in a compliant manner, yet these concentrated upon financial issues and not implementation and other evaluation type of exercise.
 


Tag: Monitoring and Evaluation Project and Programme management Quality Assurance

16.

Project Finance

The Project’s total planned cost was of USD 306,376,484.  Planned GEF financing was to be USD 1,826,484 and UNDP proposed co-financing was USD 200,000. The rest of the funding was to be provided by the following sources:  Overseas Private Investment Corporation (OPIC) USD 300,000,000; Carbon War Room (CWR) USD 3,000,000; Rocky Mountain Institute (RMI) USD 1,350,000.Actual versus planned financial data for financing and co-financing is provided below in the narrative and in the following table. 


Tag: Human and Financial resources Private Sector Financing Global Environment Facility fund

17.

Monitoring and evaluation: design at entry and implementation

As stated in the section dealing with adaptive management above, a series of standard activities (such as Inception Workshop and Report, Measurement of Means of Verification of project results, Measurement of Means of Verification for Project Progress on output and implementation, PIRs, progress reports, Audits, etc.) are imbedded in design and are a standard for this sort of project. Although the M&E plan did include a final evaluation (i.e. the present assessment) it did not include a mid-term review at the guidance of UNDP given that it was not mandated.  This has proven to be a defect in design given that there was no clear mid-point opportunity to provide feedback on activities and design (including indicators) nor to enable conditions for adaptive management. Design for monitoring and evaluation lacks depth proportionate to the complexity of the Project.  Therefore, although not mandated, a mid-term review could have been inserted as a requisite given the regional nature of the project and other factors which added complexity to implementation.   The plan for monitoring and evaluation, therefore, was not well-formulated, and therefore could not serve as an effective tool as can be expected to support project implementation. Therefore, at entry, the ranking is Moderately Unsatisfactory (MU), given that there were significant shortcomings in the setup of the M&E plan.


Tag: Monitoring and Evaluation Programme/Project Design Project and Programme management

18.

Implementing Partner and UNDP implementation/execution coordination, and operational issues

The Project Document sets up coordination and operational issues as well as proposed management arrangements. Although the management arrangements and oversight structure were fairly standard, they were not overly specified at design.  This was an issue that posed some definition problems given that the implementation modality (NGO) with a non-governmental organization outside of the operation region in a multi-country project is highly unusual for a GEF-financed UNDP-implemented intervention.  With an imprecise coordination and operational framework in an innovative modality it was rather difficult to coordinate execution to the levels needed.


Tag: Oversight Programme/Project Design Project and Programme management Coordination Civil Societies and NGOs Implementation Modality

19.

Implementing Partner and UNDP implementation/execution coordination, and operational issues

Project team reports that UNDP provided support in reporting and financial procedures throughout the implementation process. Yet there were a series of issues and communication problems between the three main actors at the management level (that is, between the IP –CWR/RMI, the funding agency – GEF, and the GEF agency — UNDP’s Barbados and the OECS office).  The main issue has been the lack of fluid communication and agile responses in reforming the log frame through indicators modifications and therefore weakness in enabling adaptive management. The understanding that these adaptations might take place in order to side-step the Project’s over ambitiousness arose late in the implementation process, and there were miscommunications, in particular with UNDP/GEF, as to the operational margin that the project had in this matter.
 


Tag: Challenges Implementation Modality Partnership Project and Programme management

20.

3.3     Project Results

Overall results

In terms of expected results, the overall objective of the TIC Project was to accelerate the transition of Caribbean economies from heavy dependence on fossil fuels to a diverse platform of renewables and energy efficiency as well as to establish a blueprint in this matter for other Small Island Developing States (SIDS). To a greater or lesser extent this has been achieved for some of the target countries involved in the initiative. Countries performed at different levels depending on the characteristic conditions that each one faces regarding transitions to renewable energy, due to internal policy, as well as due to the level of insertion that renewable energy has in the overall national aims and prospects.  

The expected results were articulated through anticipated outcomes and these, in turn, were operationalized through the generation of outputs (products, activities, processes, etc.).  The project’s final PIR of June 2019 and the Project Terminal Report as well as other similar sources, attest to the fact that products/outputs have been achieved to a large degree in several of the beneficiary countries.


Tag: Energy Effectiveness

21.

Results at the Objective Level: To accelerate the transition of Caribbean island economies from heavy dependence on fossil fuels to a diverse platform of RE/EE

  • At completion of the initiative 15 countries signed on to the Ten Island Challenge including the four countries part of the GEF-funded UNDP-Implemented project (Bahamas, Belize, Saint Lucia, Saint Vincent and the Grenadines).
  • CO2 emission reductions/year 9.7 ktCO2; percentage of share of RE in the power generation mix of TIC countries was achieved at 22%, mainly due to the installation of solar farms in Saint Lucia as well as in Saint Vincent and the Grenadines. However, here as in other segments of this intervention, it should be clear that emission reduction was not a strict direct result of the project (since, as stakeholders have clearly indicated, the investments and implementation was attributable to a great degree to other resources and mechanisms that are outside the purview of the project). Nevertheless, it can be clearly stated that in those two countries the GEF/UNDP intervention aided in obtaining these emissions reductions and in increasing the respective countries share of renewable energy in their energy matrices.

Defined as:  Accelerate the transition of Caribbean island economies from fossil fuels to renewable energy resources like wind, solar and geothermal, and energy efficiency measures. Help the participating islands to reduce their heavy dependence on fossil fuels while decreasing the cost of electricity for consumers through optimising the mix of conventional generation, renewable energy, and battery storage in a safe and reliable manner.

 


Tag: Emission Reduction Energy Effectiveness

22.

Following are highlights of the achievements as indicated in these reports as per expected outcomes and a general analysis of results concerning the role of the project.  Based on the chart above, therefore, the following is a juxtaposition of targets and results that critically analyses attainments at the results level.

Results at Outcome 1 Level:   Island-wide de-risked enabling environment for low GHG development through innovative policy tools

Number of RE/EE strategies and assessments with specific targets/number of countries where implementation of comprehensive measures (plans, strategies, policies, programmes and budgets) to achieve low-emission and climate-resilient development objectives have improved/Number of islands applying the de-risking method, resource conservation measures and Ten Island Challenge tools.   Three of the five expected renewable energy transition strategies have been completed (for Belize, Saint Lucia, as well for Saint Vincent and the Grenadines). The strategies include specific targets and recommendations for renewable energy and energy efficiency projects, based on analysis targeted to each country as well as a framework and roadmap for advancing specific renewable energy projects. They await formal national approval for their implementation in some cases.  In the Bahamas the project worked with stakeholders to outline a five-year renewable energy plan, based on stakeholder consultations. The role of the Project in these efforts was to provide technical support as well as to broker the debates and discussions between interested parties and national stakeholders (for instance, between different ministries, with utilities companies, etc.).

  • Number of Resource Conservation Measures (RCMs) modelled for health centres.In Belize, work was carried out by conducting energy audits at two main hospitals, with conservation measures modelled following the results of the audit. In addition, although not strictly RCM measures modelled, in partnership with the Caribbean Community Climate Change Centre, the program worked to create an energy retrofit guide for Caribbean healthcare facilities, which included modelling of 52 specific resource conservation measures.

Tag: Climate Change Adaptation Resilience building Energy Effectiveness

23.

Results at Outcome 2 Level:   Strengthened island capacity for integrated low GHG technical and operational planning and coordination

  • Number of stakeholder partnerships active in Ten Island Challenge knowledge management platforms disaggregated by sex, by age and by rural and urban. The Project reports articulation with three networks: CARILEC (the association of Caribbean electric utilities, and partner in the CAREC community and platform); IRENA (the International Renewable Energy Agency); WIRE (the Women in Renewable Energy Network).
  • Number of local counterparts with improved capacity to partake in RE/EE developments disaggregated by sex, by age, and by rural and urban.The Project reports that the community of practice engaged with 1100 members of the CARILEC Renewable Energy Community (CAREC), of which 106 were from the beneficiary countries. It has hosted 23 webinars and eight live learning events. It has also hosted the first CAREC Renewable Energy and Smart Grid Conference in September 2018 with over 200 participants. However, this engagement is not calibrated in detail through Project reporting. For instance, it is not measured what engagement truly entailed (that is, active participants, non – duplicative participants in different activities such as webinars and other sessions, or just people who signed up to the platform). Furthermore, this reporting assumes that engagement equals improved capacity, which is not necessarily the case.

Tag: Energy Effectiveness Country Support Platform Knowledge management

24.

Results at Outcome 3 Level:   Catalysed island funding for low GHG technology deployment

Installed RE capacity through Ten Island Challenge.  Target: 6 MW; Achieved: 6.2 MW:  Saint Lucia solar, 4 MW, Bahamas solar, 925 kW, Saint Vincent Solar 500 kW; Saint Vincent (Grenadines) solar and storage microgrid, 100 kW plus 216 kWh; Saint Vincent (Grenadines) solar and storage microgrid, 800 kW plus 600 kWh.  The specific role of the project was to prepare technical backstopping, analysis, and other aspects that helped leverage funding and catalyse an enabling environment of these installations. The target indicator for mobilization of resources was 63 million USD, the achieved level was of 13.65 million USD.  Although there were expressions at design that there would be a focus on results on how the interventions would benefit underserved communities (for instance, the last expected output in this outcome indicates this), there were no specific indicators or targets to provide measurements for this matter.  The target indicator is indicative of the number of new development partnership that would, ostensibly, benefit underserved communities and women, yet it is a product/process indicator and not a result indicator. Therefore, if this has been a result (i.e. benefits for underserved communities) it was not measured nor captured as specific target indicators.


Tag: Clean Energy Energy Resource mobilization Partnership

25.

In summary, therefore, it can be stated that the TIC project delivered a set of outputs and processes in several of the target countries. Following are highlights and some of the most salient outputs at each country level as well as at regional levels of results obtained:

  • In Bahamas the project supported the drafting a 5-Year Renewable Energy Plan, as well as analysis of energy efficiency issues/energy in public buildings.The latter being one of the few produced concrete analysis and work dealing with energy efficiency and demand issues. Project reports also investments in solar energy in the country.
  • In Belize the project facilitated the drafting of a consolidated energy plan.
  • In Saint Lucia and in Saint Vincent and the Grenadines the project supported the initiation of solar farms. Furthermore, in each of these countries the Project facilitated the drafting of a national energy transition strategies.
  • At a region-wide level a community of practice was established called CARILEC Renewable Energy Community (CAREC) in collaboration with CARILEC itself. Approximately ten percent of the members of this community of practice are from the countries targeted by the Project.
  • In Grenada no results have been reported. This is due to the fact that there were certain risk conditions present in the country that hindered engagement with national-level stakeholders. That is, the Government of Grenada and the utility company were, and continue to be, in a conflict situation.At the time of project implementation, the Government of Grenada and the utility company (GRENLEC) were even part of an international arbitration process.Furthermore, government stakeholders were not responsive about carrying out the energy audit that CWR/RMI proposed since they considered that a previous audit carried out with other stakeholders was still current, and therefore a new one unnecessary.

Tag: Clean Energy Renewable energy Effectiveness

26.

Communications and visibility

A project’s external communication not only attends to the visibility of the intervention, it also gives an account of a project’s progress and intended impact through communications, outreach and even in some cases through public awareness drives.  The TIC project did not have a specific communication strategy and there was no direct nor indirect public information drives as part of the Project.  Furthermore, the visibility of the project in relation to UNDP/GEF roles was diluted and in several instances products or processes did not or do not have attribution to the role of GEF or UNDP. The Project did not communicate this strategically, and partners in turn did not perceive this properly. This lack of clarity shaped misconceptions on the part of many stakeholders regarding the Project. For instance, several key stakeholders were not aware until very late in the implementation process that this was a GEF-funded UNDP-implemented project.


Tag: Communication Project and Programme management

27.

Relevance

When analysing relevance for the TIC Project, the scrutiny can be done at two levels.  First at the level of needs for countries involved and second at the level of formal aligning of the Project with development plans and UNDP/GEF corporate mandates.  The latter relates as to the extent to which a project and its interventions and activities are suited to local and national development priorities and needs as well as programmatic UN priorities. Regarding the former, relevance vis-a-vis the countries’ needs, it can be stated that the Project was relevant to a very large degree. 

Regarding alignment with regional plans as well as corporate and programmatic UN priorities, the TIC Project is fully aligned with both mandates.  As indicated in the Project Document, as well as in other related implementation documents, the Project is aligned with explicit policies as indicated below, current at the time of design and formulation:

  • UNDAF/SPD Outcome 1:   Enhanced capacity of national, sub-regional and regional institutions and stakeholders to: effectively manage natural resources; build resilience to the adverse impacts of climate change and natural and anthropogenic hazards; improved energy efficiency and use of renewable energy; improved policy, legal, regulatory and institutional frameworks for environmental and energy governance
  • UNDP Strategic Plan 2014-2017 Primary Outcome: Growth and development are inclusive and sustainable, incorporating productive capacities that create employment and livelihoods for the poor and excluded

 °Output 1.4. Scaled up action on climate change adaptation and mitigation across sectors which is funded and implemented

°Output 1.5. Inclusive and sustainable solutions adopted to achieve increased energy efficiency and universal modern energy access (especially off-grid sources of renewable energy)

  • Expected M-CPAP Output(s): Output 6: Improved energy efficiency and the removal of barriers to the introduction and transfer of renewable energy technology facilitated

Tag: Climate Change Adaptation Resilience building Renewable energy Relevance Institutional Strengthening National Institutions Regional Institutions

28.

Furthermore, the Project is aligned with and relevant to regional development as well as environment and climate change objectives as defined by CARICOM, OECS and other regional frameworks.  For instance, it is aligned with the “Barbados Declaration” of 2012 which entailed voluntary commitments from Small Island Developing States (SIDS) contributing to the Sustainable Energy for All (SE4ALL) initiative, reiterated at the UNCSD Rio+20 Conference.  Moreover, at the country level, Caribbean countries are focusing their long-term sustainable development strategies on the principles of climate risk management and resilience building.   In this context, the latter are understood as market transformations based on “adjustments in ecological, social, or economic systems in response to actual or expected climatic stimuli and their effects or impacts”.


Tag: Climate Change Adaptation Resilience building Environment Policy Relevance

29.

The Caribbean Community Climate Change Centre (CCCCC) has been designated by CARICOM Heads of Government as the regional coordinating agency for the response to climate change, guided by the Regional Framework for Achieving Development Resilient to Climate Change 2011-2021 and its Implementation Plan.  It is likewise a repository for regional climate change information and data. This Centre was also a partner in the implementation of several products associated to the Project in Saint Vincent and the Grenadines, clearly linking therefore the expressed relevance of regional policies with direct implementation.

 


Tag: Climate Change Adaptation Clean Energy Energy Relevance Policies & Procedures

30.

Additionally, the project correlates to the GEF Climate Change focal area which has been designed to deliver global environmental benefits in line with relevant international climate change objectives. The applicable GEF Strategic Objective and Program are GEF-6 CCM-1 Strategic Programs 1 and 2, and the applicable GEF Expected Outcome are Outcomes A, B and C for accelerated adoption of innovative technologies, policy frameworks and financial mechanisms for GHG emission reductions.  The applicable relevant GEF Outcome Indicators are: (1) Market penetration of on-grid RE (% from renewables); (2) GHG emissions from electricity generation (tons CO2eq/kWh and $/tons CO2eq); and (3) no. of jobs/beneficiaries.  The focal area Objectives, (OP/SP) are two: SP 1. Promoting access to clean and affordable energy services; and SP 2. Promoting low emission and climate resilient urban and transport infrastructure (CCM Program 3).

Given the relevance of the issue for the countries involved as well as the alignment of the Project with UNDP, GEF, regional and national mandates, the rating for this criterion is R (Relevant).


Tag: Climate Change Adaptation Emission Reduction Resilience building Clean Energy Relevance National Regional

31.

Effectiveness and Efficiency

Effectiveness and efficiency are two very inter – related concepts in project evaluations.  Efficiency is defined as the extent to which results have been delivered with the least costly resources possible. Efficiency is a measure of how economically resources/inputs (funds, expertise, time, etc.) are converted into results. The effectiveness of a project is defined as the degree to which the development intervention’s objectives were achieved.   The valorisation of effectiveness is used as an aggregate for judgment of the merit or worth of an activity, (i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives proficiently in a sustainable fashion and with a positive institutional development impact).  

Regarding efficiency, the Project has been efficient in achieving outputs/products and in achieving outcomes and effects/impact in most of the target countries, yet to a varying degree of accomplishment. Also, it has provided value-for-money since it achieved the results within budgets, agreed disbursement, etc., while leveraging investments and in-kind support from sources external to the project per se.  The efficiency in this case also relates to the internal control and internal efficiency capacities as well as meeting with the explicit responsibilities of the implementing partner (i.e. CWR/RMI) for the execution of project funds. In terms of efficiency, also, HACT and financial audits were carried out periodically, and successive audits demonstrated financial efficiency in the use of funds. Therefore, the overall ranking of efficiency is   Satisfactory (S).


Tag: Efficiency Human and Financial resources

32.

Effectiveness and Efficiency

The effectiveness, that is, the extent to which the TIC Project has attained its major relevant objectives is also varied.  In some countries and for several of the expected outcomes it did, in others it did but to some degree, and in another country it did not. Also, it is problematic to determine the effectiveness of the community of practice (CARLEC). Although the CoP was operative at the product/process level, that is it did carry out webinars/events/etc., the effect or effectiveness of such product/processes and their results has not been captured.  Therefore, the overall ranking of efficiency for the Moderately Satisfactory (MS).

 


Tag: Effectiveness Knowledge management

33.

Countries’ ownership

Assessing country ownership for the TIC project is somewhat complex given the actual implementation modality. There are elements that indicate that there was national ownership in most countries (expect Grenada) to a varying degree between countries, however. The different Government’s explicit involvement and support of the Project, and the involvement of different institutions that deal with energy, is indicative to some degree of ownership factors in this scope. Yet having been this a project implemented by a non-state actor, and additionally from outside the Caribbean region, on the other hand has curtailed to some degree the country ownership normally derived from this sort of project as well as the potential institutional development impact that such an endeavour could have had.


Tag: Implementation Modality Ownership

34.

Mainstreaming

Given that UNDP-supported GEF-financed projects are key elements in UNDP country programming, project objectives and outcomes should align with UNDP country programme strategies as well as to GEF-required global environmental benefits. When dealing with mainstreaming, evaluations also explore whether project outcomes are being mainstreamed into national policies.  

The TIC Project has help create products, also, that, if implemented, could be attuned with UNDP priorities of improved governance and improving natural resource management (for instance the renewable energy plans developed in several of the countries as well as the energy audits developed in others). Other UNDP priorities, such as sustainable human development, and a gender equality approach were not specified clearly as expected outcomes of the Project. That is, they were not specified strongly as outcomes.


Tag: Energy Natural Resouce management Policies & Procedures Institutional Strengthening

35.

Regarding development, it is noteworthy that although the project focused upon global environmental benefits derived by the reduction of emissions (linked therefore with the Project’s major metric of planned success), there were other developmental outcomes that are also related to mainstreaming. For instance, regarding equity, since generation of renewable energy is not oil-dependent, fuel surcharges are not levied nor paid by consumers, making renewable energy more equitable and less costly in countries where electricity is quite expensive. Moreover, as several stakeholders at the national levels of implementation have pointed out, the drive for renewable energy implies that a country is less prone to be affected by the variations and fluctuations of the international oil market and therefore more self-contained in relation to energy generation. Lastly, and this was included as an expected outcome, there has been job creations in the setup of renewable energy endeavours (construction and procurement), albeit a temporary, it is a benefit nonetheless that generated capacity.


Tag: Energy Jobs and Livelihoods

36.

Although the Project developed activities related to women in the renewable energy field (and harnessed some data differentiated by gender regarding women’s participation in the Project or in the associated networks) the project did not thoroughly consider thematic issues related to gender and energy issues beyond the participation of women in networks, as employees in the field, etc. In particular, the project did not seek nor actively pursue equality of access to clean energy services, for instance. As the 2019 cumulative PIR indicates, there has been no noticeable contribution to gender equality (GEN0 Atlas Gender Marker Rating).

 


Tag: Clean Energy Gender Equality

37.

Sustainability

Sustainability of an intervention and its results are examined to determine the likelihood of whether benefits would continue to be accrued after the completion of the project.   Sustainability is examined from various perspectives: financial, social, environmental and institutional.

Financial sustainability:  Financial risks to sustainability relate to the likelihood of financial and economic resources not being available once the assistance ends.  Since financial risks are at the very core of many of the issues related to renewable energy investments in the five countries involved, financial sustainability is a key issue to maintain and accrue benefits.  Even at the planning stage this issue was identified, given that it was indicated that “Perhaps most important, however, is the fact that these islands are small economies, and it can be difficult to attract investor interest and the capital needed to construct these facilities.” The plans drawn (such as the integrated resource plans and the national energy transition policies) need a strong financial substructure to be implemented, when and if approved. Moreover, the CAREC community of practice, although already established and successfully integrated to CARILEC, would also need funding for this institution to continue to run it and be sustainable in the long term. Therefore, the ranking for financial sustainability is Moderately likely (ML), given that, although there are moderate risks, there are also expectations that at least some of the outcomes will be sustained in time.


Tag: Sustainability Partnership

38.

Socio-economic risks to sustainability: When analysing socio-economic risks to sustainability, an examination is made of the potential social or political risks that may jeopardize sustainability of project outcomes.  The level of stakeholder ownership, as seen in the narrative of this report, is somewhat strong in some countries and weak in others, and this poses some socio-economic risks to sustainability. Although governments do indicate that it is in their interest that the project’s benefits continue to accrue, other risks are still identified for some countries.   Conceivably this is indicative of the countries willingness and or need to shift more of their energy matrix to renewables. The two countries that presented more concrete results (i.e. Saint Lucia and Saint Vincent and the Grenadines) were the two countries more prepared to transition fossil fuels to renewables due to the cost of energy from fossil fuels, their plans to impulse renewable energy before the project, etc. That is, several conditions were in place prior to the project that aided in buy-in and conceivably to socio-economic sustainability. Countries not fully engaged in transition to renewables for several reasons (cost of energy already used does not support cost-benefit for shifting to renewables, internal conflicts, etc.) are the ones more likely to present socio – economic risks to sustainability. Therefore, the ranking for socio-economic sustainability is Moderately Likely (ML), given that, although there are moderate risks, there are also expectations that at least some outcomes at different levels would be sustained.


Tag: Renewable energy Challenges Sustainability Ownership

39.

Institutional framework and governance risks to sustainability: At the time of the final evaluation there are no clear institutional and governance changes in most countries that would link to governance sustainability. There are no clear-cut legal frameworks, policies, governance structures and processes in place attributable to the Project as such. Yet, some of the energy-related plans generated within the scope of the TIC Project that pend approval are likely to be accepted and adopted. Therefore, the ranking for this sort of sustainability is Moderately Likely (ML) given that there is substantial risk that outcomes will not materialize in a manner attributable to the Project or will not carry on after project closure, although some outputs and activities should and are likely to carry on.


Tag: Sustainability Country Government

40.

Environmental risks to sustainability: Environmental risks to sustainability conflicts are identified as those associated with natural resource management and regarding climate change.  Weather patterns in the Caribbean due to climate change continue to affect and increasingly impact island and coastal nations due to an increase in severity of weather events such as hurricanes. At the time of the evaluation mission a hurricane category five devastated Bahamas, and this has brought to bear to stakeholders in the region several issues related to energy.  For instance, that solar farms are currently planned to withstand lesser hurricane categories than five, therefore stakeholders indicate that they will need to plan with higher scale impacts in mind than what is done currently or what was done within the Project’s framework. This includes ensuring that design and set up of investments can withstanding extreme weather conditions.  Therefore, given the moderate risks faced, the ranking for environmental sustainability is Moderately Likely (ML).


Tag: Environment Policy Natural Resouce management Sustainability Climate Change Adaptation Natural Disaster Energy

41.

With regards to sustainability, it is noteworthy to observe that the Project IP has developed a document containing an exit strategy report that deals with several of the sustainability factors indicated above.  Although the report has only been generated at closing, with little opportunity as of yet for stakeholder assimilation and ownership in the countries involved and in the region, it draws a general road-map for follow-up and sustainability (see Annex  6:  Exit Strategy Report).  It identifies specific stakeholders to whom responsibilities for follow-up and sustainability would transfer and presents what processes need to be in place to ensure that positive impact continues following closing. It does not specify many sustainability issues at the country levels, but it deals a great deal with sustainability for the community of practice, in order for CARILEC to assume the running of this component. It also has components for replication and catalysation of results in other countries besides the target countries associated to the GEF/UNDP initiative.


Tag: Sustainability Ownership

42.

Impact

The environmental impact and global environmental benefits of the project is mainly measured through the metric of greenhouse gas emissions avoided. Although expected impact indicators were not achieved at the expected level due to the over ambitiousness of these upon design, there has been clear evidence demonstrating that there have been verifiable improvements in global ecological status and verifiable reductions in stress on ecological systems due to emissions avoided. Given that the emissions avoided thus far are associated to the new solar energy endeavours and their installed renewable energy capacity, it can be stated that the impacts are sustainable, at least as long as these investments remain operational. Furthermore, the leveraged new sources of financing and investment is also a demonstrated effect that has led to impact.

Although much of the emphasis has been on impact and corresponding metrics for emissions avoided, there are other effects discerned thus far that did not have any specific measurement attached, but have been achieved nonetheless. For instance, reduction of energy costs (globally and for consumers), reduction on dependency on foreign oil, and generation of capacity in-countries to transition towards energy matrixes more broadly based on renewables.  Therefore, the impact ratings for the TIC initiative is Significant (S).


Tag: Emission Reduction Energy Challenges Impact Programme/Project Design

Recommendations
1

Design, especially of complex interventions, should contain in-depth knowledge of the areas and countries where interventions will take place, not only of the subject per se of the project being implemented, but also of the development context, political framework as well as assumptions and risks of the intervention.  There has to be extensive underpinning and analysis during design to harness strategic knowledge of the area(s)/country(ies) where a project would be implemented.  If the project is to be implemented by an organization from outside the country, then there have to be links to national/regional resource persons and expertise which can provide knowledge and information.

2

In order to enhance stakeholder engagement in implementation, and therefore impel improved results with a strong ownership potential, at design there should be a careful stakeholder analysis.  This analysis should not only include a list of potential stakeholders, yet it should contain an accurate typology, stakeholder mapping, and also indicate what are stakeholders’ concerns, incentives, goals and expectations vis-à-vis the anticipated results.

3

Outcome indicators should be robust, yet they should also be applicable and achievable within the context of implementation as well as realistic overall in terms of a project’s resources and implementation time.

4

There should be opportunities for the adjustment and reformulation of indicators and metrics package throughout the implementation process as a means for adaptive management.

  1. This should be communicated properly to the relevant partners, indicating how these changes are to be conveyed, what supporting data and information is needed for the changes, and what are the appropriate mechanisms. 
  2. This process should be communicated to relevant partners early enough in the implementation process.
  3. Project reformulations, changes, reforms and other such alterations need to be precise, and implemented as soon as early signs of failings manifest themselves.
  4. When these opportunities are not presented compellingly, UNDP should assert the need for this to take place in order to improve implementation.
5

Projects such as this one, with inherent complexities, pilot projects and interventions testing innovative implementation modalities should have a mid-term review, not only to give transparency to the achievements up to the midpoint assessment but also to act as a catalysing factor to adjust whatever needs to be adjusted at that point.

6

Projects that instrument innovative sorts of implementation modalities should have very clear guidelines from onset on all matters related to implementation.

  1. Much as more traditional implementation modalities have manuals and guidelines and procedures, these should also be present for non-governmental modalities and the same imbedded in design instruments. 
  2. Matters such as accountability, reporting, and oversight should be made specific and adapted to the modality.
  3. Particularly when implemented by entities outside a region or outside a country, staffing should include national project coordinators, in order to support the implementation as a whole in the particular country, attend to national implementation, and to anchor the project at the national level.
7

When training activities are designed, uptake and use (that is, monitoring of the application of knowledge acquired), as well as actual capacity built should be measured in order to understand the effects of these activities and to promote improvements and upscaling in the future. Indicators for increasing or building capacity should be robust measures of improved capacity, and at all possible, also capture use of conveyed capacity.

8

Multi country projects should have mechanisms for horizontal exchanges between and among the countries involved.  Not only to trade information but also to exchange lessons learned, problems and achievements between the different stakeholders from the different countries where a project is implemented.

9

 Risks within a project should not be underestimated, and a risk management framework should be drawn at design and reviewed continuously.  Once properly established, risks should be continuously monitored in order to promote whatever mitigation measures or adaptive management needs to be implemented.

10

Development projects such as this should have as its primary prospect to generate durable capacity at the national/regional levels. All efforts should be made to generate local capacity as well as introduce national issues in the resulting products. Although engaging with an entity from outside a country or region might be necessary to pilot interventions, follow up should be carried out as much as possible with national or regional institutions and/or expertise.

11

In order for projects to promote a gender equality approach, a strategy (that is an action plan based on gender analysis) should be set that addresses fully the different needs of men or women.  It should be comprehended that a “women in” approach (that is the participation of women in whatever fields or activities a project promotes) is not a comprehensive gender equality approach. Projects need to consider gender mainstreaming strategies from the onset of an intervention.

12

Exit and sustainability plans should be drawn earlier than at project finalization.  This is in order for these plans to be appropriated by the relevant parties which will implement all or some of the strategy’s options. An exit/sustainability strategy should also aim to make implementing partners and implementing agencies redundant, by accurately transferring capacity and the mechanisms that generate sustainability of achievements in the long term and not continue to be dependent upon them.

13

Projects should have a clear communication strategy. 

  1. A project’s communication strategy should be an ongoing process that generates buy-in, generates knowledge about the issues a project deals with as well as acknowledge its visibility. 
  2. A communication strategy needs to be accompanied by clear inputs where the different partners are identified (funders, implementing agency(ies), UN agencies involved). 
  3. A communication strategy should document and communicate issues, achievements, and challenges. 
  4. Also related to communication is the need to give proper visibility and transparency to all partners involved and what are their roles in a project.

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org