Energy and Environment Outcome Evaluation

Report Cover Image
Evaluation Plan:
2017-2021, Lebanon
Evaluation Type:
Outcome
Planned End Date:
07/2019
Completion Date:
05/2019
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document E&E-outcome-evaluation-final-TOR.pdf tor English 526.63 KB Posted 387
Download document 01 EVALUATION REPORT 07 FINAL.pdf report English 1057.95 KB Posted 617
Title Energy and Environment Outcome Evaluation
Atlas Project Number: 00098956
Evaluation Plan: 2017-2021, Lebanon
Evaluation Type: Outcome
Status: Completed
Completion Date: 05/2019
Planned End Date: 07/2019
Management Response: Yes
UNDP Signature Solution:
  • 1. Sustainable
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation
SDG Goal
  • Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss
  • Goal 6. Ensure availability and sustainable management of water and sanitation for all
SDG Target
  • 15.9 By 2020, integrate ecosystem and biodiversity values into national and local planning, development processes, poverty reduction strategies and accounts
  • 6.3 By 2030, improve water quality by reducing pollution, eliminating dumping and minimizing release of hazardous chemicals and materials, halving the proportion of untreated wastewater and substantially increasing recycling and safe reuse globally
  • 6.4 By 2030, substantially increase water-use efficiency across all sectors and ensure sustainable withdrawals and supply of freshwater to address water scarcity and substantially reduce the number of people suffering from water scarcity
Evaluation Budget(US $): 20,000
Source of Funding: CO
Evaluation Expenditure(US $): 21,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Philip Tortell
GEF Evaluation: No
Key Stakeholders: Ministry of Environment, Ministry of Energy, CDR
Countries: LEBANON
Lessons
Findings
1.

3.1 The Energy and Environment Programme
3.1.1Programme design and the Strategic Results Framework

There is no programme document for the Energy and Environment Programme and the only image of the programme design is as represented in the SRF as in Table 2 in Section 1.3 above and in Annex 2b which are from the CPD. As commented above, the SRF is somewhat confusing as it seems to lack the usual logical relationship between the different elements of a SRF. In an attempt to clarify the relationship between the levels of the SRF, the Team advised that the following is the
structure that has been used for implementation purposes.


Tag: Energy Environment Policy Integration Programme/Project Design Results-Based Management

2.

3.2 Programme relevance
Programme relevance is assessed at three levels 15 . Firstly, it is assessed for relevance to the Government and people of Lebanon reflecting national needs and priorities; secondly, relevance to UNDP and its corporate commitments; and, finally relevance to the Targets and Indicators set by the CPD Priority Area 4 and UNSF Outcome 3.3. It also needs to be noted that this assessment of relevance is restricted to the 17 projects that have either been initiated, or are pending or are in the hard pipeline since the commencement of this CPD period. In other words, the 19 projects which have been carried over from the previous CPD period cannot be assessed for relevance to something which was designed after they had already started implementation.


Tag: Relevance Project and Programme management

3.

3.3 Programme effectiveness
Effectiveness is a measure of the extent to which project targets have been achieved and it is best assessed when a project has been completed. This section is therefore based on the 10 projects that have finished 20 (all of which had been carried over from the previous CPD period) and that have been subjected to an evaluation.
 


Tag: Energy Effectiveness Project and Programme management Data and Statistics

4.

3.4 Implementation efficiency

Efficiency is a measure of the extent and the competence through which resources have been converted into results.


Tag: Efficiency Communication UNDP Regional Bureaux

5.

3.4.2 Personnel, expertise and knowledge sharing

The most valuable and tangible resources of the Programme are its personnel, those in the central office and those leading and managing UNDP projects and other initiatives. Also in this category are the implementation partners through whom UNDP achieves its results.

The central office serves as the hub for the Programme. This is where the Programme is designed, managed, coordinated and monitored. There is a staff of four – the Programme Manager, a Programme Associate, a Programme Assistant, and a Programme Support JPO. This immediate team is supported by the Country Office administration and support units such as Human Resources, Financial, Legal, etc.
 


Tag: Energy Efficiency Oversight Project and Programme management Country Government UNDP management UNDP Regional Bureaux Operational Services

6.

3.4.3 Financial resources

Financial resources are among the most tangible (and measurable) inputs for UNDP work and expenditure is often used as a measure of delivery, overshadowing the more important results achieved. There is only a slight tendency for such an approach with the E&E Programme.
 


Tag: Efficiency Resource mobilization Human and Financial resources Country Government UN Agencies

7.

3.4.4 Partnerships

UNDP is an implementing agency but most often it needs executing partners to realize any results; it also relies on donor partners for funding. As with the rest of the UNDP family, the E&E Programme depends almost entirely on executing partners and funding partners to achieve its goals. Worthy of special mention is the partnership that the Programme is developing with the private sector  and this is in accordance with the adopted UNDP strategy .
 


Tag: Environment Policy Partnership Country Government Donor UNDP Regional Bureaux Coordination

8.

3.4.5 Monitoring and Evaluation

The CPD discussed monitoring and evaluation for the Country Programme and proposed a number of monitoring tools. The evaluator invited the E&E Programme Team to assess the extent to which these tools have been used and how efficient they had been. The results are summarized below.
 


Tag: Monitoring and Evaluation

9.

3.4.6 Risk management

The following risks were identified for the whole of the CPD together with possible mitigation measures. The Programme Team was invited to comment on the extent, if any, that the risks did eventuate for the Programme, and if so, whether the mitigation measures worked, and what the impacts on the Programme were. From all indications, it is evident that the Team is recognizing and managing risks adequately. However, it would be an improvement if there was a discussion of risks and mitigation measures focussed on the Programme in a dedicated Programme Document.


Tag: Efficiency Risk Management

10.

3.4.7 The gender dimension

Gender equality, women’s empowerment and human rights are among the key priorities of UNDP at the corporate level. However, they are best displayed at the projects level which is the delivery mechanism for the Programme. Less than 30% of Project Managers said that their project made a contribution towards gender equality and women’s empowerment. The rest said either that it did not do so or that the question did not apply to their project. This is a disappointing result but not entirely surprising since it is known that while MDG targets in Lebanon have been met in the health and primary education sectors, they were not in poverty, gender equality and environmental sustainability.

UNDP has completed a Gender Strategy 37 which found that women’s participation in programme design and implementation contributed to reducing gender inequality; breaking traditional gender roles; increased lobbying for women’s representation and participation in national governance, and a greater number of leadership opportunities for young women. However, the Strategy also found that the Country Office “does not yet make the direct link between gender and environment. This can be explained by the fact that projects are designed to target all Lebanese citizens without distinction. The efforts for gender-mainstreaming at this level do not go further than maintaining a balance between male and female staff, ensuring a lack of discrimination against women, and, in some cases, including a theoretical section on gender”.

This shortfall is recognized by the Programme and it is useful to use a very recent project currently awaiting signature as an example of how GEEW is being addressed. This project, which deals with land degradation and land use, has an impressive 10-page Gender Analysis and Mainstreaming Plan in an annex in addition to extensive gender provisions which are integrated in the project document. The Plan identifies gender concerns and opportunities to ensure that the project considers women’s and men’s different vulnerabilities and needs as well as capacities and skills and achieves an equitable distribution of its benefits, resources, status and rights. The Plan also outlines ways for the project to influence transformative changes in the norms, cultural values and the roots of gender inequalities and discriminations. Furthermore the Plan is aligned to other strategies and processes of the project to ensure feasibility and is expected to be mainstreamed as much as
possible with on-going national processes and strategies at the time of project implementation and wherever feasible. This is indicative of the manner in which GEEW is fostered by the E&E Programme through its projects.


Tag: Efficiency Gender Equality Gender Mainstreaming Women's Empowerment MDGs

11.

4.5 Sustainability of results

The likelihood of sustainability for the benefits and outcomes of the projects was seen as very high by Project Managers. In fact, 15 (88%) said there will be sustainability, one said maybe and one said it was not guaranteed. The questionnaire also asked Project Managers if their project had a “champion” who would inherit the project benefits and outcomes. Two did not understand the question, however, the most common champions identified by the rest (76%) were the project stakeholders while a small number (23%) listed the executing partner as the likely champion.
 


Tag: Energy Waste management Water resources Sustainability Human and Financial resources Project and Programme management Civil Societies and NGOs UNDP management UNDP Regional Bureaux Capacity Building

12.

3.1.2 The Programme objectives and targets
In the absence of a Programme Document, the E&E Programme objectives are unclear. The UNDP Lebanon website 14 states that “The Environment and Energy (E&E) Programme works closely with the Government of Lebanon to provide policy support for more effective environment and energy management. The programme also works with local communities and civil sector organisations to improve livelihoods through improved natural resource management. Our aim is to move Lebanon towards achieving the Sustainable Development Goals (SDGs) more specifically Goal 7 on environmental sustainability.” This may be an indication of the Programme objectives and targets but it is not explicit.

The CPD narrative refers to the analysis carried out and the conclusion reached that what is required is a combination of a “short-term crisis response with long-term initiatives that fall under theumbrella of the new sustainable development agenda”. More specifically, for the environment and energy thematic area, the objective is seen as “to halt the environmental degradation that has been exacerbated by the crisis and negatively influenced Lebanon’s chances of securing long-term sustainable development.”

However, the Programme also takes its cue from the UNSF 2017-2020 Core Priority 3: Lebanon reduces poverty and promotes sustainable development while addressing immediate needs in a human rights/ gender-sensitive manner. Under this priority area, the Programme is meant to contribute to Outcome 3.3 which strives for the broad objective of – Lebanon has adopted measures to improve environmental governance.
 


Tag: Environment Policy Partnership Policies & Procedures Policy Advisory SDG Integration Civil Societies and NGOs

13.

3.1.3 The projects portfolio
Forming the foundation of the delivery mechanism working towards Outcome 3.3 and the scope for this evaluation, are the projects and other interventions implemented by the Energy and Environment Programme. Table 5 below represents the full list of projects, the major implementing partner, the UNSF and CPD Indicators they are meant to contribute to, and the project status.

The portfolio comprises 36 projects. Of these, 19 projects were carried over from the previous CPD period with ten having since been closed and nine still continuing. Some of the projects that have been carried over have been through a number of phases according to the PMs’ Questionnaire (see
Annex 6b) and are still continuing. The projects ran/will run for variable lengths of time, not always finishing within the original allocated time. Nine projects have been granted extensions ranging from 3 months to a full 48 months.

Six new projects have been initiated in this CPD period and all are still under implementation. In addition, two projects are pending signatures and are expected to start implementation soon. There are also another nine projects in a hard pipeline and some of these may be initiated during this CPD period.

 


Tag: Programme/Project Design Project and Programme management

14.

3.1.4 Programme resources
Programme and project resources comprise the inputs required to carry out the programme tasks. They include people, equipment, facilities, funding, knowledge and time. Key resources are discussed in section 3.4 below which assesses the extent to which resources have been converted into results.


Tag: Human and Financial resources

15.

3.2.1 Relevance to the Government and people of Lebanon
Drafting of the UNSF and the CPD was carried out in or around 2016 following a thorough assessment of the situation in Lebanon and they reflect the identified needs. Although this Common Country Assessment (CCA) was not published on this occasion because of the prevailing situation, it still formed the basis for the UNSF and the CPD.

At the Programme level, the assessment is deeper than in the CCA and it arises from the Team’s deep knowledge of the situation and their analysis of the country priorities, their close contact with various arms of the government system, the reports from Project Managers, and discussions with other stakeholders such as key NGOs.


The Programme initiated six projects, a further two are pending signature and due to start and there is a hard pipeline of nine projects. The following table lists the total 17 projects initiated or due to start within this CPD period, together with their objectives and brief descriptions, and then assesses their relevance to the Government and people of Lebanon as envisaged at the time of project formulation and since.

A full 70% of the projects are considered Very Relevant and 23% are Relevant. Only one project is seen as Moderately Relevant.

The projects in the portfolio rightly take their cue from the situation existing at the time, and reflect known government strategies and plans as well as international obligations. Of the Project Managers that responded to the related question in the Questionnaire (see Annex 6b), four appeared confused by the question. A further four saw their project contributing to the NREAP – the National Renewable Energy Action Plan. Other instruments mentioned included strategies/plans on Solid Waste and Water, and the Lebanon Crisis Response Plan.

A good measure of Lebanon’s long term priorities are the SDGs (Sustainable Development Goals) and Project Managers, in their responses to the Questionnaire, overwhelmingly said that their respective projects contributed to Lebanon’s progress towards the SDGs 16 .

It was ascertained during consultations with MoE and MoEW as well as CDR senior officials and others, that the contribution made by the E&E Programme is essential for a number of key aspects of the Government’s work. It can be concluded that the Programme and its portfolio of projects reflect the situation on the ground, current or immediately past, and are highly relevant to the Government and people of Lebanon.


Tag: Relevance Project and Programme management Country Government

16.

3.2.2 Relevance to UNDP
The UNDP Strategic Plan 17 current at the time of Programme design identified the following key
areas of focus of interest to the Programme –
(a)Effective maintenance and protection of natural capital
(b)Sustainable access to energy and improved energy efficiency
(c)Planning, policy frameworks and institutional capacities to substantially reinforce
action on climate change
(d)Assessment of key economic, social and environmental risks to the poor and
excluded
Furthermore, UNDP has identified the following set of approaches that it sees as its Signature Solutions 18 –
• Keeping people out of POVERTY
• GOVERNANCE for peaceful, just, and inclusive societies
• Crisis prevention and increased RESILIENCE

•ENVIRONMENT: nature-based solutions for development
Clean, affordable ENERGY

  • Women's empowerment and GENDER equality

The above have created the framework within which the E&E Programme could be designed and operated. By focussing the Key Areas and Signature Solutions it is possible to lead to seven thematic areas of direct interest to the E&E Programme. These are listed in the Table below together with an indicative matching of the 17 projects initiated during this CPD period.

In the above table, the first five thematic areas arise from strategic UNDP documents whereas the Biodiversity, the Ozone and the institutional capacity thematic areas do not. It would seem that seven projects are not directly relevant to the current UNDP strategic priorities, however, 10 projects
are. As such, the Programme can be considered as relevant to UNDP and its corporate commitments although slightly less than for the Government and people of Lebanon. This is also borne out by the slight majority of Project Managers (59%) who confirmed this when they asserted that their project design did take into account the UNDP commitments for gender equality, human rights and conflict sensitivity (Annex 6b) thus reiterating the Programme’s relevance to UNDP.


Tag: Biodiversity Relevance Programme Synergy Project and Programme management Results-Based Management

17.

3.2.3 Relevance to the CPD and the UNSF Outcome 3.3

The Programme addresses CPD Priority Area 4 with its focus on improving environmental governance. More specifically, it addresses the two CPD Outputs with five Indicators between them. The following table assesses the fit and relevance of the 17 projects that have been initiated or are in pipeline as they were assigned by the Programme Team to one or more CPD Indicators.

However, in assessing the relevance of the constituent projects to the Outputs and Indicators of the CPD, and through them to the three Indicators of the UNSF Outcome 3.3, the internal relevance and logic problems of the SRF became more noticeable. The table shows that some of the projects can be said to be relevant to the CPD Indicators and not to the UNSF, others are relevant to the UNSF and not to the CPD. Possible reasons for this discrepancy and the apparent problem with uniform relevance are discussed in section 4.2 below.

It is interesting to observe that 10 Project Managers (59%) identified correctly the CPD Output that their project contributed to. However, three PMs did not respond to the question and four provided an uncertain answer.

In conclusion it can be said that the Programme is highly relevant to the Government and people of
Lebanon. It is also relevant to the UNDP corporate commitments. However, it is less relevant to the CPD and the UNSF Indicators and Targets while being slightly more relevant to the strategic approach of the CPD. This apparent inconsistency appears to be caused by the confused SRF.


Tag: Relevance Project and Programme management

18.

3.3.2 Effectiveness according to the Outcome indicators and targets
Another way of looking at effectiveness is to assess project delivery of results towards the indicators and targets of the UNSF Outcome 3.3 and CPD Priority Area 4. The Programme Team has assigned each project in the portfolio to one or more Indicator/Target and this is used for the quarterly monitoring of progress. For the purpose of this evaluation, only those projects which have closed are assessed for their final effectiveness, and these all happen to be projects that have been carried over. Some projects are listed more than once since they are deemed to have contributed to more than one UNSF/CPD Target. Terminal evaluations, similar end-of-project reports and one MTR as in the table above are the main sources of information for this assessment.

The results recorded above are a mixed bunch. Contributions made by projects towards the CPD Targets do not always lead to similar gains towards the UNSF Targets and as discussed elsewhere and analysed in section 4.1 below, this is the result of the internal lack of logic and relevance between the CPD and the UNSF.
The following table, matches the achievements from the above table with the Targets and it can be concluded that the overall level or achievement and effectiveness is reasonable at the mid-term of this cycle.

Considering that none of the above projects were designed specifically to deliver the indicators and targets of the current UNSF and CPD, the results show a surprisingly reasonable fit. This is the result of the painstaking effort by the Team to select the broadest and most inclusive wording for the CPD Outputs and Targets to allow existing projects to be accommodated (rather than the other way round). Whether this is justified or has been successful is discussed in section 4.3 below.

The overall level of effectiveness of closed and evaluated projects in the E&E Programme is Satisfactory at the project level, but somewhat less certain when assessed against the Targets of the CPD. However, if the Indicators from the UNSF are assumed as truly indicative of what the Outcome 3.3 aimed for, then the conclusion is that the Outcome is very likely to be achieved. Whether this assumption is correct, is also discussed in section 4.3 below.


Tag: Effectiveness

Recommendations
1

1.1 UNDP to accept that the larger part of the portfolio’s projects is carried over from previous CPD and that these projects were fully justified and relevant when they were started.

1.2 UN system that in order to resolve the challenge facing programme teams as they try to balance known national needs with diluted UNSF Outcomes, the role of the UNSF should be to set the scene by outlining needs and priorities and focusing on guidance and direction for the various agencies

1.3 It is recommended to the E&E Programme Team that they prepare a Programme Document which should distinguish between projects that are being carried over and those newly initiated in response to new needs and priorities

2

1.4 It is recommended to the E&E Programme Team that all projects must undergo a terminal evaluation before closure so as to assess the relevance of each project to the country needs and priorities as identified in the CPD and the Programme, assess the effective delivery of results compared with the targets sought, assess the efficiency through which the resources available have been converted into results, and assess the likelihood of sustainability of the project’s results

3

1.5 It is recommended to UNDP that in an effort to increase the likelihood of sustainability of project results, projects should preferably fit within a broad and strategic context such as national plans and strategies.

4

1.6 An ex-post assessment should be carried out for each project some time (3-5 years) after project closure to assess sustainability and the circumstances that might have contributed towards or against it.

5

1.7 It is recommended to the E&E Programme Team to continue with and improve the practice of requiring the formulation and implementation of a meaningful gender strategy for each project thus contributing to the achievement of SDG-5 for Gender Equality.

6

1.8 It is recommended to the E&E Programme Team that in view of its current success and the continuing needs, priority should be accorded to interventions that deal with renewable energy, management of wastewater and prevention of pollution, sustainable land management and rehabilitation of degraded land and forests

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org