Evaluation Detail

Energy and Environment Outcome Evaluation

Evaluation Plan:: 2017-2021, Lebanon
Evaluation Type:: Outcome
Planned End Date:: 07/2019
Completion Date:: 05/2019
Status:: Completed
Management Response:: No
Evaluation Budget(US $):: 20,000

Document	Type	Language	Size	Status	Downloads
E&E-outcome-evaluation-final-TOR.pdf	tor	English	526.63 KB	Posted	262
01 EVALUATION REPORT 07 FINAL.pdf	report	English	1057.95 KB	Posted	446

Title

Energy and Environment Outcome Evaluation

Atlas Project Number:

00098956

Evaluation Plan:

2017-2021, Lebanon

Evaluation Type:

Outcome

Status:

Completed

Completion Date:

05/2019

Planned End Date:

07/2019

Management Response:

Yes

UNDP Signature Solution:

1. Sustainable

Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)

1. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation

SDG Goal

Goal 15. Protect, restore and promote sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, and halt and reverse land degradation and halt biodiversity loss
Goal 6. Ensure availability and sustainable management of water and sanitation for all

SDG Target

15.9 By 2020, integrate ecosystem and biodiversity values into national and local planning, development processes, poverty reduction strategies and accounts
6.3 By 2030, improve water quality by reducing pollution, eliminating dumping and minimizing release of hazardous chemicals and materials, halving the proportion of untreated wastewater and substantially increasing recycling and safe reuse globally
6.4 By 2030, substantially increase water-use efficiency across all sectors and ensure sustainable withdrawals and supply of freshwater to address water scarcity and substantially reduce the number of people suffering from water scarcity

Evaluation Budget(US $):

20,000

Source of Funding:

Evaluation Expenditure(US $):

21,000

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Philip Tortell

GEF Evaluation:

Key Stakeholders:

Ministry of Environment, Ministry of Energy, CDR

Countries:

LEBANON

Lessons
Findings
1.	3.1 The Energy and Environment Programme 3.1.1Programme design and the Strategic Results Framework There is no programme document for the Energy and Environment Programme and the only image of the programme design is as represented in the SRF as in Table 2 in Section 1.3 above and in Annex 2b which are from the CPD. As commented above, the SRF is somewhat confusing as it seems to lack the usual logical relationship between the different elements of a SRF. In an attempt to clarify the relationship between the levels of the SRF, the Team advised that the following is the structure that has been used for implementation purposes. Tag: Energy Environment Policy Integration Programme/Project Design Results-Based Management
2.	3.2 Programme relevance Programme relevance is assessed at three levels 15 . Firstly, it is assessed for relevance to the Government and people of Lebanon reflecting national needs and priorities; secondly, relevance to UNDP and its corporate commitments; and, finally relevance to the Targets and Indicators set by the CPD Priority Area 4 and UNSF Outcome 3.3. It also needs to be noted that this assessment of relevance is restricted to the 17 projects that have either been initiated, or are pending or are in the hard pipeline since the commencement of this CPD period. In other words, the 19 projects which have been carried over from the previous CPD period cannot be assessed for relevance to something which was designed after they had already started implementation. Tag: Relevance Project and Programme management
3.	3.3 Programme effectiveness Effectiveness is a measure of the extent to which project targets have been achieved and it is best assessed when a project has been completed. This section is therefore based on the 10 projects that have finished 20 (all of which had been carried over from the previous CPD period) and that have been subjected to an evaluation. Tag: Energy Effectiveness Project and Programme management Data and Statistics
4.	3.4 Implementation efficiency Efficiency is a measure of the extent and the competence through which resources have been converted into results. Tag: Efficiency Communication UNDP Regional Bureaux
5.	3.4.2 Personnel, expertise and knowledge sharing The most valuable and tangible resources of the Programme are its personnel, those in the central office and those leading and managing UNDP projects and other initiatives. Also in this category are the implementation partners through whom UNDP achieves its results. The central office serves as the hub for the Programme. This is where the Programme is designed, managed, coordinated and monitored. There is a staff of four – the Programme Manager, a Programme Associate, a Programme Assistant, and a Programme Support JPO. This immediate team is supported by the Country Office administration and support units such as Human Resources, Financial, Legal, etc. Tag: Energy Efficiency Oversight Project and Programme management Country Government UNDP management UNDP Regional Bureaux Operational Services
6.	3.4.3 Financial resources Financial resources are among the most tangible (and measurable) inputs for UNDP work and expenditure is often used as a measure of delivery, overshadowing the more important results achieved. There is only a slight tendency for such an approach with the E&E Programme. Tag: Efficiency Resource mobilization Human and Financial resources Country Government UN Agencies
7.	3.4.4 Partnerships UNDP is an implementing agency but most often it needs executing partners to realize any results; it also relies on donor partners for funding. As with the rest of the UNDP family, the E&E Programme depends almost entirely on executing partners and funding partners to achieve its goals. Worthy of special mention is the partnership that the Programme is developing with the private sector and this is in accordance with the adopted UNDP strategy . Tag: Environment Policy Partnership Country Government Donor UNDP Regional Bureaux Coordination
8.	3.4.5 Monitoring and Evaluation The CPD discussed monitoring and evaluation for the Country Programme and proposed a number of monitoring tools. The evaluator invited the E&E Programme Team to assess the extent to which these tools have been used and how efficient they had been. The results are summarized below. Tag: Monitoring and Evaluation
9.	3.4.6 Risk management The following risks were identified for the whole of the CPD together with possible mitigation measures. The Programme Team was invited to comment on the extent, if any, that the risks did eventuate for the Programme, and if so, whether the mitigation measures worked, and what the impacts on the Programme were. From all indications, it is evident that the Team is recognizing and managing risks adequately. However, it would be an improvement if there was a discussion of risks and mitigation measures focussed on the Programme in a dedicated Programme Document. Tag: Efficiency Risk Management
10.	3.4.7 The gender dimension Gender equality, women’s empowerment and human rights are among the key priorities of UNDP at the corporate level. However, they are best displayed at the projects level which is the delivery mechanism for the Programme. Less than 30% of Project Managers said that their project made a contribution towards gender equality and women’s empowerment. The rest said either that it did not do so or that the question did not apply to their project. This is a disappointing result but not entirely surprising since it is known that while MDG targets in Lebanon have been met in the health and primary education sectors, they were not in poverty, gender equality and environmental sustainability. UNDP has completed a Gender Strategy 37 which found that women’s participation in programme design and implementation contributed to reducing gender inequality; breaking traditional gender roles; increased lobbying for women’s representation and participation in national governance, and a greater number of leadership opportunities for young women. However, the Strategy also found that the Country Office “does not yet make the direct link between gender and environment. This can be explained by the fact that projects are designed to target all Lebanese citizens without distinction. The efforts for gender-mainstreaming at this level do not go further than maintaining a balance between male and female staff, ensuring a lack of discrimination against women, and, in some cases, including a theoretical section on gender”. This shortfall is recognized by the Programme and it is useful to use a very recent project currently awaiting signature as an example of how GEEW is being addressed. This project, which deals with land degradation and land use, has an impressive 10-page Gender Analysis and Mainstreaming Plan in an annex in addition to extensive gender provisions which are integrated in the project document. The Plan identifies gender concerns and opportunities to ensure that the project considers women’s and men’s different vulnerabilities and needs as well as capacities and skills and achieves an equitable distribution of its benefits, resources, status and rights. The Plan also outlines ways for the project to influence transformative changes in the norms, cultural values and the roots of gender inequalities and discriminations. Furthermore the Plan is aligned to other strategies and processes of the project to ensure feasibility and is expected to be mainstreamed as much as possible with on-going national processes and strategies at the time of project implementation and wherever feasible. This is indicative of the manner in which GEEW is fostered by the E&E Programme through its projects. Tag: Efficiency Gender Equality Gender Mainstreaming Women's Empowerment MDGs
11.	4.5 Sustainability of results The likelihood of sustainability for the benefits and outcomes of the projects was seen as very high by Project Managers. In fact, 15 (88%) said there will be sustainability, one said maybe and one said it was not guaranteed. The questionnaire also asked Project Managers if their project had a “champion” who would inherit the project benefits and outcomes. Two did not understand the question, however, the most common champions identified by the rest (76%) were the project stakeholders while a small number (23%) listed the executing partner as the likely champion. Tag: Energy Waste management Water resources Sustainability Human and Financial resources Project and Programme management Civil Societies and NGOs UNDP management UNDP Regional Bureaux Capacity Building
12.	3.1.2 The Programme objectives and targets In the absence of a Programme Document, the E&E Programme objectives are unclear. The UNDP Lebanon website 14 states that “The Environment and Energy (E&E) Programme works closely with the Government of Lebanon to provide policy support for more effective environment and energy management. The programme also works with local communities and civil sector organisations to improve livelihoods through improved natural resource management. Our aim is to move Lebanon towards achieving the Sustainable Development Goals (SDGs) more specifically Goal 7 on environmental sustainability.” This may be an indication of the Programme objectives and targets but it is not explicit. The CPD narrative refers to the analysis carried out and the conclusion reached that what is required is a combination of a “short-term crisis response with long-term initiatives that fall under theumbrella of the new sustainable development agenda”. More specifically, for the environment and energy thematic area, the objective is seen as “to halt the environmental degradation that has been exacerbated by the crisis and negatively influenced Lebanon’s chances of securing long-term sustainable development.” However, the Programme also takes its cue from the UNSF 2017-2020 Core Priority 3: Lebanon reduces poverty and promotes sustainable development while addressing immediate needs in a human rights/ gender-sensitive manner. Under this priority area, the Programme is meant to contribute to Outcome 3.3 which strives for the broad objective of – Lebanon has adopted measures to improve environmental governance. Tag: Environment Policy Partnership Policies & Procedures Policy Advisory SDG Integration Civil Societies and NGOs
13.	3.1.3 The projects portfolio Forming the foundation of the delivery mechanism working towards Outcome 3.3 and the scope for this evaluation, are the projects and other interventions implemented by the Energy and Environment Programme. Table 5 below represents the full list of projects, the major implementing partner, the UNSF and CPD Indicators they are meant to contribute to, and the project status. The portfolio comprises 36 projects. Of these, 19 projects were carried over from the previous CPD period with ten having since been closed and nine still continuing. Some of the projects that have been carried over have been through a number of phases according to the PMs’ Questionnaire (see Annex 6b) and are still continuing. The projects ran/will run for variable lengths of time, not always finishing within the original allocated time. Nine projects have been granted extensions ranging from 3 months to a full 48 months. Six new projects have been initiated in this CPD period and all are still under implementation. In addition, two projects are pending signatures and are expected to start implementation soon. There are also another nine projects in a hard pipeline and some of these may be initiated during this CPD period. Tag: Programme/Project Design Project and Programme management
14.	3.1.4 Programme resources Programme and project resources comprise the inputs required to carry out the programme tasks. They include people, equipment, facilities, funding, knowledge and time. Key resources are discussed in section 3.4 below which assesses the extent to which resources have been converted into results. Tag: Human and Financial resources
15.	3.2.1 Relevance to the Government and people of Lebanon Drafting of the UNSF and the CPD was carried out in or around 2016 following a thorough assessment of the situation in Lebanon and they reflect the identified needs. Although this Common Country Assessment (CCA) was not published on this occasion because of the prevailing situation, it still formed the basis for the UNSF and the CPD. At the Programme level, the assessment is deeper than in the CCA and it arises from the Team’s deep knowledge of the situation and their analysis of the country priorities, their close contact with various arms of the government system, the reports from Project Managers, and discussions with other stakeholders such as key NGOs. The Programme initiated six projects, a further two are pending signature and due to start and there is a hard pipeline of nine projects. The following table lists the total 17 projects initiated or due to start within this CPD period, together with their objectives and brief descriptions, and then assesses their relevance to the Government and people of Lebanon as envisaged at the time of project formulation and since. A full 70% of the projects are considered Very Relevant and 23% are Relevant. Only one project is seen as Moderately Relevant. The projects in the portfolio rightly take their cue from the situation existing at the time, and reflect known government strategies and plans as well as international obligations. Of the Project Managers that responded to the related question in the Questionnaire (see Annex 6b), four appeared confused by the question. A further four saw their project contributing to the NREAP – the National Renewable Energy Action Plan. Other instruments mentioned included strategies/plans on Solid Waste and Water, and the Lebanon Crisis Response Plan. A good measure of Lebanon’s long term priorities are the SDGs (Sustainable Development Goals) and Project Managers, in their responses to the Questionnaire, overwhelmingly said that their respective projects contributed to Lebanon’s progress towards the SDGs 16 . It was ascertained during consultations with MoE and MoEW as well as CDR senior officials and others, that the contribution made by the E&E Programme is essential for a number of key aspects of the Government’s work. It can be concluded that the Programme and its portfolio of projects reflect the situation on the ground, current or immediately past, and are highly relevant to the Government and people of Lebanon. Tag: Relevance Project and Programme management Country Government
16.	3.2.2 Relevance to UNDP The UNDP Strategic Plan 17 current at the time of Programme design identified the following key areas of focus of interest to the Programme – (a)Effective maintenance and protection of natural capital (b)Sustainable access to energy and improved energy efficiency (c)Planning, policy frameworks and institutional capacities to substantially reinforce action on climate change (d)Assessment of key economic, social and environmental risks to the poor and excluded Furthermore, UNDP has identified the following set of approaches that it sees as its Signature Solutions 18 – • Keeping people out of POVERTY • GOVERNANCE for peaceful, just, and inclusive societies • Crisis prevention and increased RESILIENCE •ENVIRONMENT: nature-based solutions for development Clean, affordable ENERGY Women's empowerment and GENDER equality The above have created the framework within which the E&E Programme could be designed and operated. By focussing the Key Areas and Signature Solutions it is possible to lead to seven thematic areas of direct interest to the E&E Programme. These are listed in the Table below together with an indicative matching of the 17 projects initiated during this CPD period. In the above table, the first five thematic areas arise from strategic UNDP documents whereas the Biodiversity, the Ozone and the institutional capacity thematic areas do not. It would seem that seven projects are not directly relevant to the current UNDP strategic priorities, however, 10 projects are. As such, the Programme can be considered as relevant to UNDP and its corporate commitments although slightly less than for the Government and people of Lebanon. This is also borne out by the slight majority of Project Managers (59%) who confirmed this when they asserted that their project design did take into account the UNDP commitments for gender equality, human rights and conflict sensitivity (Annex 6b) thus reiterating the Programme’s relevance to UNDP. Tag: Biodiversity Relevance Programme Synergy Project and Programme management Results-Based Management
17.	3.2.3 Relevance to the CPD and the UNSF Outcome 3.3 The Programme addresses CPD Priority Area 4 with its focus on improving environmental governance. More specifically, it addresses the two CPD Outputs with five Indicators between them. The following table assesses the fit and relevance of the 17 projects that have been initiated or are in pipeline as they were assigned by the Programme Team to one or more CPD Indicators. However, in assessing the relevance of the constituent projects to the Outputs and Indicators of the CPD, and through them to the three Indicators of the UNSF Outcome 3.3, the internal relevance and logic problems of the SRF became more noticeable. The table shows that some of the projects can be said to be relevant to the CPD Indicators and not to the UNSF, others are relevant to the UNSF and not to the CPD. Possible reasons for this discrepancy and the apparent problem with uniform relevance are discussed in section 4.2 below. It is interesting to observe that 10 Project Managers (59%) identified correctly the CPD Output that their project contributed to. However, three PMs did not respond to the question and four provided an uncertain answer. In conclusion it can be said that the Programme is highly relevant to the Government and people of Lebanon. It is also relevant to the UNDP corporate commitments. However, it is less relevant to the CPD and the UNSF Indicators and Targets while being slightly more relevant to the strategic approach of the CPD. This apparent inconsistency appears to be caused by the confused SRF. Tag: Relevance Project and Programme management
18.	3.3.2 Effectiveness according to the Outcome indicators and targets Another way of looking at effectiveness is to assess project delivery of results towards the indicators and targets of the UNSF Outcome 3.3 and CPD Priority Area 4. The Programme Team has assigned each project in the portfolio to one or more Indicator/Target and this is used for the quarterly monitoring of progress. For the purpose of this evaluation, only those projects which have closed are assessed for their final effectiveness, and these all happen to be projects that have been carried over. Some projects are listed more than once since they are deemed to have contributed to more than one UNSF/CPD Target. Terminal evaluations, similar end-of-project reports and one MTR as in the table above are the main sources of information for this assessment. The results recorded above are a mixed bunch. Contributions made by projects towards the CPD Targets do not always lead to similar gains towards the UNSF Targets and as discussed elsewhere and analysed in section 4.1 below, this is the result of the internal lack of logic and relevance between the CPD and the UNSF. The following table, matches the achievements from the above table with the Targets and it can be concluded that the overall level or achievement and effectiveness is reasonable at the mid-term of this cycle. Considering that none of the above projects were designed specifically to deliver the indicators and targets of the current UNSF and CPD, the results show a surprisingly reasonable fit. This is the result of the painstaking effort by the Team to select the broadest and most inclusive wording for the CPD Outputs and Targets to allow existing projects to be accommodated (rather than the other way round). Whether this is justified or has been successful is discussed in section 4.3 below. The overall level of effectiveness of closed and evaluated projects in the E&E Programme is Satisfactory at the project level, but somewhat less certain when assessed against the Targets of the CPD. However, if the Indicators from the UNSF are assumed as truly indicative of what the Outcome 3.3 aimed for, then the conclusion is that the Outcome is very likely to be achieved. Whether this assumption is correct, is also discussed in section 4.3 below. Tag: Effectiveness

Recommendations
1	1.1 UNDP to accept that the larger part of the portfolio’s projects is carried over from previous CPD and that these projects were fully justified and relevant when they were started. 1.2 UN system that in order to resolve the challenge facing programme teams as they try to balance known national needs with diluted UNSF Outcomes, the role of the UNSF should be to set the scene by outlining needs and priorities and focusing on guidance and direction for the various agencies 1.3 It is recommended to the E&E Programme Team that they prepare a Programme Document which should distinguish between projects that are being carried over and those newly initiated in response to new needs and priorities
2	1.4 It is recommended to the E&E Programme Team that all projects must undergo a terminal evaluation before closure so as to assess the relevance of each project to the country needs and priorities as identified in the CPD and the Programme, assess the effective delivery of results compared with the targets sought, assess the efficiency through which the resources available have been converted into results, and assess the likelihood of sustainability of the project’s results
3	1.5 It is recommended to UNDP that in an effort to increase the likelihood of sustainability of project results, projects should preferably fit within a broad and strategic context such as national plans and strategies.
4	1.6 An ex-post assessment should be carried out for each project some time (3-5 years) after project closure to assess sustainability and the circumstances that might have contributed towards or against it.
5	1.7 It is recommended to the E&E Programme Team to continue with and improve the practice of requiring the formulation and implementation of a meaningful gender strategy for each project thus contributing to the achievement of SDG-5 for Gender Equality.
6	1.8 It is recommended to the E&E Programme Team that in view of its current success and the continuing needs, priority should be accorded to interventions that deal with renewable energy, management of wastewater and prevention of pollution, sustainable land management and rehabilitation of degraded land and forests

Energy and Environment Outcome Evaluation

Share

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us