Generate Global Environmental Benefits through Environmental Education and Raising Awareness of Stakeholders Final Evaluation

Report Cover Image
Evaluation Plan:
2016-2020, Armenia
Evaluation Type:
Final Project
Planned End Date:
12/2019
Completion Date:
12/2019
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
15,700

Share

Document Type Language Size Status Downloads
Download document UNDP-G~1.DOC tor English 131.14 KB Posted 635
Download document Final Evaluation_Env. Education.docx report English 1338.57 KB Posted 676
Download document CoC EB .pdf related-document English 113.45 KB Posted 644
Title Generate Global Environmental Benefits through Environmental Education and Raising Awareness of Stakeholders Final Evaluation
Atlas Project Number: 00081939
Evaluation Plan: 2016-2020, Armenia
Evaluation Type: Final Project
Status: Completed
Completion Date: 12/2019
Planned End Date: 12/2019
Management Response: Yes
UNDP Signature Solution:
  • 1. Sustainable
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.4.1 Gender-responsive legal and regulatory frameworks, policies and institutions strengthened, and solutions adopted, to address conservation, sustainable use and equitable benefit sharing of natural resources, in line with international conventions and national legislation
SDG Goal
  • Goal 12. Ensure sustainable consumption and production patterns
  • Goal 13. Take urgent action to combat climate change and its impacts
SDG Target
  • 12.4 By 2020, achieve the environmentally sound management of chemicals and all wastes throughout their life cycle, in accordance with agreed international frameworks, and significantly reduce their release to air, water and soil in order to minimize their adverse impacts on human health and the environment
  • 13.2 Integrate climate change measures into national policies, strategies and planning
Evaluation Budget(US $): 15,700
Source of Funding:
Evaluation Expenditure(US $): 15,700
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
unknown to QA
GEF Evaluation: Yes
GEF Project Title: Generate Global Environmental Benefits through Environmental Education and Raising Awareness of Stakeholders Terminal Evaluation
Evaluation Type: Terminal Evaluation
Focal Area: Multifocal Areas
Project Type: MSP
GEF Phase: GEF-5
GEF Project ID: 5716
PIMS Number: 5309
Key Stakeholders:
Countries: ARMENIA
Lessons
Findings
1.

3. FINDINGS

While the amount of information generated by this evaluation was large, the findings presented in this chapter cover only the most essential aspects of the project and are to some extent focused on those issues and lessons that provide a better understanding of the achievements of the project and which would benefit the project stakeholders the most in similar future endeavours. The findings of this evaluation are organized in the following sections: i) Project Design; ii) Project Implementation; and, iii) Project Results.

3.1. Project Design This section examines the project's logic and design features by focusing on the adequacy of elements like the project's logic, results framework, management arrangements, identification of risks and assumptions, use of lessons derived from other projects, linkages with relevant UNDP or donor projects, UNDP?s comparative advantage in the area, planned stakeholder engagement, replication approach and exit strategies, etc. The main questions that have driven the analysis presented in this section are shown in Box 2 below.


Tag: Environment Policy Relevance Civic Engagement Local Governance Communication Integration Programme/Project Design Results-Based Management Theory of Change Country Government Education Advocacy Capacity Building

2.

3.1.1. Analysis of the Project Logic and Planning Matrix (continuation)

Legal, Strategic & Policy Frameworks Under Outcome 1, the project was expected to contribute to the integration of sustainable development and environmental concerns into strategic, policy and legal frameworks. This is a very important part of the project's activities. And actually, the project contributed with crucial inputs, such as the development of the "National Strategy on Environmental Education and Population Upbringing" approved by the Government of Armenia in February 2018, which promotes the mainstreaming of environmental considerations into the national educational framework. However, the focus of the project document is on the integration of environmental concerns into policy and legal framework and less attention is paid to the crucial issue of how these instruments get implemented. One problem that was raised by stakeholders during interviews for this evaluation was the weak implementation of the legal and policy framework, especially in the environmental sector. Integration of environmental concerns is necessary, but not sufficient for ensuring that in practice those concerns get the right degree of attention. This is an area that could have received more attention in the project document. Furthermore, implementation of environmental policy does not only require the integration of environmental concerns into strategies and policies but also subsequently into budget allocations. Plans and policies with no financial tags attached to them have no teeth. Therefore, integrating the principles and actions articulated in the Rio Conventions into routine development activities requires that commitments be made not only to environmental objectives but also cross-sectoral priorities – and in particular financing – which were developed precisely because the environmental objectives are unachievable in their absence. This requires a far greater commitment from ministries responsible for finance and planning and sectoral departments which control the bulk of financial resources and public investments. If the work of projects like this one is to have significant impact, public sector financial management and governance will need to improve. With hindsight, we know, for example, that the Ministry of Finance did not play a major role in project activities, and to a large extent, this was a result of a design that did not give it a central role in the project. Had the role of the Ministry of Finance (and others) been crafted more adequately, giving it not only a more important place in the project, but also organizing project activities more intensively around public financial management issues, the results of the project would have been more sustainable.


Tag: Relevance Civic Engagement Knowledge management Programme/Project Design Results-Based Management Theory of Change Country Government Education Advocacy SDG Integration Private Sector

3.

3.1.2. Assumptions and Risks

The risks identified in the Project Document are presented in the following table.

The identification of risks in the project document is quite comprehensive and has been conducted adequately. A number of identified risks materialized in the course of the project's lifetime. The most important risk that occurred was the so called “2018 velvet revolution”, which resulted in the establishment of a new government and major changes in the administration. Most of the government bodies that were crucial for the EEP project underwent different degrees of restructuring following the revolution. This included the Ministry of Education and Science,12 Ministry of Education, Ministry of Territorial Administration and Infrastructure, 13 and the Civil Service Office. Also, the risk defined as “institutional risks associated with poor coordination among institutional stakeholders at the national level” had an effect on the project, as the institutional environment in which the project has operated has been quite complex. Another identified risk that has played a role in the project is the one defined as “new legislation proposed by the project is not adopted by the Government and/or the Parliament”. The parliamentary approval of the Document could have addressed more carefully is a more specific formulation of the mitigating measures in response to each of the identified risks. Another risk that does not receive due attention in the analysis presented in the project document, but which did have a significant impact on the project, is the scarcity of field-specific specialists and professionals. This hindered the timely procurement of services, resulting in shifts in planned procurement modalities, re-announcement of bids, tender deadline extensions etc. This was also the reason for the delay in the start-up of the project – it took time to recruit the Project Manager (Project Coordinator) because of limited availability of professionals in the local market and the long time it took for the identification of the specialist with the required knowledge and skills.


Tag: Global Environment Facility fund Partnership Programme Synergy Programme/Project Design Results-Based Management Risk Management Strategic Positioning Coordination

4.

3.1.7. Replication Approach

As for the replication approach, the Project Document is not very explicit about what exactly is meant by replication, what aspects of the project are intended to be replicated, and how replication is supposed to occur. The project document stated that the EEP project would directly address national priorities identified through the NCSA process and confirmed through the National Electrical Annuity Plan (NEAP) and the Rio+20 National Assessment Report and that it was not about piloting or demonstrating a new approach or a new system. The project document recognized that the project's achievements should be sustained after the project end and that Armenia should have access to a greater capacity to implement environmental education and environmental awareness programmes for the implementation of the Rio Conventions. The project document also recognized that as a medium-size project this intervention was bound to have certain limitations, such as the capacity to develop skills and knowledge of all actors involved in environmental management. Based on the logic of the project document, the EEP project was to serve as a catalyst of a longer-term approach to Rio Convention implementation by improving the capacity of national actors to deliver environmental education and environmental awareness activities and by providing an enabling environment to sustain the delivery of these activities throughout the country, thereby contributing to the implementation of the Rio Conventions.


Tag: Biodiversity Environment Policy Effectiveness Relevance Local Governance Communication Knowledge management Oversight Project and Programme management Country Government Education Advocacy Civil Societies and NGOs

5.

3.2. Project Implementation

During the four years of its lifetime, the project went through a number of important stages. The following is the chronology of key events that marked the project?s conceptualization and implementation phases, spanning the 2015-2019 period.

At the request of the government, UNDP Armenia started preparation of the “project initiation format” that was approved and submitted to GEF with a letter of the Minister of Nature Protection (GEF focal point) on 28 December 2013. GEF CEO approval – 25 March 2014 Project Preparation Grant (PPG) agreement signature -18 April 2014 PPG preparation process – June 2014 - March 2015 Date of LPAC – 7 June 2015 ? Date of signature of Delegation of Authority (DoA) – 7 July 2015 Date of signature of Pro Doc – 3 November 2015 Date of recruitment of Project Manager (Project Coordinator) – 1 August 2016 Date of Inception Workshop – 24 June 2016 Date of release of Inception Report - December 2016 Dates of Board Meetings: - 7 November 2016 - 28 September 2017 - 27 December 2017 - 14 February 2019 Dates of Advisory Committee meetings - 12 September 2017 - 29 June 2018 Date of stakeholders meeting - 23-24 November 2016 Mi-Term Review – 24 November – 1 December 2017 Final Evaluation – October-November 2019 Project Closure – set for November 2019 The box below provides a summary of the process for the development of the project document.

The project experienced some delays during the implementation process due to external factors with which the project team had to grapple. First, the project kick-off experienced a half-year delay because of the late recruitment of the Project Manager, which was due to the limited availability of professionals in this area. The identification of the specialist with required knowledge and skills took time. As a result, the project, which was due to start on November 2015 based on the project document, actually started in the middle of 2016. Secondly, delays also resulted from the severe political crisis in 2017/2018, which ended with the “velvet revolution”. Some project components suffered from this event. One key component was the training for civil servants, for which the partnership with the Civil Service Office was crucial. As discussed in the previous sections of this report, the establishment of the environmental training programme within the official structures of the government was key for ensuring its sustainability. The civil service reform process that was underway during the first part of the project was expected to be completed by the end of 2018. However, the resignation of the government in April/May 2018, following the revolution, put on hold the whole process. 


Tag: Global Environment Facility fund Human and Financial resources Project and Programme management Capacity Building

6.

3.2.1. Adaptive Management

Given the political changes that Armenia experienced during the period of project implementation, the use of adaptive management by the project team has been crucial for dealing with some of the unexpected contingencies and taking advantage of emerging opportunities. While a number of adaptive strategies and actions employed by the project team were observed during the evaluation, this section will focus on those adaptations that played the biggest role. 

Political Volatility – As has been described, aa a result of the 2018 velvet revolution which ousted the previous administration and brought significant changes to the country?s political landscape and policy agenda, the project experienced a number of delays in project implementation and delivery. Given the policy nature of the project, this process of change has clearly exerted a delaying effect on project activities. For example, the co-chair of the Project Board from the Ministry of Environment has changed three times during the project?s lifetime. Further, the freezing of the civil service reform process put on hold project activities related to the training of public officials. The project team has responded to these challenges by taking measures to minimize the effect of political change and address implementation delays. It adjusted planned activities, applied various procurement modalities to identify the best match for the required services, etc. Other measures have included the re-scheduling of project activities, close monitoring of ongoing political and institutional reforms, close cooperation with key partners, etc. In spite of the delays experienced, the project was able to accomplish all planned activities within the established timelines (this will be discussed in more detail in the section on “effectiveness” of this report).


Tag: Biodiversity Global Environment Facility fund Knowledge management Partnership Country Government Education

7.

3.2.4. Project Finance

This section of the report provides a brief overview of the project's financing and expenditures, based on information provided by the project team.

Project Expenditure The project budget was revised three times to adjust the budget to official expenditures and to redistribute funds between 2018 and 2019 in line with the approved no-cost extension till November 2019. This is another good example of adaptive management by the project team. The table below shows project expenditures by outcome area for the four years of operation. As can be seen from the table below, by the time of this evaluation the project had spent a total of about US$ 715,000, or about 93% of what was budgeted for the project. This total amount spent represents about 95% of the total funding provided by GEF for this project (US$ 750,000). The project team has planned the full expending of all resources available to the project. Further, as can be seen from the table above, the project started with a slow execution rate in 2016, but subsequently the pace of implementation accelerated in the following three years. The table also shows that Outcome 2, which includes the training for civil and community servants has been the project's largest component, taking up about 35% of total expenditure. Outcome 3, which contains the micro capital grants, has also been a large component with about 30% of total spending. Project management costs have constituted about 10% of total expenditure, which is a good indicator of efficiency.

Budget Execution Table 6 below shows the project's execution rates for each year and for all years together based on planned budgets planned. Two versions of the budget are shown – one version compares expenditure to what was planned originally in the project document and the other compares expenditure to what was planned after the revision of the budget. This allows us to see the execution rates based on the old and the revised budget. Execution rates are also shown by component. The table shows that the rate of fund utilization has varied though by year and component. Years 2016 and 2017 have had quite low execution rates compared to the original budget, mainly as a result of delays discussed in the previous sections. For year 2017, execution was low even when compared to the revised budget and the main reason for this is weak execution of the third component (the micro-capital schemes). A much better execution rate can be noted in the table below during the last couple of years of implementation. Execution for the third component picks up in 2018, with more than US$ 120,000 spent on the micro-grant initiatives. Overall, the year 2018 was a turn-around year for the project, with total expenditure reaching about US$ 280,000 and execution 75%. Overall, all three project components have had execution rates above 90% for the whole duration of the project (based on the revised budget). The project team has planned to utilize all project resources by the end of the project. 


Tag: Operational Efficiency Efficiency Global Environment Facility fund Government Cost-sharing

8.

3.2.5. Monitoring and Evaluation

Design at Entry

Overall, the M&E tools identified in the Project Document have been appropriate and have included standard instruments used in UNDP projects. The most foundational M&E tool of the project, the Results and Resources Framework, has been simple and straightforward, as was pointed out under the discussion of the project's design in the previous sections. Planned activities, outputs and associated costs are discussed in a detailed manner. Also, most indicators, baselines and targets are generally adequate and well-identified. The exception are some indicators that could have been identified more specifically. Previous sections of this report also identified some additional shortcomings with regard to the project's design. The design of the Monitoring and Evaluation provided in the Project Document comprises the standard tools used in most UNDP projects in accordance with established UNDP and GEF procedures. The table below summarizes these tools, as outlined in the Project Document. While the monitoring and evaluation tools laid out in the Project Document were adequate in maintaining quality control, the project design suffered from some shortcomings that have been pointed out. Further, as has been stated in the previous sections of this report, the project design was quite ambitious for the nature of this project and the resources and timelines that it involved. Overall, the rating of the Monitoring and Evaluation design at the entry point is “Moderately Satisfactory”.


Tag: Effectiveness Global Environment Facility fund Monitoring and Evaluation Programme/Project Design Project and Programme management Results-Based Management

9.

3.2.6. Execution and Implementation

Performance of the Executing Agency (MoE)

As the Implementing Partner of this project, MoE was foreseen in the Project Document to be directly responsible for the execution of the project. Although not a high-visibility ministry and with limited human resource capacities, MoE has delivered this responsibility adequately. The project has received the right degree of attention from the ministry officials. The project manager has had close contacts with MoE and has received substantial support from MoE on all the components of the project. The Deputy Minister has been directly involved with the project and has served as the co-chair of the Project Board, attending board meetings regularly. In interviews for this evaluation, the Deputy Minister and other ministry staff were intimately informed about the activities of the project and quite eager to make this project yield all the expected results. Further, stakeholders interviewed for this evaluation confirmed that MoE has had a close engagement with the project. 

Given the positive role that MoE has played in ensuring that the project achieved all its targets, despite all its constraints and the challenged presented by the political environment, the rating for the Executing Agency in this project is “Satisfactory”. 


Tag: Implementation Modality Policies & Procedures Project and Programme management Strategic Positioning Country Government Coordination

10.

3.3. Project Results

This section provides an assessment of the project?s progress in the accomplishment of RRF targets, as well as examination of achievement along the standard dimensions of UNDP evaluations: i) relevance - the extent to which the project was relevant to the country?s priorities and needs; ii) effectiveness - whether the project was effective in achieving the planned results; iii) efficiency - whether the process of achieving results was efficient; iv) sustainability - the extent to which project benefits are likely to be sustained; and, v) gender mainstreaming – the extent to which considerations related to gender have been incorporated into project activities.


Tag: Relevance Global Environment Facility fund Government Cost-sharing Public administration reform Results-Based Management Capacity Building Civil Societies and NGOs

11.

3.3.3. Effectiveness

Effectiveness in the context of this assessment means the extent to which the project achieved what it planned to achieve at the outset. This section provides a brief overview of the project's main achievements along with the three outcome areas identified in the project document.

Outcome 1: Legal, policy, institutional and strategic frameworks

One of the most important achievements of the project in this area is the development and approval by the government of the "National Strategy on Environmental Education and Population Upbringing" through which the mainstreaming of environmental concerns is expected to take place into national education system. Another important aspect of the project is the development of the package on legal instruments/amendments related to Environmental Education (the content of the package is shown in Table 11 below). This package has been developed and submitted to the Ministry of Environment for approval. Further, the project has worked closely with the National Assembly in developing two keystone pieces of legislation on environmental policy and education (draft "Law on Ecological Information" and draft "Law on Ecological Policy"). The two drafts are under discussion in the Parliament and are expected to receive approval soon. The achievement of targets for Outcome 1 (legal and institutional changes) has been the most challenging for the project, as the approval of legal documents and adoption of laws, policies and strategic documents by the government has taken quite a long time. Also, the significant changes in the administration due to political volatility have played a complicating role. The project team has learned that it is not realistic to complete such an ambitious agenda within the short lifespan of a project like EEP. 


Tag: Effectiveness Knowledge management Policies & Procedures Education Micro-credit Capacity Building Civil Societies and NGOs

12.

3.3.4. Efficiency

To assess efficiency, the report focuses on a number of parameters that are closely associated with efficient project management. These parameters are categorized into the following categories: i) budget execution rates; ii) timeliness of project activities; and, iii) synergies and linkages with other similar initiatives in the country. 

Expenditure and Budget Execution Rates Table 17 below shows project expenditures by the outcome area for the four years of operation. As can be seen from the table, by the time of this evaluation the project had spent a total of about US$ 715,000, or about 93% of what was budgeted for the project. This total amount spent represents about 95% of the total funding provided by GEF for this project (US$ 750,000). The project team has planned to expend all available resources by the end of the project. Further, the project started with a slow execution rate in 2016, but subsequently the pace of implementation accelerated in the following three years. The table also shows that Outcome 2, which includes the training for civil and community servants has been the project's largest component, taking up about 35% of total expenditure. Outcome 3, which includes the micro capital grants, has also been a large component with about 30% of total expenditure. Project management costs have constituted about 10% of total expenditure, which is a good indicator of efficiency. Table 18 below shows the project's execution rates for each year and for all years together based on planned budgets planned. Two versions of the budget are shown – one version compares expenditure to what was planned originally in the project document and the other compares expenditure to what was planned after the revision of the budget. This allows us to see the execution rates based on the old and the revised budget. Execution rates are also shown by component. The table shows that the rate of fund utilization has varied though by year and component. Years 2016 and 2017 have had quite low execution rates compared to the original budget, mainly as a result of delays discussed in the previous sections. For the year 2017, the execution was low even when compared to the revised budget and the main reason for this is the weak execution of the third component (the micro-capital schemes). A much better execution rate can be noted in the table below during the last couple of years of implementation. Execution for the third component picks up in 2018, with more than US$ 120,000 spent on this outcome (of which about US$ 33,000 spent on micro capital grants). Overall, the year 2018 was a turn-around year for the project, with total expenditure reaching about US$ 280,000 and execution 75%. Overall, all three project components have had execution rates above 90% for the whole duration of the project (based on the revised budget).


Tag: Efficiency Gender Equality Gender Mainstreaming Partnership Programme Synergy Capacity Building Data and Statistics Civil Societies and NGOs

13.

3.3.5. Sustainability

While the sustainability of project outcomes is shaped by a number of factors, the focus of this section is on risks related to financial, sociopolitical, institutional, and environmental sustainability of project outcomes. 

Financial resources

Although the main objective pursued by this project is the promotion of environmental education and awareness-raising, the ultimate goal is the mainstreaming of environmental concerns into national strategies and policies. This is the most sustainable way of dealing with environmental issues. However, mainstreaming does not only mean integrating environmental concerns into strategies and policies, but also subsequently into budget allocations. Plans and policies with no financial tags attached to them have no teeth. Therefore, integrating the principles and actions articulated in the Rio Conventions into routine development activities requires that commitments be made not only to environmental objectives, but also cross-sectoral priorities – and in particular financing – which were developed precisely because the environmental objectives are unachievable in their absence. This requires far greater commitment from ministries responsible for finance and planning and sectoral departments, which control the bulk of financial resources and public investments. If the work of projects like this one, and ultimately the Conventions, are to have a significant impact, public sector financial management and governance will need to improve. With hindsight, we know, for example, that the Ministry of Finance did not play a major role in project activities, and to a large extent, this was a result of a design that did not give it a central role in the project. Had the role of the Ministry of Finance (and others) been crafted more adequately, giving it not only a more important place in the project, but also organizing project activities more intensively around public financial management issues, the results of the project would have been more sustainable. As far as support from international organizations on aspects related to capacity development is concerned, there is a sufficient degree of interest for this kind of work to ensure that the necessary amount of financing will be available. This will also depend on UNDP's continued engagement in this area and its ability to position itself as a natural leader for this type of work. Given the above-mentioned, the likelihood of sustainability of the project's outcomes from a financial perspective is rated as “Moderately Likely”


Tag: Sustainability Operational Efficiency Ownership Project and Programme management

14.

Continues from Finding 3

3.1.4. Linkages to other Relevant Projects in the UNDP Portfolio

Although UNDP has always had an active involvement in the environmental sector, the Project Document does not examine potential linkages that could have been forged between the EEP project and other relevant UNDP projects under implementation or in the pipeline. In effect, during the implementation stage, the EEP project collaborated with the following UNDP projects: - UNDP-GEF Mainstreaming Sustainable Land and Forest Management in Mountain Landscapes of North-Eastern Armenia. - UNDP-RTF “Regulatory Framework to Promote Energy Efficiency in Countries of the Eurasian Economic Union” - UNDP-GEF “Development of Armenia's Fourth National Communication to the UNFCCC and Second Biennial Update Report” Projects. Through its analysis of the intervention and contextual factors, the project document could have provided a more structured framework for cooperation between the EEP project and ongoing UNDP activities in this area.


Tag: Climate Change Adaptation Climate change governance Environment Policy Relevance Partnership Programme Synergy Project and Programme management Strategic Positioning

15.

Continues from Finding 4

- Environmental training for civil servants – How to make the content prepared by this project part and parcel of the obligatory and optional training programme for all civil servants? What are the challenges that the responsible organization (the Civil Service Office) faces in this regard?; - Environmental training for community servants – How to integrate the content developed by this project permanently into the training programme developed and managed by the MoTAI? - Environmental Education Curricula – How to integrate the environmental dimension into the standard education curricula at the national level? Given the above-mentioned issues, the replication approach and the broader issue of sustainability would have benefited from better definitions and more in-depth analysis.

3.1.8. Management arrangements The project was designed to be implemented under the national implementation (NIM) modality, but also including Direct Project Services (DPS) provided by UNDP, specific to project inputs. The project's organizational structure foreseen in the project document is shown in the organogram shown in the figure below. The Project Document foresaw the following organizational arrangements:

Project Board (PB) - Responsible for making consensus-based decisions, in particular, when guidance is required by the Project Coordinator (PC). The Board was envisaged to play a critical role in project monitoring and evaluations by assuring the quality of these processes and associated products, and by using evaluations for improving performance, accountability and learning. The Project Board was to ensure that required resources are committed. It was to also arbitrate on any conflicts within the project and negotiate solutions to any problems with external bodies.


Tag: Relevance Sustainability Human and Financial resources Knowledge management Oversight Project and Programme management Institutional Strengthening National Institutions

16.

Continues from Finding 6

Online Delivery of Training – The online delivery of training was another adaptive solution that the project found for the problem of effective delivery of the training content developed by the project. The development of online (distant) learning courses was launched based on the new requirement of training for civil servants.; Training of Trainers (ToT) – The project?s focus on the training of trainers was another adaptive measure of the project. The scarcity of professional facilitators/trainers in environmental education reinforced the importance of organizing additional capacity building programmes for trainers in higher educational entities involved in natural resource management. Given such lack of field specialists teaching environmental education, the project decided to focus on training for selected professors and lecturers. With the involvement of the American University of Armenia, the project developed 11 modules, which formed the basis of the Training of Trainers. Overall, the ability of the project team to act swiftly to evolving needs and emerging opportunities is commendable.

3.2.2. Partnership Arrangements

The project's partnership arrangements included a wide range of stakeholders from national and sub-national governments, organizations representing the development and environmental professionals and specialists, academia, NGOs and donor organizations. The project was implemented by the Ministry of Environment (formerly the Ministry of Nature Protection), following UNDP's National Implementation Modality (NIM). The Ministry of Education, Science, Culture and Sport (formerly Ministry of Education and Science) was designated in the project document as the primary beneficiary of the project, based on its mandate of addressing educational policy in the country, including environmental education. However, during implementation, the partnership arrangements turned out to some extent differently from how they were envisaged in the project document. What turned out to have required more attention in the project document was the role of other stakeholders that have not been mentioned, but which eventually played an important role in the project. These included the Parliament, Ministry of Territorial Administration and Infrastructure, Public Administration Academy, Civil Service Office, a number of sub-national government authorities, the American University of Armenia, etc. The figure below shows the project's key components and the stakeholders that played a key role in each component.'


Tag: Effectiveness Efficiency Knowledge management Monitoring and Evaluation Partnership Project and Programme management Results-Based Management Capacity Building

17.

3.1.1. Analysis of the Project Logic and Planning Matrix (continuation from Finding 1 and Finding 2)

Environmental Education Curricula Another matter that would have benefitted from greater foresight in the project document is the sustainability of environmental education in the general education system. The integration of environmental training in the educational curricula on a permanent basis is the most effective way of ensuring continuity and large-scale impact in this dimension. Again, this is an area in which the project worked closely with the Ministry of Education and Science, but the Project Document itself provided limited guidance on the matter of sustainability. 

Results and Resources Framework The Results and Resources Framework (RRF), which is underpinned by three outcomes and seven outputs, as shown in Annex V, has been revised once. During the first Project Board meeting in November 2016, a number of questions were raised, inter alia, on the nature and suitability of the indicators included in the logical framework. An international consultant was hired to, among other things, review the output indicators, which resulted in a revised framework. Overall, this framework is simple and straightforward. Planned activities, outputs and associated costs are discussed in a detailed manner. Also, most indicators, baselines and targets are generally adequate and well-identified. However, as the project's Mid-Term Review noted, some of the indicators are not SMART – specific, measurable, achievable, realistic and timely. For example, the following outcome and output indicators are not useful because they are not defined in clear terms and are not directly measurable. More clarity would have been useful on how exactly these indicators are to be understood and interpreted. - Use of EE and environmental awareness tools to address NRM; - Citizens involvement in decision-making to address NRM issues; - Adequate institutional set-up with a clear mandate to carry out EE activities. 

Engagement of the Private Sector Another design weakness is the insufficient linkages between the mainstreaming of the conventions and the crucial role of the private sector in sustainable development. The private sector plays an important role in ensuring that economic activity does not go against environmental and social concerns. The private sector can also be involved as an actor that contributes with insights and resources to the solution of systemic sustainable development problems. While some engagement of the private sector took place in the training activities undertaken by the project, in the project document and planned activities, the role of the private sector is quite limited.


Tag: Knowledge management Programme/Project Design Results-Based Management Theory of Change Capacity Building SDG Integration National Institutions Private Sector

18.

3.3.2. Relevance (Continuation from Finding 10)

This section provides an assessment of the relevance of the project. While there may be many criteria for assessing relevance, here it will be assessed along the following dimensions: i) relevance to the country's needs and priorities; ii) relevance to country's international commitments; and, iii) relevance to UN Country Priorities and UNDPs Country Mandate and Strategy. 

Relevance to the country’s needs and priorities The project was designed at the request of the government, with the involvement of the respective key ministries. The importance of the environmental education and awareness for the government is emphasized in various policy documents, including sectoral strategies. The significance of environmental education and awareness for the government is further corroborated by the approval of the National Strategy on Ecological Education in 2018, which was developed with the support of the EEP project. The project has addressed the critical capacity needed to raise environmental literacy in Armenia by strengthening the capacity of national institutions to deliver environmental education programmes. This need was first identified in the NCSA conducted in 2003-2004 and confirmed subsequently by the assessment conducted for the Rio+20 National Assessment Report. The need for action to address this need was also included in the 2014-2025 Prospective Development Strategic Programme. 


Tag: Environment Policy Relevance Knowledge management Policies & Procedures Programme/Project Design Awareness raising Institutional Strengthening National Institutions

19.

3.3.3. Effectiveness (continuation from Finding 11)

Outcome 3: Capacity of community-based organizations

As has been mentioned, the project awarded three micro capital grants to local CBOs through which more than 600 decision-makers were trained on environmental education issues. Through “Environmental Education Challenge” call, three regional CBOs were selected to receive financial support under the micro-capital grant modality to implement Environmental Education and public awareness-raising projects. As a result, around 600 representatives from local communities and decision makers participated in workshops, conducted by the selected CBOs. Overall, five manuals and educational booklets were developed. Further, six environmental proposals were developed and presented to governmental entities and local self-governing bodies for further support for implementation. The project also supported the introduction of the Climate Box tool into the education system. The project supported the translation of the manual into Armenian and its adaptation to local settings, the organization of the first Training of Trainers for public school teachers and methodologists, and the hosting of the first international conference on “addressing climate change through education” with the participation of representatives from eight countries.The project also organized in March 2019 a National Contest on environmental initiatives designed by schoolchildren aged 12-17.


Tag: Climate Change Adaptation Environment Policy Effectiveness Knowledge management Education Technology Capacity Building Data and Statistics Displaced People

20.

3.3.5. Sustainability (Continuation from Finding 13)

Institutional framework and governance The project's sustainability from a governance and institutional perspective is related to the likelihood that mainstreaming outcomes will be sustained beyond the project's completion and will eventually contribute to environmental sustainability. It is also important that national institutions retain or enhance their capacity to respond to unmet demand for mainstreaming after the project has ceased to exist. The project has developed institutional mechanisms to retain the sustainability of results. This has been primarily achieved by developing laws and strategies that create obligations for the government to promote environmental education and awareness-raising. The most important example of this is the development of the "National Strategy on Environmental Education and Population Upbringing" approved by the government in February 2018. This strategy promotes the mainstreaming of environmental concerns into the national educational framework. Also, the set of draft laws developed in cooperation with the Parliament, and which are still pending approval, create incentives for the mainstreaming of environmental concerns into the policy making process. But, as has been discussed in previous sections of this report, what could have received greater attention by the project here is the implementation of the laws and strategies. It is one thing to have these instruments on paper, and it is another implementing them in reality. As mentioned in the previous section, one key requirement for implementation is that financing is secured in a sustainable fashion by institutionalizing sustainability concerns in public budget allocations. Another example of sustainability in this project is the attention that the project paid to the importance of the Training of Trainers. Given the lack of specialists teaching environmental-related matters, the project identified the need to organize more trainings of trainers. This decision has strengthened the prospects of sustainability because the project has contributed to the creation of a cadre of better-qualified trainers in the country.

For all these positive aspects of the project, one area that would have required a more sustainable approach is the training for public approaches. As has already been mentioned in this report, the project document does not identify in clear terms a sustainable platform for the delivery of this training that would continue to exist in the long-run, way after the end of the project's lifetime. Ideally, the mechanism for the delivery of this training content should have been conceived in the project document and should have built on existing structures. Firmly integrating the environmental training on a permanent basis into the mechanisms used by the Civil Service Office and the Ministry of Territorial Administration and Infrastructure for the delivery of mandatory or elective training programmes for the public administration would have strengthened the sustainability of the results of the project. This is a challenging task because the issue of training for public officials is linked to the reform of the civil service system in the country which is a process that has been ongoing for many years and that has faced many challenges, given the political sensitivities that it involves. Ideally, the civil service reform should address the effective provision of training for public officials, which should include in itself training on environmental matters as one of many topics covered by the training programme.


Tag: Environment Policy Sustainability Public administration reform Knowledge management Policies & Procedures Institutional Strengthening National Institutions

Recommendations
1

Project stakeholders should consider options for how to make the results of this project more permanent by integrating the training content that has been produced into the official training package that is provided to public officials by the Civil Service Office, the Ministry of Territorial Administration and Infrastructure, line ministries, and local government authorities (including institutions of self-government at the sub-national level). 

Further, stakeholders should follow up on the issue of education curricula and see to what extent it will be possible to integrate the Climate Box content into the official nation-wide education curricula for general education. 

The project document mentioned the development of an exit strategy. It might be a good idea for the project team to develop an exit strategy before the final closure of the project in which they can identify options for ensuring the sustainability of the project’s components and products, some of which have been highlighted in this report.

2

In future interventions that involve intensive training components such as this project, UNDP and stakeholders are advised to pay greater attention to the measurement of the quality of training, and more importantly the absorption of the training content by the participants. This requires that two things are put in place: first, a feedback system for collecting the participants’ assessment of the training received and tracking the trainees over time to understand to what extent they are using the concepts and skills gained through the training programme. This will enable the providers of training to understand how best to tailor training programme, so that they can have a real impact over time.

3

In future projects that target awareness raising for specific target groups or the general population, UNDP and the project stakeholders should take a more strategic approach in the design of the interventions by incorporating behavioral insights and elements of social psychology that focus not only on the information that is shared, but also on the instruments, channels and techniques that are utilized for sharing that information. These efforts should be driven by the primary goal of changing behavior and not just raising the level of awareness.

Management Response Documents
1. Recommendation:

Project stakeholders should consider options for how to make the results of this project more permanent by integrating the training content that has been produced into the official training package that is provided to public officials by the Civil Service Office, the Ministry of Territorial Administration and Infrastructure, line ministries, and local government authorities (including institutions of self-government at the sub-national level). 

Further, stakeholders should follow up on the issue of education curricula and see to what extent it will be possible to integrate the Climate Box content into the official nation-wide education curricula for general education. 

The project document mentioned the development of an exit strategy. It might be a good idea for the project team to develop an exit strategy before the final closure of the project in which they can identify options for ensuring the sustainability of the project’s components and products, some of which have been highlighted in this report.

Management Response: [Added: 2019/12/28] [Last Updated: 2020/11/20]

The recommendation is relevant and acceptable.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Online training is incorporated into the training platform of the Civil Service office (http://elearning.cso.gov.am/course/index.php?categoryid=1) and will be utilized as part of training for civil servants. The Ministry of Territorial Administration and Infrastructure expressed readiness and interest for integrating the developed educational materials into the training system for community servants and the staff of the Ministry. In addition, the developed materials are available for usage by other UNDP projects. The Climate Box content is planned to pilot by June 2020 in the selected schools. The project Exit Strategy will be developed and discussed during the Project Board and Validation Workshop.
[Added: 2019/12/28] [Last Updated: 2020/12/23]
Implementing partners 2020/12 Completed as planned History
2. Recommendation:

In future interventions that involve intensive training components such as this project, UNDP and stakeholders are advised to pay greater attention to the measurement of the quality of training, and more importantly the absorption of the training content by the participants. This requires that two things are put in place: first, a feedback system for collecting the participants’ assessment of the training received and tracking the trainees over time to understand to what extent they are using the concepts and skills gained through the training programme. This will enable the providers of training to understand how best to tailor training programme, so that they can have a real impact over time.

Management Response: [Added: 2019/12/28] [Last Updated: 2020/11/20]

The recommendation is relevant and acceptable.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Public Administration Academy of Armenia (PAARA), being the main implementing partner for the training of 1,000 decision makers, ensured that relevant feedback mechanism for collecting the participants’ assessment of the training is in place. Pre/Post training assessment was collected and analyzed. Moreover, preliminary agreement is achieved with PAARA to conduct impact measurement of the delivered trainings by monitoring and post-training impact reporting mechanism for the period of one year.
[Added: 2019/12/28] [Last Updated: 2020/12/23]
UNDP; Public Administration Academy of Armenia 2020/12 Completed as planned History
3. Recommendation:

In future projects that target awareness raising for specific target groups or the general population, UNDP and the project stakeholders should take a more strategic approach in the design of the interventions by incorporating behavioral insights and elements of social psychology that focus not only on the information that is shared, but also on the instruments, channels and techniques that are utilized for sharing that information. These efforts should be driven by the primary goal of changing behavior and not just raising the level of awareness.

Management Response: [Added: 2019/12/28] [Last Updated: 2020/11/20]

The recommendation is relevant and acceptable.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
The recommendation will be considered by UNDP in the design and development of future projects, specifically those with components related to the capacity building and awareness raising. The project’s design will ensure that impact measurement and management, including efforts driven by the primary goal of changing behavior is well articulated and incorporated into the Theory of Change (ToC).
[Added: 2019/12/28] [Last Updated: 2020/12/23]
UNDP 2020/12 Completed as planned History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org