Final project evaluation for the “Tackling Air Pollution in the City of Skopje”

Report Cover Image
Evaluation Plan:
2021-2025, North Macedonia
Evaluation Type:
Final Project
Planned End Date:
07/2022
Completion Date:
07/2022
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document TOR_International Evaluator SIDA_20Apr2022.docx tor English 89.82 KB Posted 28
Download document Air Pollution_TOR National Project Evaluator_Final 20.Apr2022.docx tor English 218.76 KB Posted 20
Download document FER Air Pollution final version with Executive Summary.docx report English 180.40 KB Posted 26
Title Final project evaluation for the “Tackling Air Pollution in the City of Skopje”
Atlas Project Number: 00110061
Evaluation Plan: 2021-2025, North Macedonia
Evaluation Type: Final Project
Status: Completed
Completion Date: 07/2022
Planned End Date: 07/2022
Management Response: Yes
Focus Area:
  • 1. Poverty
  • 2. Energy
  • 3. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.1.1 Capacities developed across the whole of government to integrate the 2030 Agenda, the Paris Agreement and other international agreements in development plans and budgets, and to analyse progress towards the SDGs, using innovative and data-driven solutions
SDG Goal
  • Goal 1. End poverty in all its forms everywhere
  • Goal 11. Make cities and human settlements inclusive, safe, resilient and sustainable
  • Goal 13. Take urgent action to combat climate change and its impacts
  • Goal 3. Ensure healthy lives and promote well-being for all at all ages
  • Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all
SDG Target
  • 1.5 By 2030, build the resilience of the poor and those in vulnerable situations and reduce their exposure and vulnerability to climate-related extreme events and other economic, social and environmental shocks and disasters
  • 11.6 By 2030, reduce the adverse per capita environmental impact of cities, including by paying special attention to air quality and municipal and other waste management
  • 13.2 Integrate climate change measures into national policies, strategies and planning
  • 3.9 By 2030, substantially reduce the number of deaths and illnesses from hazardous chemicals and air, water and soil pollution and contamination
  • 7.1 By 2030, ensure universal access to affordable, reliable and modern energy services
Evaluation Budget(US $): 20,000
Source of Funding: Project funds
Evaluation Expenditure(US $): 14,413
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Aldo Magoga International Evaluator
Natasha Markovska National Evaluator
GEF Evaluation: No
Key Stakeholders: Ministry of Environment and Physical Planning
Countries: MACEDONIA, THE FORMER YUGOSLAV REPUBLIC OF
Comments:

The evaluation is commissioned by UNDP in close cooperation with the Donor (Sida).

Lessons
1.

1.1.  Lessons learned

Since international factors can affect energy prices, consider diversification of sources and self-production energy measures, such as solar energy and solar panel heating systems. Targeting target groups towards the most vulnerable sectors of the population is a promising practice to be developed in future interventions. Changes of authority are often obstacles to project continuity and sustainability. The PT's positive articulation and communication with the municipal authorities could start as early as during the election campaign with the candidates to minimise the handover time.


Findings
1.

1.1.  Findings

1.1.1.    Relevance

The project's relevance, already high in the formulation phase due to the high pollution level in Skopje, has increased in recent months due to the international crisis and the increase in energy and fuel prices. The project is designed to support the advancement of the country's EU accession agenda and is aligned to and will use locally owned systems to develop sustainable outcomes with strong local ownership.

The project properly focuses on the causal factors (lack of a monitoring system, effective inter-sectoral coordination mechanisms, insufficient legislative framework, lack of evidence of the effectiveness of pollution reduction measures in households, low awareness and widespread polluting behaviours). It contributes to achieving the national priorities and the outcomes set in the UNDP Country Project Document 2016-2020 signed by the Macedonian Government. The stakeholders involved in the project have been well identified: ministries, municipalities, academia, auditors, and households (focusing on the most vulnerable).

The role of CSOs is limited, and the design has not sufficiently considered the need for more substantive support in inter-sectoral articulation. Another limitation identified by several of the interviewed stakeholders has been the non-inclusion of solar-based heating sources since the conditions were not favourable at the time of the design.

1.1.2.    Efficiency

Despite the complications caused by COVID-19, the overall performance is good. In June 2022, approximately 91% of the project time, the budget execution is 71%. The project team plan foresees 95%-100% of execution in the available time. The institutional change in the city of Skopje has made coordination and communication with the new municipal authorities challenging during the first semester of 2022. The costs of the project activities and products have been in line with national standards.

The decision to support the most vulnerable households with interventions limited to thermal insulation (windows) and heating systems has allowed extending the number of beneficiaries cost-efficiently. The enthusiastic, motivated and qualified team of UNDP and the implementation mechanisms proved efficient, ensuring a constant and attentive follow-up of the activities with the stakeholders involved.

1.1.3.    Effectiveness

The project shows relevant progress towards the achievement of the outputs. The most significant are the Monitoring System, the excellent web-based platform and heating surveys (C1),  the correspondent tables for the rulebook on Labelling of Energy-Related Products to the template given by the Secretariat for European Affairs (C2), the measures addressing the causes of pollution for 120 household heating and the revision of Subsidy models (C3), the marketing survey and capacity/gap assessments to design and implement training and awareness-raising activities (C4). Apportionment study is the first scientific study in the country, and in general, the project is based on a solid partnership with science. The project also sought to primarily support households from different vulnerable groups like low-income households, persons with special needs, persons with disabilities, single-parent families and others.

Improvements in heating systems have made it possible to reduce pollution in households, demonstrating a low-emission district concept to contribute to medium-term changes in how air pollution is tackled in the country (C3). Nevertheless, the effects of components 1, 2 and 4 are still incipient or partial. The web platform (C1) is used only by university researchers but not by governments, CSOs, citizens and the private sector. Progress in the corresponding tables but not towards legislative changes (C2). No evidence (behavioural change survey not implemented) of the effectiveness of the communication strategy, but increased participants to calls and decreased scepticism.

1.1.4.    Impact

Significant steps forward have been achieved, but the project only partially established a fully functional platform that brings all traditional and non-traditional partners to work together, the limitations in the legal framework remain, and, above all, funds for the public sector to implement a plan based on the excellent products (monitoring system and subsidy program for low-emission energy-efficient districts) are minimal, with little inter-institutional coordination and weak municipal technical and financial capacities. CSOs are still facing challenges in participating in a meaningful dialogue with the Government to identify solutions and implement appropriate actions and measures.

Increased media coverage of air pollution can be linked to the training sessions for media and the public call to tackle air pollution. More than 15 companies were involved in the training session. Partial and anecdotal evidence of the changes in the behaviours of citizens. Significant challenges in the coming winter due to rising electricity prices could hurt the overall impact of the project. The institutional ownership of the two main project partners and beneficiaries, the MEPP and the City of Skopje, is partial. Some challenges emerged due to the recent changes in municipal authorities.

1.1.5.    Sustainability/Ownership

The overall and long-term sustainability of the project faces challenges due to the budgetary constraints of the public sector, which hinder the dissemination and replicability of the project's excellent results in the national territory. The limited role of civil society organisations in the project limits the long-term effects and sustainability. The same applies to the private sector, which was considered in the research for the retailers of heating devices and training for them to offer more EE products. The short time available for the project implementation and the difficulties caused by COVID have made it challenging to involve these two important stakeholders.

1.1.6.    Gender perspective

Even if the project does not have a relevant gender approach, in the implementation, the gender perspective has been positively considered, especially in the participation of women: women participated very actively in the working groups and technical teams (C1) and the selection criteria, oriented to support the most vulnerable population, included prioritisation of female-headed households (C3). All institutions have and apply gender and inclusion policies.


Recommendations
1

Recommendation 1. For future projects, foresee a more active role of the CSOs, especially in promoting their role in the articulation with the public sector. Evaluate the possibility of creating a complementary project planning and monitoring mechanism (Technical Committee) with a more active role for the target institutions and CSOs. Provide for the identification of alternatives to electric heating systems (biomass, photovoltaic energy) to minimise possible negative impacts caused by the cost of electricity. Carry out a gender analysis and thus strengthen the gender approach. The same is true for the population that lives in a situation of greater vulnerability.

2

Recommendation 2: Ensure constant monitoring of the activities that remain to be completed in the scheduled times (C 3). Properly consider climatic factors in the programming to measure pollution due to home heating during the summer. Pay attention to planning to prevent delays and bottlenecks for the implementation of the activities, to observe and measure the results and effects of the implemented activities before the end of the project.

3

Recommendation 3: Strengthen the participation of citizens, government institutions, the private sector and CSOs in the web platform, spreading its use in social networks or providing specific activities and workshops. Support the articulation in existing/new coordination spaces between institutions and CSOs. Maintain and enhance the partnership with academia and science to enable a robust and comprehensive knowledge base for the informed design of policies and measures for tackling air pollution.

4

Recommendation 4: For future projects, extend the coverage of the subsidy model to other vulnerable municipalities, linking such initiatives to national policies promoted by the Ministry of the Environment with other funds, especially from the European Union, to achieve more impact and sustainability. Consider working with the Parliament and National Commissions on Energy and Environment to encourage the discussion and approval of new regulations and legislative changes. Once the impact of COVID-19 has ended or is limited, evaluate the possibility of working with schools and children and adolescents to sensitise them, incorporating the project issues into the educational curriculum.

5

Recommendation 5: Encourage the municipal appropriation and diffusion of the technologies and policies of the project. To increase the financial capacities of the municipality of Aerodrom directly supported (C3) and of the possible municipalities supported with future projects, favour the appropriation of such municipalities to the benefits of projects that improve the municipal capacities of access to European, national (also of the private sector) and other international funds, as promoted by BMCPI project. Influencing municipal environmental policies by replicating and institutionalising the subsidy model (and others, such as tax incentives, loans with low-interest rates, cash or in-kind subsidies) in coordination with the Ministry of the Environment, the academy, the CSOs and the private sector.

6

Recommendation 6: Based on the positive experience of targeting component 3, continue targeting this and other future interventions towards the most vulnerable population groups, including low-income women. Develop a gender analysis, with the participation of the target groups and beneficiaries, before the presentation of new proposals or as the initial activity of a new project, in such a way that the design of the intervention may have such an approach explicitly reflected in outputs, activities, and budget.

1. Recommendation:

Recommendation 1. For future projects, foresee a more active role of the CSOs, especially in promoting their role in the articulation with the public sector. Evaluate the possibility of creating a complementary project planning and monitoring mechanism (Technical Committee) with a more active role for the target institutions and CSOs. Provide for the identification of alternatives to electric heating systems (biomass, photovoltaic energy) to minimise possible negative impacts caused by the cost of electricity. Carry out a gender analysis and thus strengthen the gender approach. The same is true for the population that lives in a situation of greater vulnerability.

Management Response: [Added: 2022/09/21]

The recommendation is partially accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 For future projects, foresee a more active role of the CSOs
[Added: 2022/09/21]
UNDP Project/ Environment Portfolio 2024/03 Initiated Initiated/included in the project document for the planned second phase of the project. Collaboration with 10 CSOs is foreseen in the second phase of the project that is expected to start in October 2022.
1.3 Provide for the identification of alternatives to electric heating systems (biomass, photovoltaic energy) to minimize possible negative impacts caused by the cost of electricity
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2024/03 Initiated Initiated/included in the project document for the planned second phase of the project.
1.1 For future projects, foresee a more active role of the CSOs
[Added: 2022/09/21]
UNDP Project/ Environment Portfolio 2022/07 Completed In July 2022 Low-Value Grant Agreements were signed with 4 CSOs.
1.4 Carry out a gender analysis and thus strengthen the gender approach
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/09 Completed National consultant contracted to carry out gender specific analyses. Recommendation shall be taken into consideration in the implementation of the second phase of the project.
1.2 Evaluate the possibility of creating a complementary project planning and monitoring mechanism (Technical Committee)
[Added: 2022/09/21]
Government of North Macedonia 2022/09 No Longer Applicable [Justification: Such Technical Committee already exist and is established by the Government. Based on the experience and lessons learnt so far, such technical committees cannot function if established at a project level.]
2. Recommendation:

Recommendation 2: Ensure constant monitoring of the activities that remain to be completed in the scheduled times (C 3). Properly consider climatic factors in the programming to measure pollution due to home heating during the summer. Pay attention to planning to prevent delays and bottlenecks for the implementation of the activities, to observe and measure the results and effects of the implemented activities before the end of the project.

Management Response: [Added: 2022/09/21]

The recommendation is accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Ensure constant monitoring of the activities that remain to be completed in the scheduled times (C3)
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2024/01 Initiated Initiated/included in the project document for the planned second phase of the project. Within the next project phase the project will also engage Monitoring associate on full time basis.
2.2 Properly consider climatic factors in the programming to measure pollution due to home heating during the summer
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2024/03 Initiated Initiated/included in the project document for the planned second phase of the project. The follow-up project activities will be implemented in longer period and there will be more time for implementation of out-door energy efficient activities and measures.
2.3 Pay attention to planning to prevent delays and bottlenecks for the implementation of the activities, to observe and measure the results and effects of the implemented activities before the end of the project
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/06 Initiated Initiated/included in the project document for the planned second phase of the project. The follow-up phase proposed two surveys on public awareness to be conducted – at the very beginning of the project period and at its end.
2.1 Ensure constant monitoring of the activities that remain to be completed in the scheduled times (C3)
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/07 Completed All activities are completed. The Supervising company constantly followed the work progress and on regular basis informed the project team.
3. Recommendation:

Recommendation 3: Strengthen the participation of citizens, government institutions, the private sector and CSOs in the web platform, spreading its use in social networks or providing specific activities and workshops. Support the articulation in existing/new coordination spaces between institutions and CSOs. Maintain and enhance the partnership with academia and science to enable a robust and comprehensive knowledge base for the informed design of policies and measures for tackling air pollution.

Management Response: [Added: 2022/09/21] [Last Updated: 2022/09/21]

The recommendation is accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Strengthen the participation of citizens, government institutions, the private sector and CSOs in the web platform, spreading its use in social networks or providing specific activities and workshops.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2024/03 Initiated Initiated/included in the project document for the planned second phase of the project. The second project phase envisions strengthened participation of citizens, government institutions, the private sector and CSOs in the coordination activities and decision-making processes.
3.2 Support the articulation in existing/new coordination spaces between institutions and CSOs.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/01 Initiated Initiated/included in the project document for the planned second phase of the project. The recommendation is accepted and found to be relevant. In the planned second phase specific activities and events for coordination among the key stakeholders are envisioned.
3.3 Maintain and enhance the partnership with academia and science to enable a robust and comprehensive knowledge base for the informed design of policies and measures for tackling air pollution.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/01 Initiated Initiated/included in the project document for the planned second phase of the project. The recommendation is accepted and found to be relevant, hence the cooperation with University Goce Delcev as Responsible Party will continue in the second project phase.
3.1 Strengthen the participation of citizens, government institutions, the private sector and CSOs in the web platform, spreading its use in social networks or providing specific activities and workshops.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/08 Completed The web platform is upgraded with 4 additional modules to enable better coordination of the interested parties and implementation of air quality plans.
4. Recommendation:

Recommendation 4: For future projects, extend the coverage of the subsidy model to other vulnerable municipalities, linking such initiatives to national policies promoted by the Ministry of the Environment with other funds, especially from the European Union, to achieve more impact and sustainability. Consider working with the Parliament and National Commissions on Energy and Environment to encourage the discussion and approval of new regulations and legislative changes. Once the impact of COVID-19 has ended or is limited, evaluate the possibility of working with schools and children and adolescents to sensitise them, incorporating the project issues into the educational curriculum.

Management Response: [Added: 2022/09/21]

The recommendation is partially accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 For future projects, extend the coverage of the subsidy model to other vulnerable municipalities, linking such initiatives to national policies promoted by the Ministry of the Environment with other funds, especially from the European Union, to achieve more impact and sustainability.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/03 Initiated The lessons learnt from the first project is that such activity should be incorporated into the ministries and municipal programmes to support energy efficiency and energy poverty, and implemented by the respective entities and not by the project itself. The study about subsidies developed within the first phase of the project, shall be used as a basis to open discussions about respective subsidy models for pilot municipalities.
4.3. Once the impact of COVID-19 has ended or is limited, evaluate the possibility of working with schools and children and adolescents to sensitise them, incorporating the project issues into the educational curriculum.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/03 Initiated At conceptual/planning level. Relevant activities will be initiated with the start of the Project’s upcoming phase. This approach is also considered for the second project phase, where the in-door pollution will be measured in schools/kindergartens and hence awareness raising activities developed.
4.3. Once the impact of COVID-19 has ended or is limited, evaluate the possibility of working with schools and children and adolescents to sensitise them, incorporating the project issues into the educational curriculum.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/09 Completed By the end of the first project phase – in 2 schools and 1 kindergarten the in-door sets and out-door monitoring station were connected on TVs presenting real-time air quality data for these institutions.
4.2 Consider working with the Parliament and National Commissions on Energy and Environment to encourage the discussion and approval of new regulations and legislative changes.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/09 No Longer Applicable [Justification: The first phase project provided support to the respective Ministries, which are responsible to further communicate the sub-law related improvements with the Parliament or other relevant institutions. The second project phase includes activities related to development of air quality plans for the pilot municipalities, and capacity building of inspection system for implementation of the Law on the Quality of Ambiental Air and other related laws. ]
5. Recommendation:

Recommendation 5: Encourage the municipal appropriation and diffusion of the technologies and policies of the project. To increase the financial capacities of the municipality of Aerodrom directly supported (C3) and of the possible municipalities supported with future projects, favour the appropriation of such municipalities to the benefits of projects that improve the municipal capacities of access to European, national (also of the private sector) and other international funds, as promoted by BMCPI project. Influencing municipal environmental policies by replicating and institutionalising the subsidy model (and others, such as tax incentives, loans with low-interest rates, cash or in-kind subsidies) in coordination with the Ministry of the Environment, the academy, the CSOs and the private sector.

Management Response: [Added: 2022/09/21]

The recommendation is partially accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Encourage the municipal appropriation and diffusion of the technologies and policies of the project. To increase the financial capacities of the municipality of Aerodrom directly supported (C3) and of the possible municipalities supported with future projects, favour the appropriation of such municipalities to the benefits of projects that improve the municipal capacities of access to European, national (also of the private sector) and other international funds, as promoted by BMCPI project. Influencing municipal environmental policies by replicating and institutionalising the subsidy model (and others, such as tax incentives, loans with low-interest rates, cash or in-kind subsidies) in coordination with the Ministry of the Environment, the academy, the CSOs and the private sector.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2023/04 Initiated At conceptual/planning level. Relevant activities will be initiated with the start of the Project’s upcoming phase. Within the second project phase the study about subsidies, shall be used as a basis to open discussions about respective subsidy models for pilot municipalities. Both the Air Pollution project and the BMCPI project within the second phase period will work under the same framework, and hance exchange knowledge and good practices. This will be the case also with the other UNDP projects (e.g. GFF) working on similar issues. Municipality of Aerodrom is not directly targeted within the second project phase.
6. Recommendation:

Recommendation 6: Based on the positive experience of targeting component 3, continue targeting this and other future interventions towards the most vulnerable population groups, including low-income women. Develop a gender analysis, with the participation of the target groups and beneficiaries, before the presentation of new proposals or as the initial activity of a new project, in such a way that the design of the intervention may have such an approach explicitly reflected in outputs, activities, and budget.

Management Response: [Added: 2022/09/21]

The recommendation is partially accepted.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.2 Develop a gender analysis, with the participation of the target groups and beneficiaries, before the presentation of new proposals or as the initial activity of a new project, in such a way that the design of the intervention may have such an approach explicitly reflected in outputs, activities, and budget.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/09 Completed National consultant contracted to carry out gender specific analyses. Recommendation shell be taken into consideration in the implementation of the second phase of the project
6.1. Based on the positive experience of targeting component 3, continue targeting this and other future interventions towards the most vulnerable population groups, including low-income women.
[Added: 2022/09/21]
UNDP Project/Environment Portfolio 2022/09 No Longer Applicable [Justification: As agreed with the project donor, the second project phase shall not continue the subsidy programme for most vulnerable categories of citizen for introducing more environmentally friendly means for household heating (e. g. replacement of old, inefficient stoves). ]

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org