- Evaluation Plan:
- 2017-2022, Brazil
- Evaluation Type:
- Final Project
- Planned End Date:
- 09/2018
- Completion Date:
- 09/2018
- Status:
- Completed
- Management Response:
- No
- Evaluation Budget(US $):
- 20,000
BRA09/G31 Market Transformation for Energy Efficiency in Buildings TE
Share
Title | BRA09/G31 Market Transformation for Energy Efficiency in Buildings TE | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Atlas Project Number: | 00048524 | ||||||||||||||
Evaluation Plan: | 2017-2022, Brazil | ||||||||||||||
Evaluation Type: | Final Project | ||||||||||||||
Status: | Completed | ||||||||||||||
Completion Date: | 09/2018 | ||||||||||||||
Planned End Date: | 09/2018 | ||||||||||||||
Management Response: | No | ||||||||||||||
Focus Area: |
|
||||||||||||||
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021) |
|
||||||||||||||
SDG Goal |
|
||||||||||||||
SDG Target |
|
||||||||||||||
Evaluation Budget(US $): | 20,000 | ||||||||||||||
Source of Funding: | GEF | ||||||||||||||
Evaluation Expenditure(US $): | 20,000 | ||||||||||||||
Joint Programme: | No | ||||||||||||||
Joint Evaluation: | No | ||||||||||||||
Evaluation Team members: |
|
||||||||||||||
GEF Evaluation: | Yes
|
||||||||||||||
Key Stakeholders: | Ministry of Environment | ||||||||||||||
Countries: | BRAZIL |
Lessons | |||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Findings | |||||||||||||||||||
1. | 3.1 Project scope and design Tag: Efficiency Relevance Global Environment Facility fund Procurement Programme/Project Design Theory of Change International Financial Institutions |
||||||||||||||||||
2. | 3.2 Barrier analysis and vertical logic The vertical logic expressed in the RF is reasonably well but unfortunately, a number of assumptions proved invalid. By consequence, the Project does not lead to a sufficient and necessary set of outcomes for delivery of the project objectives. The definition of outcomes and outputs is not always respected. The most striking deficiency concerns Output 4.2 (“At least 250 projects approved under the EEGM and provided with guarantees”). This is obviously not an output (i.e. a direct result of GEF funds and co-finance support). This result is only achieved on the assumption of a strong, positive ESCO market response to the EEGM, which in itself cannot be controlled by the Project, but may happen if Outcome 4 is delivered and relevant, and if external factors do not prevent the market from responding within the timeframe set. Output 4.2 should have been set at impact level (project objective) and not at output level and even then Tag: Relevance Programme/Project Design Results-Based Management |
||||||||||||||||||
3. | 3.3 Project implementation The management and project execution arrangements for a nationally implemented (NIM) project as described in the Project Document, were not fully implemented. Brazilian Law establishes that the functions of NPD and NPC cannot be transferred to another party and must rest with MMA. Hence the NPC is national public servant and cannot be recruited from project funds as usual. Specific Terms of Reference for these functions were drafted in the Project Document. Legislation further prohibits the deployment of an externally-funded PMU inside a Federal Government Ministry. Hence a Project is either implemented by Ministry staff itself with national resources (which requires priorities and capacities are in place); or it relies on externally-based individuals hired with GEF funds and supported by assigned UNDP CO staff. In 2012, a Technical Advisor (TA) was contracted (based at UNDP and funded from the Project) to complement MMA competences and facilitate liaison between UNDP and the Government, reportedly after lengthy reviews of the Terms of Reference. There are a range of factors here that relate to the way how International Cooperation projects are handled within the Brazilian Federal Government. From the perspective of this Project - and probably the GEF portfolio in Brazil as a whole - there is certainly room for clarification of roles and functions, and for more expedite procedures linking UNDP, the Executing Agency, and the GEF Operational Focal Point. It is highlighted that the OFP in Brazil does not assume the functions as expected by GEF. From the perspective of public administration in Brazil, there may be good grounds for the current modus operandi. A more active role of the Government could have ensured more adequate coordination between the GEF Agencies. With respect to Project implementation, UNDP CO has learned from these experiences and has therefore moved to a de facto assisted NIM with a key role for the Technical Advisor. Tag: Relevance Human and Financial resources Implementation Modality Policies & Procedures Project and Programme management |
||||||||||||||||||
4. | 3.3.2 Financial Monitoring and Reporting In the absence of a Steering Committee and annual tri-partite meetings, project management lacked the support to move forward. Records of strategic and operational management decisions are not available until 2013. A strategic focus towards annual results was absent and AWP targets were eventually moved to the next year. Until 2015, a cumulative lag between expenditures and budget planning was built up, as shown in the next figure for the “3E Project”. Actual expenditure figures are taken from the consolidated Combined Delivery Reports (CDRs) over the period 2011-2017. After 2015, expenditures exceed the original budget. (See Annex G for more figures). The Project assigned to MMA (BRA/09/G31 components 1 and 2) was audited three times by the Controladoria-Geral da União (CGU). The audits covered the Project years 2011-2014; 2015-2016; and 2017, their scope being defined as follows: (a) relation between Project execution and work plans; (b) adequacy of internal control mechanisms finance, assets and administration; (c) conformity of expenditures with Prodoc and applicable standards and regulation, including documented justifications; and (d) implementation of previous audit recommendations; expenditures in audit year; location of assets and equipment not registered by the executing agency (MMA). The audits state that: (i) the CDRs are trustworthy and the cost items included supported by evidence and adequately documented; (ii) the internal control procedures were adequate.
Tag: Efficiency Relevance Human and Financial resources Policies & Procedures Procurement |
||||||||||||||||||
5. | 3.3.3 Monitoring and evaluation The IW was a 1-day event and the IR is a brief 3-page document which falls short of the scope stipulated in the Prodoc. A formal M&E plan was not produced. For reference, the IR plus the (non-consolidated) agenda proposals by UNPD and IDB are attached (Annex J). As one can see, the IW did not conclude but rather initiated a process towards the establishment of the NPSC and NPC; neither did it prepare the first AWP. Follow-up actions were identified and assigned to the partners, but eventually the process took two years. As a lesson learnt, UNDP nowadays pays more attention to the inception process and considers it a key milestone in the project implementation cycle. Similarly, IDB usually requires a series of conditions prior to be in place before disbursement of project funds can start – these include having the full project team and administrative capacity in place. Another lesson drawn from this (and other) projects is, that having more than one ministry at the same project level (in this case MMA and MME) tends to be challenging. Based on the limited records available, the Evaluator concludes that a reasonable effort was made by UNDP and IDB to follow-up on the project implementation process and progress. Tag: Challenges Communication Monitoring and Evaluation Quality Assurance |
||||||||||||||||||
6. | 3.3.4 Project documentation and reporting The TE has found the reports produced by the Project and subcontractors of high quality and incorrespondence with their Terms of Reference issued by UNDP and MMA. Atla Consulting produced a series of presentations for explaining and promoting the EEGM among market actors. Inputs from Atla and IDB were sent to UNDP to feed into the PIRs. Project management reporting is reflected in Atlas and in a narrative form in the annual PIRs. Written evidence from the early project years (2010-2012) is scarce, which is primarily explained by the difficulties to establish the project management structure and the absence of a contracted Project Coordinator. Once the Project started operations and with the TA on board, reporting became expedite. As a suggestion, progress reporting per outcome in the PIRs could be more concise and in response to the indicators. To this extent, hands-on guidelines for implementing RBM could be issued or enforced since concise, standardized reporting saves time and facilitates tracking of progress over subsequent PIRs. With these comments in mind, the Evaluator rates reporting as Satisfactory (S). Tag: Monitoring and Evaluation Quality Assurance |
||||||||||||||||||
7. | 3.3.4.1 Knowledge management Knowledge Management (KM) was not an explicit aspect of project design under GEF 4 and GEF 5 but nowadays it is. MMA has created a web portal to make available products and presentations of the Project to a broader public. Analysis and conclusions concerning EE market development and the achievements of the GEF project in particular are not made however. Analysis and the aggregation of results adds value to the KM process as it facilitates accessibility to information and interpretation of results by external stakeholders (cooperation agencies, other governments, professionals, among others). The Evaluator emphasizes the Prodoc instructions to the IP to keep track of relevant project information from Project start. For some reason, effective KM tends to be problematic. One reason is probably that IPs in the first project years tend to focus on securing political support within the organization; and on fully understanding the project problem and context. Another reason can be a lack of experience and vision to know upfront which information and data should be systematized. The development of a template to this purpose during the inception phase may assist the IP (specifically the NPC) to structure the KM process. From a broader perspective, UNDP may consider including KM into a web-based application. An alternative approach can be to hire a consultancy before MTR and TE to collect and systematize project information and extract lessons and knowledge. This option is foreseen already in the standard M&E plan. However, this approach is less robust in a context with frequent staff rotation. Current UNDP Guidelines require the NPC to be under contract until completion of the TE. In practice, this may be hard to enforce as consultants tend to look for a new job during the last year of a project. Therefore, the Evaluator highly recommends to implement an active KM system throughout the duration of a Project. Tag: Communication Country Support Platform Human and Financial resources Knowledge management |
||||||||||||||||||
8. | 3.3.5 Coordination between stakeholders Coordination with sector stakeholders was limited during project design. The key Government stakeholders took part in the Inception Workshop. During the first project year, coordination between MMA and MME did not result in the envisaged Project management structure. Coordination greatly improved once the TAC was established with participation of key Ministries and other public entities.
Tag: Partnership Country Government Coordination |
||||||||||||||||||
9. | 3.3.6 Quality of IA and EA implementation 3.3.6.1 UNDP implementation UNDP made a considerable advocacy effort to support MMA to install the Project institutions but also ran into the (legal) limitations to install a full-fledged PMU within the Government. The TE lacks information for a proper assessment of UNDP staff capacity and resources and its relation to the use of the Agency Fee. As a general appraisal, projects executed under NIM still need substantial CO support in terms of advocacy, strategic guidance and operational support, which is not duly accounted for during project design. The CO therefore provided assistance that went beyond the scope of a typical NIM. Tag: Global Environment Facility fund Implementation Modality Oversight Quality Assurance UNDP management Technical Support |
||||||||||||||||||
10. | 3.3.6.2 IDB implementation No adaptive management has been applied to re-define project activities or budget to deliver on the key GEF indicators within the Project’s timeframe. The value created by the GEF resources managed by IDB (US$ 10M) is poor and opportunities to diversify and test alternative financial and non-financial solutions were not explored. On the other hand, about USD 9M remain unused. Tag: Implementation Modality Partnership Quality Assurance International Financial Institutions |
||||||||||||||||||
11. | 3.3.6.3 MMA implementation Constraints in public administration law impede a full-fledged NIM within the Government. Since the Project team consisted of public servants, their availability for the Project was not full-time and activities expected under NIM sometimes got compromised. No detailed records of time dedication by Government staff to the Project have been made available, hence a quantitative assessment of IP capacity cannot be made. Back-up from UNDP and the appointment of a Project Technical Advisor were needed to circumvent this problem, putting substantial management tasks in hands of UNDP (but not political choices, which rest with MMA). The tandem UNDP-MMA performed well but with higher transaction costs than supposed under a NIM. With respect to reporting, reference is made to the comments in section 3.3.6.1; it is likely that UNDP and the TA did the larger share of work. Concerning the focus on results, also see under UNDP. Tag: Effectiveness Implementation Modality Ownership Policies & Procedures Country Government |
||||||||||||||||||
12. | 3.4 Project results Information on achieved results and impact was not fully consolidated by end 2017. The Evaluator has used the following sources as means of verification: (1) PIR 2017; (2) Reports by subcontractors of the Project; (3) Email communication from Atla Consulting (23 January 2018); (4) IDB Invest/Atla EEGM Project Tag: Operational Efficiency |
||||||||||||||||||
13. | 3.4.1 Overall Project results Tag: Energy Efficiency Relevance Resource mobilization |
||||||||||||||||||
14. | Effectiveness The figures are presented in the following table to allow for a comparison with the targets. As can be seen in the table, the Project underperformed with respect to leveraging of investment in EE under the Project (31%). It delivered about 54% of the direct emission reductions. In this context, the overambitious targets and the Project design centered on the EEGM, are recalled.
Tag: Emission Reduction Effectiveness Data and Statistics |
||||||||||||||||||
15. | Efficiency The main changes in type of budget category are: (i) reduction in international consultants from US$ 281,250 to US$ 34,494; (ii) reduction in local consultants from US$ 1,250,000 to US$ 24,633; and (iii) increase of contractual services by companies from US$ 333,420 to US$ 1,883,378. Tag: Efficiency Human and Financial resources Project and Programme management Data and Statistics |
||||||||||||||||||
16. | Cofinance The participation of public staff in the project deserves acknowledgement as a demonstration of Government commitment to the Project objectives and direct support to its implementation. This cofinance was not tracked during the Project. Work spent by MMA staff (5 people) on the Project amounts to approx. BRL 1.9 million (USD 560,000) over the period 2014-2017. The Evaluator assesses the in-kind support from MMA and other Government partners as very substantial and of high quality (adding to the implementation capacity and results of the Project). It may represent a value in the range of USD 2-3 million over the full project period. Tag: Efficiency Government Cost-sharing |
||||||||||||||||||
17. | 3.5 Project results per outcome 3.5.1.1 Description of activities and delivered outputs The purpose of Outcome 1 was to develop capacity in Brazil for identification, formulation, implementation and management of EE projects in the buildings sector. The RF indicators (see table section 3.5.1.3) essentially refer to individuals and institutions trained (6, 7) and people informed or made aware on EE (7). The identified target groups included buildings administrators, owners, technical staff, service providers (consultants and ESCOs), architects and engineers, and banks and other financial institutions, for a total of 5,000 trainees. Ample geographical coverage was foreseen through activities in at least 10 large cities. The Prodoc provided a detailed list of topics per target group.
Tag: Effectiveness Communication Capacity Building |
||||||||||||||||||
18. | 3.5.1.2 Relevance 3.5.1.3 Effectiveness Tag: Effectiveness Relevance Ownership Capacity Building |
||||||||||||||||||
19. | 3.5.1.4 Efficiency Since it is unlikely that this progress would have been achieved without GEF support, incrementality of the latter seems assured. Based on these considerations, the Evaluator rates efficiency as: Satisfactory (S). Tag: Efficiency Partnership |
||||||||||||||||||
20. | 3.5.2 Outcome 2: Access to EE services and commercial financing for public sector buildings enhanced with the support and strengthening of existing public initiatives. (Budget: US$1,183,330 GEF; US$ 160,000 cofinance) 3.5.2.1 Description of activities and delivered outputs Outcome 2 was revised in the SR considering the observations made in the MTR 98 . The Project commissioned a study to assess the status of performance contracts in the private sector in Brazil (since these contracts were virtually non-existent in the public sector), which indicated that such contracts should be used with special caution if applied to the public sector. Alternative contract modalities for the public sector were to be considered as well, such as: Public-Private Partnerships (PPPs), Differentiated Contracting Regime (RDC), and submission of projects for calls for proposal from Energy Concessionaires, according to ANEEL's new Energy Efficiency Program Procedures (PROPEE). The Substantive Revision therefore re-frames the PBI as the set of Federal Government baseline activities and proposes to take benefit from the Project BRA/09/G31 to enhance these. The RF indicators for outcome 2 (see table section 3.5.2.3) refer to the delivery of supportive instruments for benchmarking of public buildings (8), capacity building (9, overlapping with Outcome 1), calls for projects (9, 10) and preparation of a few demonstration pilots. Investment in two public buildings was realized during the Project, which is captured in the original indicators (the ANEEL headquarters and the MMA Bloco B building at the Esplanada Monumental, both in Brasilia). The investment proposals were channeled through the local electricity distribution company (CEB) under the ANEEL PEE program. Also, a study was concluded into the design of a public building program to access carbon funds under application of CDM methodologies. The benchmarking component took benefit from baseline work in this area by the Conselho Brasileiro de Construção Sustentável (CBCS) which had launched its operational energy performance programme in 2013 coordinated by a Technical Committee. CBCS was contracted by the Project to execute energy audits Tag: Energy Operational Efficiency Policies & Procedures Procurement Private Sector Technical Support |
||||||||||||||||||
21. | 3.5.2.2 Relevance Considering the positive reception of Project results by the Federal Government and the progress made in EE policy, the Evaluator rates this Outcome as Satisfactory (S).
Tag: Relevance Policies & Procedures |
||||||||||||||||||
22. | 3.5.2.3 Effectiveness
Tag: Energy Effectiveness Operational Efficiency Project and Programme management |
||||||||||||||||||
23. | 3.5.2.4 Efficiency Tag: Energy Efficiency |
||||||||||||||||||
24. | 3.5.4 Outcome 4: Energy Efficiency Guarantee Mechanism (EEGM) made available to stimulate EE investment through ESCOs (Budget: US$ 10,195,000 GEF; US$ 120,217,250 cofinance)
3.5.4.1 Description of activities and delivered outputs Tag: Energy Efficiency Operational Efficiency Private Sector Data and Statistics |
||||||||||||||||||
25. | 3.5.4.2 Relevance The EEGM was not tested under favorable market conditions. Timing was also prematue in the absenceof a substantial volume of ESCOs with the right profile to benefit from the facility. An analysis of market drivers points into the direction that the EEGM is not the game changer for developing the EE market in Brazil as assumed at Project design. Instead, other barriers including more flexible public procurement modalities and the lack of integrated retrofit solutions provided by the sector, have been identified as more relevant. These barriers were further explored and are partially addressed under Outcomes 1 and 2 (and 3). Tag: Challenges Efficiency Relevance Policies & Procedures |
||||||||||||||||||
26. | 3.5.4.3 Effectiveness The effectiveness of the EEGM to induce a market transformation was marginal and its ability to trigger investment (and GHG reductions) lagged behind the expectations. However, a (small) portfolio of EE building projects was implemented thanks to the EEGM. This is an important achievement compared to the baseline situation. The Evaluator therefore rates its effectiveness as Moderately Unsatisfactory (MU). Tag: Effectiveness Efficiency Energy Challenges |
||||||||||||||||||
27. |
Demand for the EEGM fell far behind the original projections, which clearly indicates that its role as a market agent is weaker than expected. Also, response from FIs was low. On the other hand, many project proposals were received, indicating a need for funding. The effectiveness of the EEGM to induce a market transformation was marginal and its ability to trigger investment (and GHG reductions) lagged behind the expectations. However, a (small) portfolio of EE building projects was implemented thanks to the EEGM. This is an important achievement compared to the baseline situation. The Evaluator therefore rates its effectiveness as Moderately Unsatisfactory (MU). Tag: Effectiveness Operational Efficiency |
||||||||||||||||||
28. | 3.5.4.4 Efficiency Tag: Efficiency Global Environment Facility fund |
||||||||||||||||||
29. | 3.6 Sustainability, Impact and Catalytic effects 3.6.1 Sustainability PROJECT OVERALL SUSTAINABILITY
Tag: Sustainability |
||||||||||||||||||
30. | 3.6.2 Impact and catalytic effects RATING PROJECT IMPACT 119 & CATALYIC EFFECTS
Tag: Emission Reduction Energy Impact |
||||||||||||||||||
31. | 3.6.3 Partnerships Tag: Partnership |
Recommendations | |
---|---|
1 | Recommendations1. The exit strategy for the EEGM provided by IDB Invest is essentially based on financial-operational considerations. It does not provide answers to the initial questions how an EEGM should be shaped in Brazil, how relevant it is and for whom, and to whom the EEGM should be handed over. It is recommended to UNDP, IDB and the Government to initiate a dialogue on the Project content, its key results and exit strategy as soon as possible. |
2 | 2. The position and interests of Atla Consulting should be considered; although contracted as a service provider, Atla has acted as a financial agent in the market for a product that is no longer serviced by the supplier. This situation affects the perception and credibility of EEGM product and its agent by the market. Therefore, Atla should soonest receive from IDB all relevant information for making opportune business decisions. |
3 | 3. The Project achieved a substantial degree of problem ownership that should not get lost. One would like the role of the Project to oversee and convoke the market to be continued. This function is not automatically assumed by MMA. The Evaluator would recommend MMA and UNDP to engage with MME on this point and evaluate the options. One suggestion is to continue the TAC as a kind of consultative organ for public programs (ANEEL PEE and others). Essentially, this recommendation concerns the identification of an exit strategy for Outcomes 1 and 2. |
4 | 4. The question arises to what extent the EE market is ready now and what challenges remain. It is recommended to MMA and UNDP to carry out a wrap-up exercise to systematize the results and experiences obtained. The contributions to the final Project seminar (December 2017) may serve as a useful starting point.Ideally, a road map with key actions can be devised capitalizing on the expertise andknow-how that exists under the partnerships established during the Project. This activity can be finalized in 2018. |
5 | 5. While acknowledging the efforts made so far, it is recommended to MMA, UNDP and IDB to devise an explicit Knowledge Management system for the Project. With a large number of actors in the field, there is a need for concentration, validation and dissemination of information and expertise. Ideally, the legacy of the 3E Project can be used to start an information clearinghouse (repository) on EE in Brazil. This involves continuous management, resources, and a business model. |
6 | 6. A specific recommendation to MMA and MME is to consider integration of an RE/EE information function into the ANEEL PEE program and operating it from the resources available under this program. A successful information clearing house offering validated data, best practices, technology factsheets, calculation tools, etc., is a valuable asset for outreach to the market, thereby speeding up project pipeline development and increasing quality of proposals. |
7 | 7. According to MMA, the applicable GEF and UNDP procedures and guidelines were not always clear, limiting the Implementing Partner’s effectiveness and ability to make adjustments to the project strategy and targets. It is recommended to UNDP to periodically consolidate applicable guidelines, manuals, etc., to ensure that information is offered to the IP in a consistent manner. An annual workshop or webinar for project coordinators may be considered, possibly to be held in the months prior to PIR delivery and organized by the regional UNDP-GEF office. |