Evaluation Resource Centre
Login
» Brazil » 2017-2022 » BRA09/G31 Market Transformation for Energy Efficiency in Buildings TE

BRA09/G31 Market Transformation for Energy Efficiency in Buildings TE

Report Cover Image
Evaluation Plan:
2017-2022, Brazil
Evaluation Type:
Final Project
Planned End Date:
09/2018
Completion Date:
09/2018
Status:
Completed
Management Response:
No
Evaluation Budget(US $):
20,000

Share

Title BRA09/G31 Market Transformation for Energy Efficiency in Buildings TE
Atlas Project Number: 00048524
Evaluation Plan: 2017-2022, Brazil
Evaluation Type: Final Project
Status: Completed
Completion Date: 09/2018
Planned End Date: 09/2018
Management Response: No
Focus Area:
  • 1. Energy
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 2.5.1 Solutions developed, financed and applied at scale for energy efficiency and transformation to clean energy and zero-carbon development, for poverty eradication and structural transformation
SDG Goal
  • Goal 7. Ensure access to affordable, reliable, sustainable and modern energy for all
  • Goal 9. Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation
SDG Target
  • 7.1 By 2030, ensure universal access to affordable, reliable and modern energy services
  • 7.2 By 2030, increase substantially the share of renewable energy in the global energy mix
  • 9.4 By 2030, upgrade infrastructure and retrofit industries to make them sustainable, with increased resource-use efficiency and greater adoption of clean and environmentally sound technologies and industrial processes, with all countries taking action in accordance with their respective capabilities
Evaluation Budget(US $): 20,000
Source of Funding: GEF
Evaluation Expenditure(US $): 20,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Nationality
Remi Rijs Msc
GEF Evaluation: Yes
GEF Project Title: Market Transformation for Energy Efficiency in Buildings
Evaluation Type: Terminal Evaluation
Focal Area: Biodiversity
Project Type: FSP
GEF Phase: GEF-4
GEF Project ID: 2941
PIMS Number: 3665
Key Stakeholders: Ministry of Environment
Countries: BRAZIL
Lessons
Findings
1.

3.1 Project scope and design
Poor design has repeatedly been invoked in the PIRs as the main problem affecting Project delivery; the same stance was taken by the MTR. In the view of the Evaluator, poor design is not so obvious and therefore requires a closer look. Project preparation was done by UNDP and IDB Headquarters. Communication was maintained with the GEF on milestones and to receive feedback on the EEGM and the arrangements between the Agencies. The Prodoc and RF are detailed and probably exceed the average quality of GEF projects from this period, although there are some inconsistencies. The proposed indicators are appropriate though not always “SMART”. The objective-level targets are overly ambitious but this is a known issue with GEF CC projects from those years. The questions raised by GEF reviewers were addressed.

Tag: Efficiency Relevance Global Environment Facility fund Procurement Programme/Project Design Theory of Change International Financial Institutions

2.

3.2 Barrier analysis and vertical logic

The vertical logic expressed in the RF is reasonably well but unfortunately, a number of assumptions proved invalid. By consequence, the Project does not lead to a sufficient and necessary set of outcomes for delivery of the project objectives. The definition of outcomes and outputs is not always respected. The most striking deficiency concerns Output 4.2 (“At least 250 projects approved under the EEGM and provided with guarantees”). This is obviously not an output (i.e. a direct result of GEF funds and co-finance support). This result is only achieved on the assumption of a strong, positive ESCO market response to the EEGM, which in itself cannot be controlled by the Project, but may happen if Outcome 4 is delivered and relevant, and if external factors do not prevent the market from responding within the timeframe set. Output 4.2 should have been set at impact level (project objective) and not at output level and even then
it was very ambitious (unrealistic).

Tag: Relevance Programme/Project Design Results-Based Management

3.

3.3 Project implementation
3.3.1 Management arrangements

The management and project execution arrangements for a nationally implemented (NIM) project as described in the Project Document, were not fully implemented. Brazilian Law establishes that the functions of NPD and NPC cannot be transferred to another party and must rest with MMA. Hence the NPC is national public servant and cannot be recruited from project funds as usual. Specific Terms of Reference for these functions were drafted in the Project Document. Legislation further prohibits the deployment of an externally-funded PMU inside a Federal Government Ministry. Hence a Project is either implemented by Ministry staff itself with national resources (which requires priorities and capacities are in place); or it relies on externally-based individuals hired with GEF funds and supported by assigned UNDP CO staff.  In 2012, a Technical Advisor (TA) was contracted (based at UNDP and funded from the Project) to complement MMA competences and facilitate liaison between UNDP and the Government, reportedly after lengthy reviews of the Terms of Reference.

There are a range of factors here that relate to the way how International Cooperation projects are handled within the Brazilian Federal Government. From the perspective of this Project - and probably the GEF portfolio in Brazil as a whole - there is certainly room for clarification of roles and functions, and for more expedite procedures linking UNDP, the Executing Agency, and the GEF Operational Focal Point. It is highlighted that the OFP in Brazil does not assume the functions as expected by GEF. From the perspective of public administration in Brazil, there may be good grounds for the current modus operandi. A more active role of the Government could have ensured more adequate coordination between the GEF Agencies. With respect to Project implementation, UNDP CO has learned from these experiences and has therefore moved to a de facto assisted NIM with a key role for the Technical Advisor.

Tag: Relevance Human and Financial resources Implementation Modality Policies & Procedures Project and Programme management

4.

3.3.2 Financial Monitoring and Reporting

In the absence of a Steering Committee and annual tri-partite meetings, project management lacked the support to move forward. Records of strategic and operational management decisions are not available until 2013. A strategic focus towards annual results was absent and AWP targets were eventually moved to the next year. Until 2015, a cumulative lag between expenditures and budget planning was built up, as shown in the next figure for the “3E Project”. Actual expenditure figures are taken from the consolidated Combined Delivery Reports (CDRs) over the period 2011-2017. After 2015, expenditures exceed the original budget. (See Annex G for more figures).

The Project assigned to MMA (BRA/09/G31 components 1 and 2) was audited three times by the Controladoria-Geral da União (CGU). The audits covered the Project years 2011-2014; 2015-2016; and 2017, their scope being defined as follows: (a) relation between Project execution and work plans; (b) adequacy of internal control mechanisms finance, assets and administration; (c) conformity of expenditures with Prodoc and applicable standards and regulation, including documented justifications; and (d) implementation of previous audit recommendations; expenditures in audit year; location of assets and equipment not registered by the executing agency (MMA). The audits state that: (i) the CDRs are trustworthy and the cost items included supported by evidence and adequately documented; (ii) the internal control procedures were adequate.

 

Tag: Efficiency Relevance Human and Financial resources Policies & Procedures Procurement

5.

3.3.3 Monitoring and evaluation
UNDP has a range of instruments for project monitoring and evaluating progress and results, including the mandatory: (i) Project Inception Workshop and Report (IW/IR); (ii) Annual reporting (APR/PIR); (iii) Quarterly progress reports (QPR); (iv) Annual work plans (AWP); (v) Steering Committee meetings; (vi) Tripartite Reviews (TPR); and (vii) Mid-term and Terminal Evaluations (MTR/TE). These can be complemented with: (viii) field visits by UNDP (CO and RTA) to the project; and (ix) ad-hoc evaluations and expert missions. An M&E plan is supposed to be finalized at inception stage 68 and should include a time schedule of programmed M&E events.

The IW was a 1-day event and the IR is a brief 3-page document which falls short of the scope stipulated in the Prodoc. A formal M&E plan was not produced.  For reference, the IR plus the (non-consolidated) agenda proposals by UNPD and IDB are attached (Annex J). As one can see, the IW did not conclude but rather initiated a process towards the establishment of the NPSC and NPC; neither did it prepare the first AWP. Follow-up actions were identified and assigned to the partners, but eventually the process took two years. As a lesson learnt, UNDP nowadays pays more attention to the inception process and considers it a key milestone in the project implementation cycle. Similarly, IDB usually requires a series of conditions prior to be in place before disbursement of project funds can start – these include having the full project team and administrative capacity in place. Another lesson drawn from this (and other) projects is, that having more than one ministry at the same project level (in this case MMA and MME) tends to be challenging.

Based on the limited records available, the Evaluator concludes that a reasonable effort was made by UNDP and IDB to follow-up on the project implementation process and progress.

Tag: Challenges Communication Monitoring and Evaluation Quality Assurance

6.

3.3.4 Project documentation and reporting

The TE has found the reports produced by the Project and subcontractors of high quality and incorrespondence with their Terms of Reference issued by UNDP and MMA. Atla Consulting produced a series of presentations for explaining and promoting the EEGM among market actors. Inputs from Atla and IDB were sent to UNDP to feed into the PIRs.

Project management reporting is reflected in Atlas and in a narrative form in the annual PIRs. Written evidence from the early project years (2010-2012) is scarce, which is primarily explained by the difficulties to establish the project management structure and the absence of a contracted Project Coordinator. Once the Project started operations and with the TA on board, reporting became expedite. As a suggestion, progress reporting per outcome in the PIRs could be more concise and in response to the indicators. To this extent, hands-on guidelines for implementing RBM could be issued or enforced since concise, standardized reporting saves time and facilitates tracking of progress over subsequent PIRs. With these comments in mind, the Evaluator rates reporting as Satisfactory (S).

Tag: Monitoring and Evaluation Quality Assurance

7.

3.3.4.1 Knowledge management

Knowledge Management (KM) was not an explicit aspect of project design under GEF 4 and GEF 5 but nowadays it is. MMA has created a web portal to make available products and presentations of the Project to a broader public. Analysis and conclusions concerning EE market development and the achievements of the GEF project in particular are not made however. Analysis and the aggregation of results adds value to the KM process as it facilitates accessibility to information and interpretation of results by external stakeholders (cooperation agencies, other governments, professionals, among others).

The Evaluator emphasizes the Prodoc instructions to the IP to keep track of relevant project information from Project start. For some reason, effective KM tends to be problematic. One reason is probably that IPs in the first project years tend to focus on securing political support within the organization; and on fully understanding the project problem and context. Another reason can be a lack of experience and vision to know upfront which information and data should be systematized. The development of a template to this purpose during the inception phase may assist the IP (specifically the NPC) to structure the KM process. From a broader perspective, UNDP may consider including KM into a web-based application.

An alternative approach can be to hire a consultancy before MTR and TE to collect and systematize project information and extract lessons and knowledge. This option is foreseen already in the standard M&E plan. However, this approach is less robust in a context with frequent staff rotation. Current UNDP Guidelines require the NPC to be under contract until completion of the TE. In practice, this may be hard to enforce as consultants tend to look for a new job during the last year of a project. Therefore, the Evaluator highly recommends to implement an active KM system throughout the duration of a Project.

Tag: Communication Country Support Platform Human and Financial resources Knowledge management

8.

3.3.5 Coordination between stakeholders

Coordination with sector stakeholders was limited during project design. The key Government stakeholders took part in the Inception Workshop. During the first project year, coordination between MMA and MME did not result in the envisaged Project management structure. Coordination greatly improved once the TAC was established with participation of key Ministries and other public entities.


In later Project years, a number of partnerships was established or promoted. The Project successfully fostered engagement with stakeholders in peripheral states of Brazil. It organized events and seminars for Project dissemination and interaction with - and between- stakeholders. The Evaluator attended the final Seminar held in Brasilia (14-15 December 2017) which demonstrated a comprehensive view on the challenges and opportunities for EE in public and private buildings in Brazil. In summary, the Evaluator judges stakeholder coordination as Satisfactory (S).

Tag: Partnership Country Government Coordination

9.

3.3.6 Quality of IA and EA implementation
For evaluation of the IA’s role and performance – rather than the Project itself – reference is made to the evaluation questions (Annex D, Table 4a-c).

3.3.6.1 UNDP implementation
The Evaluation observes a reasonable focus on the achievement of results although more holistically than on a year-by-year basis. Progress tracking is hampered by the lack of annual targets in the Prodoc. Since the Project started with great delay and the end targets were not realistic, the PIRs made a continuous effort to explain these circumstances. This situation was rather uncomfortable for UNDP and its partners.

UNDP made a considerable advocacy effort to support MMA to install the Project institutions but also ran into the (legal) limitations to install a full-fledged PMU within the Government. The TE lacks information for a proper assessment of UNDP staff capacity and resources and its relation to the use of the Agency Fee. As a general appraisal, projects executed under NIM still need substantial CO support in terms of advocacy, strategic guidance and operational support, which is not duly accounted for during project design. The CO therefore provided assistance that went beyond the scope of a typical NIM.

Tag: Global Environment Facility fund Implementation Modality Oversight Quality Assurance UNDP management Technical Support

10.

3.3.6.2 IDB implementation
In a tandem with Atla Consulting, IDB Invest provided adequate support to the EEGM and had a team for project appraisal and issuance and monitoring of the PCGs. The Evaluator had no access to information to judge internal staff capacity and resources but observes that financial reporting was adequate and shared with UNDP to feed into the PIRs. Annual progress monitoring does not show a focus to keep the GEF project on track towards attainment of the committed GHG reductions and investment. However, achieved GHG reductions and investment at end-of-project were duly reported.

No adaptive management has been applied to re-define project activities or budget to deliver on the key GEF indicators within the Project’s timeframe. The value created by the GEF resources managed by IDB (US$ 10M) is poor and opportunities to diversify and test alternative financial and non-financial solutions were not explored. On the other hand, about USD 9M remain unused.
 

Tag: Implementation Modality Partnership Quality Assurance International Financial Institutions

11.

3.3.6.3 MMA implementation
MMA acted as the lead Implementing Partner (Executing Agency) for the Project, delivering the NPD and NPC and supporting staff. During the first years, MMA could not establish the NPSC as MME decided not to take part in the Project. Once it was agreed to change the project structure and create the TAC, implementation became more expedite. The justification of the SR may serve to demonstrate the level of ownership and understanding of the Project context which had grown with MMA (see AnnexF).

Constraints in public administration law impede a full-fledged NIM within the Government. Since the Project team consisted of public servants, their availability for the Project was not full-time and activities expected under NIM sometimes got compromised. No detailed records of time dedication by Government staff to the Project have been made available, hence a quantitative assessment of IP capacity cannot be made. Back-up from UNDP and the appointment of a Project Technical Advisor were needed to circumvent this problem, putting substantial management tasks in hands of UNDP (but not political choices, which rest with MMA). The tandem UNDP-MMA performed well but with higher transaction costs than supposed under a NIM. With respect to reporting, reference is made to the comments in section 3.3.6.1; it is likely that UNDP and the TA did the larger share of work. Concerning the focus on results, also see under UNDP.

Tag: Effectiveness Implementation Modality Ownership Policies & Procedures Country Government

12.

3.4 Project results
In line with the Evaluation methodology, the project outcomes will be assessed and rated on the dimensions relevance, effectiveness, and efficiency. Relevance refers to the appropriateness of the Project to address a development problem in line with national priorities and GEF and UNDP objectives. Effectiveness considers the ability of the Project to reach the objectives set forth. Efficiency refers to the cost-effectiveness of the Project and the delivery of results in relation to its cost and the projected timeline.

Information on achieved results and impact was not fully consolidated by end 2017. The Evaluator has used the following sources as means of verification: (1) PIR 2017; (2) Reports by subcontractors of the Project; (3) Email communication from Atla Consulting (23 January 2018); (4) IDB Invest/Atla EEGM Project
Presentation (16 February 2018; updated 19 June 2018); and (5) Spreadsheet by NPC (7 June 2018). A detailed evaluation of achieved direct and indirect GHG emission reductions (from the EEGM and investment in public buildings) was not prepared prior to the TE and as far as the Evaluator can observe, input data to this purpose were not collected and centralized for consolidation. The Evaluator has had no insight in the feasibility studies underpinning the investments made. Neither did the Chiller Project (BRA/12/G77) track the GHG emission reductions that can be attributed to the retrofits.

Tag: Operational Efficiency

13.

3.4.1 Overall Project results
Relevance
As a general appraisal, the Project was clearly aligned with national and global environmental priorities as described in the CEO ER. Rising energy costs in Brazil, increased fiscal budget controls for public buildings, a need for mobilizing investment capital to reduce the GHG footprint of the building sector, in a context
of progressive global climate change, reconfirm the relevance of the Project’s subject and scope. With positive developments in the financial market and increased awareness of ESCOs and building operators, the design of appropriate financing mechanisms also gained relevance.

Tag: Energy Efficiency Relevance Resource mobilization

14.

Effectiveness
The achievement of the Project in terms of the objective-level targets mainly involves the leveraged investment, energy savings and associated GHG reductions (GEF-5 CCM indicators). No comprehensive data were made available to the Evaluator, hence an approximation was made based on the limited information and some educated guesses (see Annex J). The results are summarized in the next table. These figures provide a reasonable indication of the direct contribution of the project to the GEF CCM indicators.

The figures are presented in the following table to allow for a comparison with the targets. As can be seen in the table, the Project underperformed with respect to leveraging of investment in EE under the Project (31%). It delivered about 54% of the direct emission reductions. In this context, the overambitious targets and the Project design centered on the EEGM, are recalled.

 

Tag: Emission Reduction Effectiveness Data and Statistics

15.

Efficiency
The Evaluation has reviewed a sample of the services and goods procured under the Project. Since price levels vary from country to country, it is difficult to make comparisons with Projects in other countries. Reference (unit) prices are not available for tailored consultancies and services as requested under the Project. As a general appraisal, the Evaluator judges the value of the services and products received as acceptable or good in relation to their cost. The financial audits FY-2014 and FY-2016 arrived at similar conclusions.
As per 31 December 2017, the Project had disbursed or committed 92% of its GEF funds, according to the following Table.

The main changes in type of budget category are: (i) reduction in international consultants from US$ 281,250 to US$ 34,494; (ii) reduction in local consultants from US$ 1,250,000 to US$ 24,633; and (iii) increase of contractual services by companies from US$ 333,420 to US$ 1,883,378.
 

Tag: Efficiency Human and Financial resources Project and Programme management Data and Statistics

16.

Cofinance

The participation of public staff in the project deserves acknowledgement as a demonstration of Government commitment to the Project objectives and direct support to its implementation. This cofinance was not tracked during the Project. Work spent by MMA staff (5 people) on the Project amounts to approx. BRL 1.9 million (USD 560,000) over the period 2014-2017. The Evaluator assesses the in-kind support from MMA and other Government partners as very substantial and of high quality (adding to the implementation capacity and results of the Project). It may represent a value in the range of USD 2-3 million over the full project period.

Tag: Efficiency Government Cost-sharing

17.

3.5 Project results per outcome
3.5.1 Outcome 1: Enhanced EE investments through Capacity Building in EE in private and public sector buildings. (Budget: US$ 1,368,170 GEF; US$ 500,000 cofinance)

3.5.1.1 Description of activities and delivered outputs

The purpose of Outcome 1 was to develop capacity in Brazil for identification, formulation, implementation and management of EE projects in the buildings sector. The RF indicators (see table section 3.5.1.3) essentially refer to individuals and institutions trained (6, 7) and people informed or made aware on EE (7). The identified target groups included buildings administrators, owners, technical staff, service providers (consultants and ESCOs), architects and engineers, and banks and other financial institutions, for a total of 5,000 trainees. Ample geographical coverage was foreseen through activities in at least 10 large cities. The Prodoc provided a detailed list of topics per target group.


In order to reach its objective to build market capacity (indicator 5), the implemented activities were broadened to include certification of energy professionals on measuring and evaluation methodologies pursuant to the rules of ANEEL ´s PEE programme and the Efficiency Valuation Organization (EVO) International Performance Measurement and Verification Protocol (IPMVP). In total, the Project enhanced professional competences among approx. 1,000 individuals, including IPMVP certification (59) and creating capacities to apply methodologies under the ANEEL PEE, which is an asset for project pipeline development.
 

Tag: Effectiveness Communication Capacity Building

18.

3.5.1.2 Relevance
The outputs delivered are certainly supportive for EE market development, as such, relevance of this outcome is rated Highly Satisfactory (HS).

3.5.1.3 Effectiveness
The following table assesses the achievement of the outputs delivered in relation to the indicators for Outcome 1 in the Project’s SRF. Based on a review of the material provided (training material, websites) as well as interviews with stakeholders, quality and appropriateness of the activities is evaluated as very
good. Although the original target of 5,000 professionals has not been reached, there has been overcompliance in other activities. The alignment with PBE/Edifica and the ANEEL EE program further contributed to the effectiveness of the capacity building. As such, the Evaluator assesses the effectiveness as Highly Satisfactory (HS).
 

Tag: Effectiveness Relevance Ownership Capacity Building

19.

3.5.1.4 Efficiency
The close interaction with baseline programmes and the establishment of partnerships also point into high efficiency. A question that can be raised is the distribution of the activities over the project time span, as most training was carried out in the period 2014-2017 instead of spread out over 7 years as originally proposed. Albeit somewhat of a rush towards the end (especially in 2017), there are good indications that timing was more favorable towards the end of the Project period (2015-2017) and the market more receptive for absorbing the delivered TA products.

Since it is unlikely that this progress would have been achieved without GEF support, incrementality of the latter seems assured. Based on these considerations, the Evaluator rates efficiency as: Satisfactory (S).

Tag: Efficiency Partnership

20.

3.5.2 Outcome 2: Access to EE services and commercial financing for public sector buildings enhanced with the support and strengthening of existing public initiatives. (Budget: US$1,183,330 GEF; US$ 160,000 cofinance)

3.5.2.1 Description of activities and delivered outputs

Outcome 2 was revised in the SR considering the observations made in the MTR 98 . The Project commissioned a study to assess the status of performance contracts in the private sector in Brazil (since these contracts were virtually non-existent in the public sector), which indicated that such contracts should be used with special caution if applied to the public sector. Alternative contract modalities for the public sector were to be considered as well, such as: Public-Private Partnerships (PPPs), Differentiated Contracting Regime (RDC), and submission of projects for calls for proposal from Energy Concessionaires, according to ANEEL's new Energy Efficiency Program Procedures (PROPEE). The Substantive Revision therefore re-frames the PBI as the set of Federal Government baseline activities and proposes to take benefit from the Project BRA/09/G31 to enhance these.

The RF indicators for outcome 2 (see table section 3.5.2.3) refer to the delivery of supportive instruments for benchmarking of public buildings (8), capacity building (9, overlapping with Outcome 1), calls for projects (9, 10) and preparation of a few demonstration pilots. Investment in two public buildings was realized during the Project, which is captured in the original indicators (the ANEEL headquarters and the MMA Bloco B building at the Esplanada Monumental, both in Brasilia). The investment proposals were channeled through the local electricity distribution company (CEB) under the ANEEL PEE program. Also, a study was concluded into the design of a public building program to access carbon funds under application of CDM methodologies.

The benchmarking component took benefit from baseline work in this area by the Conselho Brasileiro de Construção Sustentável (CBCS) which had launched its operational energy performance programme in 2013 coordinated by a Technical Committee. CBCS was contracted by the Project to execute energy audits
in 20 public buildings to identify the share of different energy end-uses in total building consumption, and to assess building energy performance in terms of floor area and occupation. 99 The buildings were selected after a public call and represent a variety of buildings across Brazil as presented in the next table. This activity included the installation and monitoring of electricity metering systems in the buildings for a 12-month period.
 

Tag: Energy Operational Efficiency Policies & Procedures Procurement Private Sector Technical Support

21.

3.5.2.2 Relevance
Based on the products delivered through combined GEF and baseline efforts, it seems reasonable to assume that the Federal Government has now most of the tools in hands to start implementing EE in public buildings. It is noted that the direct target (10 RFPs at End-of-Project) has not yet been met. If political back-up is sustained and (public and private) capital is available, one would expect a swift offtake of EE investments in public building in the near future. A post-project impact study should be able to verify this. In this context, the key role of MPDG, the Central de Compras and the Tribunal de Contas for mainstreaming of EE into the Federal Government, is also highlighted.

Considering the positive reception of Project results by the Federal Government and the progress made in EE policy, the Evaluator rates this Outcome as Satisfactory (S).

 

Tag: Relevance Policies & Procedures

22.

3.5.2.3 Effectiveness
The following table reviews the outputs delivered in relation to the SR for Outcome 2. The pursued activities are not always well aligned with the defined outputs. However, the Project managed to implement a work package that is consistent and supportive for the Outcome objective (i.e. to enable a public building EE programme to take off). While one cannot consider such a programme to be fully operational as yet (see indicator 9), a number of the legal and conceptual barriers with respect to contracting modalities for EE in public buildings (including EPC) seem to be greatly reduced or removed altogether. Normative instruments (NI 02/2014) and enforcement (TUC 1056/2017) have been approved providing a stick for implementing energy management systems and EE investments in the sector.


On the other hand, pipeline development is still limited to opportunities in the vicinity of the key project partners (MMA, ANEEL) hence it is still early to speak about a developed public sector EE market. The Evaluator would recommend a post-project evaluation of market off-take, for example by assessing the portfolio of the ANEEL PEE project during the next years and recording investment sums and GHG emission reductions. In conclusion, the Evaluator rates the effectiveness of this outcome as Satisfactory (S).

Tag: Energy Effectiveness Operational Efficiency Project and Programme management

23.

3.5.2.4 Efficiency
The change in approach to address EE in the public sector as outlined in the SR implied the mobilization of public sector baseline initiatives. This approach strengthened the incrementality principle for this outcome and thereby the efficiency of the allocated GEF resources. Although not tracked as co-finance,
the value of the baseline support actually provided to this component is substantially larger than the US$ 160,000 estimated at Project design. The wide scope and quality of the delivered products indicate good value for money. The Evaluator therefore rates the efficiency as Highly Satisfactory.

Tag: Energy Efficiency

24.

3.5.4 Outcome 4: Energy Efficiency Guarantee Mechanism (EEGM) made available to stimulate EE investment through ESCOs (Budget: US$ 10,195,000 GEF; US$ 120,217,250 cofinance)

 

3.5.4.1 Description of activities and delivered outputs
The purpose of Outcome 4 was to implement the EEGM in Brazil, promote it in the market and secure financing for investment in EE systems to reduce electricity consumption and achieve GHG emission reductions. The indicators are given in the RF (see section 3.5.4.3). Since substantial effort was dedicated to promotion and outreach, two indicators are presented which were not included in the PIRs. The EEGM was a pilot to test whether a partial guarantee mechanism would also work in the Brazilian market. This involved the assumptions that: (i) ESCOs are a relevant delivery mechanism for EE in Brazil; and (ii) debt financing is a critical barrier for ESCOs. The EEGM was opened on 24 May 2013 and ended 23 May 2018. Original maximum exposure to any single ESCO was up to USD 2,500,000 and IDB’s exposure
limit up to 50% of an EE project amount, at any time under PCG; these restrictions were later relaxed to USD 5,000,000 and 80%, respectively. At Project design, eligible Partial Credit Guarantees (PCGs) were in the range of USD 150,000-800,000; the upper limit was in 2016 extended to USD1,600,000. The maximum
PCG tenor was 7 years.

Tag: Energy Efficiency Operational Efficiency Private Sector Data and Statistics

25.

3.5.4.2 Relevance
The poor acceptance of the EEGM by national FIs, combined with the high share of ESCOs that could not close project finance, suggests that ESCOs still face great difficulties to access debt capital. It is unlikely that the current institutional and legal set-up would be the configuration-of-choice for a guarantee mechanism for EE in Brazil (as discussed in section 3.1). The EEGM presents an operational loss which might be mitigated by increasing PCG commissions. IDB concludes that transactional costs were high and that the program as it was designed is not financially sustainable – hence IDB will not continue the EEGM. However, information about price elasticity and competitiveness of the EEGM viz-a-viz other financing options has not been quantified. The gathered data set is too small to draw hard conclusions whether the EEGM can work in Brazil, under which conditions, and for which type of beneficiaries.

The EEGM was not tested under favorable market conditions. Timing was also prematue in the absenceof a substantial volume of ESCOs with the right profile to benefit from the facility. An analysis of market drivers points into the direction that the EEGM is not the game changer for developing the EE market in Brazil as assumed at Project design. Instead, other barriers including more flexible public procurement modalities and the lack of integrated retrofit solutions provided by the sector, have been identified as more relevant. These barriers were further explored and are partially addressed under Outcomes 1 and 2 (and 3).

Tag: Challenges Efficiency Relevance Policies & Procedures

26.

3.5.4.3 Effectiveness
Demand for the EEGM fell far behind the original projections, which clearly indicates that its role as a market agent is weaker than expected. Also, response from FIs was low. On the other hand, many project proposals were received, indicating a need for funding.

The effectiveness of the EEGM to induce a market transformation was marginal and its ability to trigger investment (and GHG reductions) lagged behind the expectations. However, a (small) portfolio of EE building projects was implemented thanks to the EEGM. This is an important achievement compared to the baseline situation. The Evaluator therefore rates its effectiveness as Moderately Unsatisfactory (MU).

Tag: Effectiveness Efficiency Energy Challenges

27.

 

Demand for the EEGM fell far behind the original projections, which clearly indicates that its role as a market agent is weaker than expected. Also, response from FIs was low. On the other hand, many project proposals were received, indicating a need for funding.

The effectiveness of the EEGM to induce a market transformation was marginal and its ability to trigger investment (and GHG reductions) lagged behind the expectations. However, a (small) portfolio of EE building projects was implemented thanks to the EEGM. This is an important achievement compared to the baseline situation. The Evaluator therefore rates its effectiveness as Moderately Unsatisfactory (MU).

Tag: Effectiveness Operational Efficiency

28.

3.5.4.4 Efficiency
Since the actual demand fell behind the projections at Project design stage, nearly 90% of the GEF grants allocated to the EEGM remained untouched. The Evaluator considers that with more active monitoring of the EEGM’s exposure and liquidity, part of the remainder could have been re-allocated to other activities to support the Project objectives and accelerate the envisioned EE market transformation.
Another perspective is to consider that USD 1 M operational budget translated into USD 17.5M leveraged capital for project investment. This achievement is 2-3 times better than the typical cofinancing ratios for GEF CCM projects, under the premise that the remainder (approx. USD 9M) will be released for reprogramming. As such, the Evaluator rates the efficiency of this outcome as: Moderately Satisfactory (MS).

Tag: Efficiency Global Environment Facility fund

29.

3.6 Sustainability, Impact and Catalytic effects

3.6.1 Sustainability

PROJECT OVERALL SUSTAINABILITY

CRITERIA RATING COMMENTS
Overall likelihood of sustainability ML  
Financial resources ML The EEGM was not a decisive factor to achieve financial closure of EE projects. Public funding for
EE is available under ANEEL PEE, but morefunding
is needed to cover the market.
Socio-economic L Economic revenues of EE are positive.
Political ML Political support is not guaranteed
Environmental L With Brazil’s power sector leaning towards
increased thermal generation, the impact of EE is
even more relevant.

 

Tag: Sustainability

30.

3.6.2 Impact and catalytic effects

RATING PROJECT IMPACT 119 & CATALYIC EFFECTS

Criteria Rating Observations 
Environmental status improvement M Minor, but verifiable reduction of GHG emissions due to
EE investments.
Environmental stress reduction N Not measured but negligible.
Progress towards stress/status change S Positive contribution to regulatory and policy changes to foster EE technologies in buildings, thereby reducing GHG emissions compared to the baseline scenario.
Catalytic effects not to be rated Delivered EE and chiller demonstration projects are a showcase for replication in public and private sector in Brazil. Capacity Building, M&V certifiers, legal solutions to performance contracts in public sector, participation in public calls.

 

Tag: Emission Reduction Energy Impact

31.

3.6.3 Partnerships
The Project established or promoted a substantial number of partnerships with, and between stakeholders. In alignment with the revised approach outlined in the SR, these partnerships build forth on, and complement, current baseline initiatives and are likely to be sustained after Project termination.
The Evaluator confirms the close collaboration with these partners, and other stakeholders, through interviews and events, and concludes that partnership building was Highly Satisfactory (HS).

Tag: Partnership

Recommendations
1

Recommendations1. The exit strategy for the EEGM provided by IDB Invest is essentially based on financial-operational considerations. It does not provide answers to the initial questions how an EEGM should be shaped in Brazil, how relevant it is and for whom, and to whom the EEGM should be handed over. It is recommended to UNDP, IDB and the Government to initiate a dialogue on the Project content, its key results and exit strategy as soon as possible.
2

2. The position and interests of Atla Consulting should be considered; although contracted as a service provider, Atla has acted as a financial agent in the market for a product that is no longer serviced by the supplier. This situation affects the perception and credibility of EEGM product and its agent by the market. Therefore, Atla should soonest receive from IDB all relevant information for making opportune business decisions.
3

3. The Project achieved a substantial degree of problem ownership that should not get lost. One would like the role of the Project to oversee and convoke the market to be continued. This function is not automatically assumed by MMA. The Evaluator would recommend MMA and UNDP to engage with MME on this point and evaluate the options. One suggestion is to continue the TAC as a kind of consultative organ for public programs (ANEEL PEE and others). Essentially, this recommendation concerns the identification of an exit strategy for Outcomes 1 and 2.
4

4. The question arises to what extent the EE market is ready now and what challenges remain. It is recommended to MMA and UNDP to carry out a wrap-up exercise to systematize the results and experiences obtained. The contributions to the final Project seminar (December 2017) may serve as a useful starting point.Ideally, a road map with key actions can be devised capitalizing on the expertise andknow-how that exists under the partnerships established during the Project. This activity can be finalized in 2018.
5

5. While acknowledging the efforts made so far, it is recommended to MMA, UNDP and IDB to devise an explicit Knowledge Management system for the Project. With a large number of actors in the field, there is a need for concentration, validation and dissemination of information and expertise. Ideally, the legacy of the 3E Project can be used to start an information clearinghouse (repository) on EE in Brazil. This involves continuous management, resources, and a business model.
6

6. A specific recommendation to MMA and MME is to consider integration of an RE/EE information function into the ANEEL PEE program and operating it from the resources available under this program. A successful information clearing house offering validated data, best practices, technology factsheets, calculation tools, etc., is a valuable asset for outreach to the market, thereby speeding up project pipeline development and increasing quality of proposals.
7

7. According to MMA, the applicable GEF and UNDP procedures and guidelines were not always clear, limiting the Implementing Partner’s effectiveness and ability to make adjustments to the project strategy and targets. It is recommended to UNDP to periodically consolidate applicable guidelines, manuals, etc., to ensure that information is offered to the IP in a consistent manner. An annual workshop or webinar for project coordinators may be considered, possibly to be held in the months prior to PIR delivery and organized by the regional UNDP-GEF office.
Management response not available

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org