Mid-term evaluation Climate change resilient production landscapes and socio-economic networks advanced in Guatemala.

Report Cover Image
Evaluation Plan:
2015-2019, Guatemala
Evaluation Type:
Mid Term Project
Planned End Date:
01/2018
Completion Date:
04/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
20,000

Share

Document Type Language Size Status Downloads
Download document UNDP-AF Midterm Evaluation ToR 23 agst 2017.pdf tor Spanish 685.95 KB Posted 51
Download document PPRCC Guatemala PIMS 4383 MTE Final 28 04 18 ENGLISH.pdf report English 870.60 KB Posted 59
Title Mid-term evaluation Climate change resilient production landscapes and socio-economic networks advanced in Guatemala.
Atlas Project Number: 60326
Evaluation Plan: 2015-2019, Guatemala
Evaluation Type: Mid Term Project
Status: Completed
Completion Date: 04/2018
Planned End Date: 01/2018
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.5. Inclusive and sustainable solutions adopted to achieve increased energy efficiency and universal modern energy access (especially off-grid sources of renewable energy)
Evaluation Budget(US $): 20,000
Source of Funding:
Evaluation Expenditure(US $): 20,000
Joint Programme: No
Mandatory Evaluation: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Javier Jahnsen
GEF Evaluation: No
Key Stakeholders: Ministry of the Environment and natural Resources and Municipalities.
Countries: GUATEMALA
Comments:

Adaptation Fund

Lessons
Findings
Recommendations
1

Planning of monitoring activities of the four project results from this point until the end of the intervention using a critical path, that is, the sequence of activities to be implemented to strengthen monitoring. This includes the analysis of the variation of the scope of the indicators vis à vis the planned values (especially with under- performing indicators), the delegation of specific responsibilities for monitoring and verification of data quality.

2

Improvement of the descriptive and analytical content of the progress reports presented. Although the reports have specific format requirements, it is recommended to attach as an annex (i) a scoreboard of indicators by result with a system of colors to identify gaps and comments on how the products of the activities contribute to results, (ii) a table of financial analysis by component with the corresponding explanations on identified execution gaps; and (iii) a qualitative analysis on the consistency between the percentage of financial execution by component and the percentage of achievement of the targets.

3

In future similar interventions, a counterpart should be included within the budgets of community projects in order to strengthen the ownership of activities. The counterpart contribution could be in the form of work, inputs or financial resources of local organizations.

4

The content of the progress reports presented must coincide with each of the activities included in the grant agreements.  For example, if a result is "Implement N adaptation practices in X hectares", it would be expected that the progress reports, in addition to containing qualitative information on the activities of the result, contained information on the absolute increase in the number of adaptation practices implemented in a certain area. If there were variations in the scope of the expected targets, the PCL members and the technician in charge should include their insights and ways in which the gaps would be closed.

5

The field visit reports to the PCLs by the Management Unit staff should include additional specificity in terms of the problems identified, the recommendations provided and the follow up process.

6

In the final reports of the PCLs, the term impact must be substituted, given that what is reported, for the most part, are products / results. The effects must clearly demonstrate how they are aligned with a sustainability strategy following the logic of the results chain.

7

In future interventions, disbursements for overhead should be linked to performance (5% at the contract signature and 5% at midterm) with clear milestones of what institutional strengthening means.

8

For those completed PCLs, it is recommended that they have a physical file that consolidates the financial and programmatic information.  In addition, they should review and propose a realistic and measurable sustainability strategy that is coordinated with OLs.

9

The Management Unit should make an exhaustive analysis of the way in which the phase 1 PCLs will implement the sustainability measures included in their proposals. Similarly, the Management Unit must evaluate the effectiveness and efficiency of each PCL in achieving results.

10

For each PCL with a productive component, the OL must prepare a business plan that allows proper assessment of the planned growth and profitability scenarios for at least two years after the end of the PCL.  The business plan to be developed should consider, at least, the following sections:  description of the need and of the product or service that satisfies it, approach of strategies and competitive advantages, determination of the business development model / value chain, marketing (profile of the client and the potential market, analysis of the competition, description of the product , prices, advertising, distribution channels, sales projections, production and sales process, accounting and finance (including indicators such as liquidity, investment recovery period, profitability and balance points).

11

Plan and conduct workshops to exchange experiences between OL and OC with similar interventions (e.g. PCL of honey producers) that contribute to the elaboration of a business plan.

12

Identify people in the communities who can replicate the training and deliver the material for that purpose.

13

The inter-institutional strategic plans and the municipal development plans must be used as sustainability instruments. For this, the role of the municipal and community authorities is key to making the strategies of these plans visible in the municipal budgets, thus adopting the strategies as their own. This could be measured by observing which of the twelve municipalities actually carry out the key aspects of these strategies.

14

The Project should consider that it has an opportunity to share with different partners a compendium of lessons learned, good practices and knowledge management on specific topics such as the results of the basin approach in the context of adaptation and in light of international commitments on climate change and the results of the implementation of adaptation measures by community organizations that lead social processes in their localities.   Sharing this information, either in the form of compendiums of lessons learned or additional audiovisual material is a way towards the replication of good practices of the PPRCC.

15

To strengthen their role, the Inter-Institutional Support Committees should plan their meetings from this moment until the end of the project, analyze the manuals generated by the PPRRC and take advantage of information on0 project progress that has been shared with them throughout the intervention.

16

Given that the  PPRCC is considered a successful project, the MARN should consider the possibility of raising financial resources for its replication in other vulnerable areas of the country where the projections of loss of aptitude for certain crops or the decrease of adaptive capacity, for example, are expected to be more extreme. In the study "Guatemala's agriculture and climate change: Where are the priorities for adaptation? "Bouroncie et al, 2015) it is suggested that "the areas suitable for the cultivation of maize will decrease in the lower areas of Petén, Huehuetenango, Quiché Alta Verapaz and Izabal" and that the municipalities of these departments have the lowest degree of satisfaction of their needs, which decreases their adaptive capacity.

17

The Management Unit must evaluate the compliance of the collaboration letters signed with the ministries and other institutions and present the results of this assessment during the closing workshop and in the final report to the Project Board.

18

The Project Board has the challenge of strengthening its key and strategic role in the final stretch of the project. In addition to its performance as guarantor of oversight (macro aspects) of the project, the Board should focus its intervention on the comprehensive analysis of (i) macro financial management (relationship between disbursements, budgets and overall project execution) beyond the reports presented by the PPRCC, (ii) progress towards the achievement of results (analyzing the variations between targets and achievements and (iii) requesting the MARN Project Unit to prepare the exit strategy. This strategy should take into account the following guidelines:

  1. Select the national stakeholders that will intervene in the joint monitoring of the activities and results proposed by the project, institutions that participate in the multi-sectoral approach to climate change adaptation, development NGOs that work in municipalities, representatives of municipal authorities and municipal delegations of state institutions.
  2. The Project Board, supported by the Management Unit, should act as coordinator of the exit strategy of the project and, after its closure, transfer responsibility for continuity to the group of selected stakeholders.
  3. Systematization of answers to the question: Which PPRCC activities should be sustained?
  4. Joint preparation of a schedule to close the PPRCC and detail the nature, term and cost of activities to which continuity should be given.  
  5. Inclusion of indicators that allow rapid monitoring of the activities that make up the exit strategy. These indicators correspond to the percentage of planned activities that were executed and the percentage of commitments that had to be maintained and that were fulfilled by the different institutions.
  6. The exit strategy should incorporate the following information:

 

Strategy / Output activity

Who will be responsible?

Date on which the strategy will be executed

How will it be monitored?

What is the cost of this activity

 

 

 

 

 

 

1. Recommendation:

Planning of monitoring activities of the four project results from this point until the end of the intervention using a critical path, that is, the sequence of activities to be implemented to strengthen monitoring. This includes the analysis of the variation of the scope of the indicators vis à vis the planned values (especially with under- performing indicators), the delegation of specific responsibilities for monitoring and verification of data quality.

Management Response: [Added: 2018/12/11] [Last Updated: 2018/12/11]

The project is in the process of improving the monitoring and quality of information from field technical data, cross-checking field information, reviewing information reported by field technicians at three levels: Small Grants Projects Coordinators, monitoring and evaluation technician, and coordination with the Project Management Unit.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Elaborate the critical route with sequential activities that will allow the strengthen of the monitoring of those indicators with pending goal.
[Added: 2018/12/11]
Project Management Unit (PMU) 2019/01 Initiated The Project has a monitoring matrix, which is updated monthly. A critical route will be developed for the indicators with a goal to be met.
2. Recommendation:

Improvement of the descriptive and analytical content of the progress reports presented. Although the reports have specific format requirements, it is recommended to attach as an annex (i) a scoreboard of indicators by result with a system of colors to identify gaps and comments on how the products of the activities contribute to results, (ii) a table of financial analysis by component with the corresponding explanations on identified execution gaps; and (iii) a qualitative analysis on the consistency between the percentage of financial execution by component and the percentage of achievement of the targets.

Management Response: [Added: 2018/12/11]

Key Actions:

3. Recommendation:

In future similar interventions, a counterpart should be included within the budgets of community projects in order to strengthen the ownership of activities. The counterpart contribution could be in the form of work, inputs or financial resources of local organizations.

Management Response: [Added: 2018/12/11]

Key Actions:

4. Recommendation:

The content of the progress reports presented must coincide with each of the activities included in the grant agreements.  For example, if a result is "Implement N adaptation practices in X hectares", it would be expected that the progress reports, in addition to containing qualitative information on the activities of the result, contained information on the absolute increase in the number of adaptation practices implemented in a certain area. If there were variations in the scope of the expected targets, the PCL members and the technician in charge should include their insights and ways in which the gaps would be closed.

Management Response: [Added: 2018/12/11]

Key Actions:

5. Recommendation:

The field visit reports to the PCLs by the Management Unit staff should include additional specificity in terms of the problems identified, the recommendations provided and the follow up process.

Management Response: [Added: 2018/12/11]

Key Actions:

6. Recommendation:

In the final reports of the PCLs, the term impact must be substituted, given that what is reported, for the most part, are products / results. The effects must clearly demonstrate how they are aligned with a sustainability strategy following the logic of the results chain.

Management Response: [Added: 2018/12/11]

Key Actions:

7. Recommendation:

In future interventions, disbursements for overhead should be linked to performance (5% at the contract signature and 5% at midterm) with clear milestones of what institutional strengthening means.

Management Response: [Added: 2018/12/11]

Key Actions:

8. Recommendation:

For those completed PCLs, it is recommended that they have a physical file that consolidates the financial and programmatic information.  In addition, they should review and propose a realistic and measurable sustainability strategy that is coordinated with OLs.

Management Response: [Added: 2018/12/11]

Key Actions:

9. Recommendation:

The Management Unit should make an exhaustive analysis of the way in which the phase 1 PCLs will implement the sustainability measures included in their proposals. Similarly, the Management Unit must evaluate the effectiveness and efficiency of each PCL in achieving results.

Management Response: [Added: 2018/12/11]

Key Actions:

10. Recommendation:

For each PCL with a productive component, the OL must prepare a business plan that allows proper assessment of the planned growth and profitability scenarios for at least two years after the end of the PCL.  The business plan to be developed should consider, at least, the following sections:  description of the need and of the product or service that satisfies it, approach of strategies and competitive advantages, determination of the business development model / value chain, marketing (profile of the client and the potential market, analysis of the competition, description of the product , prices, advertising, distribution channels, sales projections, production and sales process, accounting and finance (including indicators such as liquidity, investment recovery period, profitability and balance points).

Management Response: [Added: 2018/12/11]

Key Actions:

11. Recommendation:

Plan and conduct workshops to exchange experiences between OL and OC with similar interventions (e.g. PCL of honey producers) that contribute to the elaboration of a business plan.

Management Response: [Added: 2018/12/11]

Key Actions:

12. Recommendation:

Identify people in the communities who can replicate the training and deliver the material for that purpose.

Management Response: [Added: 2018/12/11]

Key Actions:

13. Recommendation:

The inter-institutional strategic plans and the municipal development plans must be used as sustainability instruments. For this, the role of the municipal and community authorities is key to making the strategies of these plans visible in the municipal budgets, thus adopting the strategies as their own. This could be measured by observing which of the twelve municipalities actually carry out the key aspects of these strategies.

Management Response: [Added: 2018/12/11]

Key Actions:

14. Recommendation:

The Project should consider that it has an opportunity to share with different partners a compendium of lessons learned, good practices and knowledge management on specific topics such as the results of the basin approach in the context of adaptation and in light of international commitments on climate change and the results of the implementation of adaptation measures by community organizations that lead social processes in their localities.   Sharing this information, either in the form of compendiums of lessons learned or additional audiovisual material is a way towards the replication of good practices of the PPRCC.

Management Response: [Added: 2018/12/11]

Key Actions:

15. Recommendation:

To strengthen their role, the Inter-Institutional Support Committees should plan their meetings from this moment until the end of the project, analyze the manuals generated by the PPRRC and take advantage of information on0 project progress that has been shared with them throughout the intervention.

Management Response: [Added: 2018/12/11]

Key Actions:

16. Recommendation:

Given that the  PPRCC is considered a successful project, the MARN should consider the possibility of raising financial resources for its replication in other vulnerable areas of the country where the projections of loss of aptitude for certain crops or the decrease of adaptive capacity, for example, are expected to be more extreme. In the study "Guatemala's agriculture and climate change: Where are the priorities for adaptation? "Bouroncie et al, 2015) it is suggested that "the areas suitable for the cultivation of maize will decrease in the lower areas of Petén, Huehuetenango, Quiché Alta Verapaz and Izabal" and that the municipalities of these departments have the lowest degree of satisfaction of their needs, which decreases their adaptive capacity.

Management Response: [Added: 2018/12/11]

Key Actions:

17. Recommendation:

The Management Unit must evaluate the compliance of the collaboration letters signed with the ministries and other institutions and present the results of this assessment during the closing workshop and in the final report to the Project Board.

Management Response: [Added: 2018/12/11]

Key Actions:

18. Recommendation:

The Project Board has the challenge of strengthening its key and strategic role in the final stretch of the project. In addition to its performance as guarantor of oversight (macro aspects) of the project, the Board should focus its intervention on the comprehensive analysis of (i) macro financial management (relationship between disbursements, budgets and overall project execution) beyond the reports presented by the PPRCC, (ii) progress towards the achievement of results (analyzing the variations between targets and achievements and (iii) requesting the MARN Project Unit to prepare the exit strategy. This strategy should take into account the following guidelines:

  1. Select the national stakeholders that will intervene in the joint monitoring of the activities and results proposed by the project, institutions that participate in the multi-sectoral approach to climate change adaptation, development NGOs that work in municipalities, representatives of municipal authorities and municipal delegations of state institutions.
  2. The Project Board, supported by the Management Unit, should act as coordinator of the exit strategy of the project and, after its closure, transfer responsibility for continuity to the group of selected stakeholders.
  3. Systematization of answers to the question: Which PPRCC activities should be sustained?
  4. Joint preparation of a schedule to close the PPRCC and detail the nature, term and cost of activities to which continuity should be given.  
  5. Inclusion of indicators that allow rapid monitoring of the activities that make up the exit strategy. These indicators correspond to the percentage of planned activities that were executed and the percentage of commitments that had to be maintained and that were fulfilled by the different institutions.
  6. The exit strategy should incorporate the following information:

 

Strategy / Output activity

Who will be responsible?

Date on which the strategy will be executed

How will it be monitored?

What is the cost of this activity

 

 

 

 

 

 

Management Response: [Added: 2018/12/11]

Key Actions:

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org