- Evaluation Plan:
- 2016-2020, Indonesia
- Evaluation Type:
- Final Project
- Planned End Date:
- 07/2019
- Completion Date:
- 06/2019
- Status:
- Completed
- Management Response:
- Yes
- Evaluation Budget(US $):
- 20,000
00068398 Strategic Planning and Action to Climate Resilience (SPARC) GEF Terminal Evaluation
| Document | Type | Language | Size | Status | Downloads |
|---|---|---|---|---|---|
 ToR Terminal Evaluation Consultant.pdf |
tor | English | 351.22 KB | Posted | 1134 |
| Final Terminal Evaluation UNDP SPARC Project Indonesia 2 May 2019 signed.pdf |
report | English | 1455.60 KB | Posted | 1366 |
| Title | 00068398 Strategic Planning and Action to Climate Resilience (SPARC) GEF Terminal Evaluation | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Atlas Project Number: | 00068398 | ||||||||||||||
| Evaluation Plan: | 2016-2020, Indonesia | ||||||||||||||
| Evaluation Type: | Final Project | ||||||||||||||
| Status: | Completed | ||||||||||||||
| Completion Date: | 06/2019 | ||||||||||||||
| Planned End Date: | 07/2019 | ||||||||||||||
| Management Response: | Yes | ||||||||||||||
| Focus Area: |
|
||||||||||||||
| Corporate Outcome and Output (UNDP Strategic Plan 2018-2021) |
|
||||||||||||||
| SDG Goal |
|
||||||||||||||
| SDG Target |
|
||||||||||||||
| Evaluation Budget(US $): | 20,000 | ||||||||||||||
| Source of Funding: | SPARC Project | ||||||||||||||
| Evaluation Expenditure(US $): | 18,318 | ||||||||||||||
| Joint Programme: | No | ||||||||||||||
| Joint Evaluation: | No | ||||||||||||||
| Evaluation Team members: |
|
||||||||||||||
| GEF Evaluation: | Yes
|
||||||||||||||
| Key Stakeholders: | |||||||||||||||
| Countries: | INDONESIA |
| Lessons | |
|---|---|
| 1. | Some lessons learn revealed from the process of final evaluation; 1. Success of the SPARC project can be attributed in large measure to the commitment and endorsement by national, provincial and district government staff. A high level of country ownership is reflected in the adoption the SPARC model in policies, planning documents and action plans promoted by all levels of government. Indonesia’s Village Law (Law No.6 of 2014) and the associated VDF provides vital ongoing government financial commitment to SPARC actions that will continue to support the communities directly involved in the project and which can support actions in new communities through scaling up initiated by district and provincial governments. 2. The high efficiency of SPACR implementation led to an expansion into additional villages and a new district and a two year extension without requiring an increased budget. Monitoring and evaluation reports and observations during the TE determined the intended outputs of SPARC have been achieved and in many cases exceeded. While SPARC was highly successful in meeting targets that verify the achievement of project outputs and outcomes, the project could have done more to ensure and support the scaling up of the successes demonstrated. Some of learning from project implementationat at the field are as follow;
3. Dispite of the achievements made, Challenges may occur to sustain the institutional capacity created in government and for government to replicate the SPARC model in the villages, districts and provinces which did not directly participate in the SPARC project.
|
| Findings | |
| 1. | Project Design / FormulationSPARC is Indonesia’s first nationally coordinated CC project financed through an instrument of the CC Convention (GEF Special Climate Change Fund) which worked with provincial and district governments to strengthen the climate resilience of rural communities. It is a pilot project that worked in one of Indonesia’s more vulnerable provinces, NTT, where it selected and worked with 46 villages in four districts based on a Village Vulnerability and Climate Risk Index developed by the project. The success of SPARC has potential for replication across Indonesia, given the ability of the national implementing partner, MoEF Directorate for Climate Change Adaptation, to work with other provincial and district governments to address the needs of vulnerable communities. Tag: Rural development Climate Change Adaptation Disaster risk management Resilience building Global Environment Facility fund |
| 2. | At a global scale SPARC contributes to GEF objectives by reducing the vulnerability of the pilot rural communities to the adverse impacts of CC and increasing their adaptive capacity to respond to the impacts of climate change. SPARC contributed to UNDP country program outcomes (2011- 2015 and 2016-2020) by incorporating CC policies into NTT provincial and district planning and policy documents which address rural communities’ needs in response to a changing climate and contributed to inclusive, sustainable economic growth for these communities.
Tag: Climate Change Adaptation Global Environment Facility fund Urban Inclusive economic growth Policy Advisory |
| 3. | UNDP’s experience working with governments to develop strategies for integrated planning for inclusive and sustainable growth and projects that support economic opportunities to address poverty, inequality and exclusion are reflected in the logical framework of the project that has two clear outcomes. One outcome is to build institutional capacity within the NTT provincial government and within four district governments to integrate climate resilience into their current planning and development programs. The second outcome is to actively work with the government, particularly the four district governments, to strengthen the livelihoods of rural communities vulnerable to a changing climate. Tag: Programme/Project Design Inclusive economic growth Jobs and Livelihoods Poverty Reduction Vulnerable |
| 4. | There are many aspects of the project design of SPARC which have contributed to efficiency, effectiveness and successful achievement of project outcomes. To begin, actions taken at the community level in Indonesia must be linked to enabling policies at the district, province and national levels. Recognizing this, SPARC had the MoEF Directorate for Climate Change Adaptation as its implementing partner, which has an excellent knowledge of and a lead responsibility for the implementation of national climate change policies, including the National Action Plan on Climate Change Adaptation (RAN-API). The MoEF provides the authority necessary for SPARC to work with provincial and district governments developing new capacity in CCA planning, leading to the development and introduction of climate resiliency policies into local Mid-Term Development Plans (RPJMD). Of particular importance is the lead role of MoEF in the Climate Village Program or Proklim which served as the focal point for SPARC activities in project communities. Also as the national lead for CCA, MoEF is the relevant body to replicate the success of SPARC in other provinces within Indonesia Tag: Climate Change Adaptation Implementation Modality Project and Programme management Country Government Institutional Strengthening Policy Advisory |
| 5. | Within the province of NTT, SPARC worked closely with and through the provincial and district planning offices (Bappeda). The Bappeda offices provided an excellent focal point for the SPARC project given the ability of these offices to facilitate the networking required to engage relevant government bodies (e.g. environment, agriculture, disaster risk reduction, infrastructure, meteorology, etc.) and to provide leadership for the development, advocacy and adoption of CCA policies in RPJMD. Tag: Local Governance Project and Programme management Advocacy Coordination |
| 6. | Finally, while SPARC identified the need for capacity and policy development in government (Outcome 1) it also supported the concept of locally appropriate, community supported CCA actions aimed at those persons who are most in need (Outcome 2). In this regard the SPARC project began by developing a Village Vulnerability and Climate Risk Index for NTT. The index was used to select districts and villages that were included in the project and the index may be used to continue and replicate the work of the SPARC project in the remaining villages of the province. Tag: Climate Change Adaptation Environmental impact assessment Vulnerable |
| 7. | In addition to the key stakeholders of the SPARC project (MoEF, Provincial and District Bappeda, and the project communities) there were a number of other stakeholders who were engaged in the project to assist in addressing the diverse nature of actions required to address climate change. These included relevant environment, agriculture, meteorology, DRM, and infrastructure departments/agencies of the government across different levels including the national, provincial and district level providing technical assistance and capacity building. Additionally, non- government stakeholders included media for CCA advocacy, Bank NTT for private corporate funding of CCA and the University of Nusa Cendana (UNDANA) located in Kupang, NTT to develop a “climate and development” program providing CCA capacity building. Tag: Partnership Country Government Capacity Building Technical Support Civil Societies and NGOs National Institutions Private Sector Regional Institutions |
| 8. | The ProDoc assumptions identified for the project objective and two project outcomes proved to be realistic. The project did benefit from the support it received from the government, and this was most notable among Bappeda staff who demonstrated an understanding of climate impacts experienced by communities and the role government can play in assisting communities with CCA actions. As assumed government staff and communities were very receptive to the capacity building and technical support provided by SPARC and they were willing to collaborate, organize and implement CCA actions which provided benefits to the participating communities. Tag: Climate Change Adaptation Capacity Building Technical Support Civil Societies and NGOs Regional Institutions |
| 9. | The risk of competing government priorities identified in the ProDoc is very real; nonetheless the government appears to take seriously the well-being of rural communities and are aware of hardships faced by subsistence livelihoods made more difficult by an increasingly variable climate. Each of the various government stakeholders involved in SPARC showed a strong commitment in their respective contributions and willingness to collaborate in a multi-sectoral CCA program for rural communities. Staff turnover or transfer resulting in a loss of CCA capacity built up during SPARC remains a high risk as identified in the ProDoc. This concern was articulated by project and government staff during the field evaluation. To some extent SPARC could have addressed this through additional capacity building of government towards the end of the project. Tag: Ownership Risk Management Country Government Capacity Building National Institutions Regional Institutions |
| 10. | SPARC builds on existing government programs that support CCA (e.g. RAN-API), it supported planning mechanisms such as provincial and district planning committees (Bappeda) and their development of medium term development plans (RPJMD) where CCA policy was incorporated. SPARC supports the bottom-up approach to development promoted in Indonesia and this facilitates working directly with communities at risk through financial support available from the Village Development Fund (VDF). SPARC is in line with UNDP Country Program (2011-2015 and 2016-2020) outcomes identified for climate change in regard to capacity building and policy development. Tag: Climate Change Adaptation Programme Synergy Capacity Building |
| 11. | There were many government and non-government partners involved in SPARC, the field evaluation observed some significant alignment of the project with other development projects operating in NTT. Some of the alignments observed in the field and in PIR reports which have occurred over the course of the project include:
Tag: Harmonization Partnership Programme Synergy |
| 12. | Project Implementation
The SPARC project started effectively in August 2013 and received “satisfactory” implementation progress ratings from the UNDP Country Office Programme Officer and UNDP GEF Technical Adviser in PIRs from 2014 to 2018. Tag: Effectiveness Implementation Modality |
| 13. | Adaptive Management of Project DesignProject documentation on the adaptation of SPARC activities over the course of the project indicate the Project Management Unit (PMU) was quick to identify potentials barriers to project success and responded with innovative management responses ensuring the success of project outcomes. Tag: Innovation Programme/Project Design Project and Programme management |
| 14. | Three examples of adaptive management measures undertaken by the project as drawn from PIRs include the following: Originally, it was envisaged that the SPARC project would influence the formulation of RPJMD and integrate climate change concerns into it. However, a slight delay in the project start up proved that it was difficult to do so in a substantive manner. Nonetheless, the project team’s technical advisory resulted in the inclusion of references to climate risks, though they were somewhat generic, which provided a legal basis and future scope for the SPARC project to mainstream climate change concerns into the RPJMD’s annual action plan and potential revision at the midterm of the RPJMD cycle. Tag: Climate Change Adaptation Implementation Modality Project and Programme management Technical Support |
| 15. | The project districts selected through the Village Vulnerability and Climate Risk Index posed implementation challenges due the fact that the districts were located on three different islands all requiring air or boat transport from the provincial capital in Kupang and they had more limited telecommunication contact. This was considered to have potential negative impact on the efficacy and sustainability of planned capacity development activities in the districts due to the logistical challenges which would reduce the frequency and coverage of the capacity development activities. To overcome this challenge, the SPARC project decided to put a more emphasis on building capacity at the sub-national level in Kupang where substantial capacity development could be provided to government staff. Government staff in Kupang receiving the full benefit of SPARC capacity development could in future utilize and transfer their knowledge assisting district government staff. Working in Kupang also led to a curriculum development partnership with UNDANA which created a graduate program targeting local government planners, and increased training provided to enhance the capacity of local agricultural extension officers, irrigation officers, and technical officers in key sectors, and village officials. Tag: Disaster Risk assessments Challenges Local Governance Implementation Modality Country Government Capacity Building Operational Services Regional Institutions Vulnerable |
| 16. | In the third year of the project several factors led Project Board Members to recommend expansion of the project into 12 additional villages within the existing project districts and to expand into an additional 13 villages in a new district, East Manggarai. The board also recommended a one year extension of the project to ensure successful project expansion. The positive results supporting project expansion included the successful incorporation of CCA into the RPJMD for NTT and RPJMDes for the three targeted districts, substantial capacity development of and networking among government staff, the successful implementation of CCA activities in project communities and the availability of unspent funds. Tag: Climate Change Adaptation Implementation Modality Project and Programme management Capacity Building |
| 17. | Project Monitoring and EvaluationThe design template developed for Quarterly Monitoring Reports (QMR) and Internal Project Assurance Reporting (IPAR) was comprehensive and provided the opportunity for clear tracking of project activities and finance against targets established in the ProDoc. The QMR table included information in columns under the following headings; Expected Output, Atlas Activity, Budget (for the year), Expenditure, and Link between Activity and Output. The QMR/IPAR provided for the TE covered the period 2015 to the 3rd quarter of 2018.
Tag: Monitoring and Evaluation Results-Based Management |
| 18. | Review of QMR tables showed that, for the most part, Sections 1 Budget Delivery against Outputs and Section 2 Analysis of progress at Output level did provide detailed reporting on project activities that permit a clear understanding of the link between project activities and project outputs. The QMR however rarely provided information on the expenditure of atlas activities reported on. Also the QMR tables do not provide an explanation for atlas activities that are shown as planned and budgeted for a given year but apparently not carried out (i.e. no information was reported). The lack of information regarding expenditures for activities that were carried out prevents an assessment of project over or under spending against proposed annual budgets. Tag: Challenges Human and Financial resources Monitoring and Evaluation |
| 19. | The detailed documentation of project activities provides substantial information regarding project outputs, including the number and type of activities, the number of participants engaged in project activities and a breakdown of the number of men and women engaged (occasionally gender was not recorded). While some of this information can be found summarized in QMR Section 2: Analysis of progress at Output level and in annual PIR, the reporting of project M&E could be enhanced by providing a more comprehensive summary table listing project activities, participants, and gender reported within the 30 to 40 page QMRs. Tag: Gender Parity Monitoring and Evaluation |
| 20. | Portions of the QMR do not provide detailed reporting in files provided for the TE. In particular, Section 1: Budget Delivery against Outputs, components 3 and 4; Section 3: Cross-Cutting Issues; Section 4: Lessons Learned Log; and Section 5: Comments and questions from the Project Team. It is assumed Section 6 Clearance, has been completed and signed copies are available in UNDP CO. Sections of the IPAR are also incomplete. Tag: Challenges Monitoring and Evaluation |
| 21. | SPARC monitoring does not evaluate the success or potential negative impacts of the wide range of CCA and AIG activities supported by the project. In particular it would be beneficial to include participatory monitoring and evaluation, with indicators developed, monitored and reported on by beneficiaries. Knowledge of project success in terms of increased crop yields, income generation, ground water levels, etc. is important to inform successful project replication and scaling up. For example, SPARC provided infrastructure and capacity development to initiate community drip irrigation systems. Communities reported vegetable gardening during the dry season in areas that were not previously used. In addition, to providing nutritious food for the community and increasing food security, surplus crops were sold providing economic benefits. Community members interviewed were proposing to significantly expand the area under drip irrigation in the near future. Enhanced water security for communities was an important output of outcome 2 and the project provided infrastructure in the form of solar powered water pumps, piping, storage and distribution systems that dramatically improved the livelihoods of participating communities. Unfortunately little is known regarding the sustainability of ground water resources and communities reported the drying of wells in the past and the disappearance of springs. Surface and groundwater resource sustainability in a karst (limestone) landscape can change dramatically over time as new pathways for water movement evolve, including both the loss and re- appearance of available water. Community monitoring of ground water levels in wells may help to inform the variability and sustainability of the available water resources. Tag: Water resources Civic Engagement Monitoring and Evaluation |
| 22. | Project Coordination and Operation
Management arrangements of the project had some challenging logistics. The UNDP implementing partner MoEF, was seated within the national government in Jakarta and the SPARC project office implementing project activities in NTT province was located in Kupang, a two hour flight from Jakarta. The implementing partner expressed a desire to have more oversight over staff implementing field activities which would have used additional project budget for travel and accommodation. Secondly the project office in Kupang coordinated activities that required flights to district government offices and substantial car travel to project communities. The project office identified difficulties of travelling to some districts which at times were not possible during the rainy season due to limited access to small uncontrolled airstrips and restricted car travel to some project communities that were cutoff due to landslides caused by heavy rains or earthquakes. Tag: Challenges Human and Financial resources Oversight Project and Programme management Operational Services |
| 23. | Despite the logistical challenges, the role of MoEF was effective in its ability to provide authority for the project office to work effectively with the provincial and district governments. And as SPARC intentionally selected communities based on high vulnerability, it was inevitable that these communities were located in areas that are difficult to access. Working with local implementing partners (facilitators) in each district was an effective strategy for the project office to remain in contact with government and community stakeholders participating in the project. Tag: Civic Engagement Local Governance Implementation Modality Project and Programme management Country Government Coordination |
| 24. | Project FinanceThe SPARC project was provided with a $5M grant from the GEF SCCF, added to this were grants made available form UNDP ($100,000), Bank NTT ($155,565) and NGI ($35,600) for a total budget of $5,291,165. In-kind co-financing of SPARC was $74,210,654 (see co-financing Table 3 and discussion below). The SPARC project summary table is provided below:
Tag: Efficiency Human and Financial resources Operational Efficiency |
| 25. | Project documentation of financing provided in QMR/IPAR and PIR reports does not provide sufficient detail to permit verification of planned budgets as outlined in the ProDoc. The graph below updated to December 2018 based on Atlas/CDR reports shows cumulative expenditures over the course of the project from project inception in 2013 to closure in 2018. The approved budget (ProDoc) of $5M was originally to be disbursed by December 2016 however this was extended to December 2018 midway through the project, based on a recognition that SPARC was ahead of target achieving project outcomes and excess budget was available to extend the implementation of community CCA activities. The project board approved an expansion of the target area of the project and an extension of the project to December 2018. The graph also shows the Atlas approved budget for December 2018 exceeds the original ProDoc approved budget as a result of grants made available through co-funding as discussed below. It is anticipated the total budget will be utilized as the project is still active completing final closure steps.
Tag: Human and Financial resources Operational Efficiency |
| 26. | The SPARC project benefited from co-financing grants and in-kind support provided by UNDP, grants provided by Bank NTT and NGI as well as in-kind support from the NTT provincial government as shown in the table below. The grant provided by Bank NTT was made through a unique collaboration between UNDP and Bank NTT, whereby SPARC was provided a substantial grant of $155,565 from Bank NTT’s CSR fund to be utilized for CCA secure livelihoods activities in project communities such as micro-hydro and solar water pumps. The grant provided by NGI resulted from collaboration with the SPARC project with the $35,600 utilized for CCA soil improvement activities. Tag: Government Cost-sharing Human and Financial resources |
| 27. | The very large amount of in-kind support provided by the NTT government represents a province-wide program called “Anggur Merah” (Red Wine) which ran from 2011 to 2018. Anggur Merah provided development assistance grants from the NTT Regional Budget amounting to approximately $58M distributed to 3,270 villages throughout NTT to support economic development aimed at improving the welfare of local communities. Some SPARC villages did access Anggur Merah to strengthen CCA activities, however detailed accounting was not available for evaluation. The Anggur Merah program was receptive to the SPRAC project and in some cases funding from Anggur Merah has directly supported SPARC actions in project communities and it has likely supported similar actions in other communities in NTT Tag: Government Cost-sharing Human and Financial resources |
| 28. | Substantial in-kind support to SPARC provided by national, provincial and district government staff participating in the project is not currently reflected in the co-financing table given the unavailability of an assignable monetary value. The success of SPARC is in large measure attributable to the enthusiastic participation of government staff in all aspects of the project, including project design, capacity building provided technical experts, policy development and advocacy at the national, provincial and district levels, and the implementation, monitoring and support of CCA actions in project communities. Tag: Government Cost-sharing Human and Financial resources |
| 29. | Project ResultsOverall resultsSPARC has produced satisfactory measurable development change as a result of the project inputs and activities. There has been effective use of financial, human and material resources with satisfactory implementation, expansion of the project at the mid-term and the successful achievement of all project outcomes. The implementation actions effectively mobilized government and community stakeholders resulting in active participation, new forms of collaboration, advocacy of the SPARC project model and commitments to sustain the project outcomes. Tangible outputs included new CCA policies and actions plans incorporated into government planning documents and CCA activities enhanced or introduced and adopted by project communities increasing their resilience to a more variable and changing climate. Tag: Climate Change Adaptation Resilience building Effectiveness Advocacy Policy Advisory |
| 30. | The challenge will be to sustain the institutional capacity created in government and for government to replicate the SPARC model in the villages, districts and provinces which did not directly participate in the SPARC project. Tag: Sustainability Institutional Strengthening Challenges |
| 31. | SPARC outcomes also provide sustainable environmental benefits by introducing renewable forms of energy use (solar water pumps), water conserving agricultural methods (mulching, drip irrigation) and tree nursery and tree planting programs. Tag: Forestry Energy Effectiveness |
| 32. | Relevance
Government staff at all levels indicated the impact of climate change on rural agricultural communities in Indonesia is of serious concern and that it is the responsibility of government to provide assistance. A statement from the Head of a District Bappeda is consistent with the responses received:
Tag: Agriculture Rural development Climate Change Adaptation Country Government Vulnerable |
| 33. | Government staff acknowledged the importance of communication and coordination both horizontally (i.e. across National Ministries, or Provincial Offices or District Agencies) and vertically (e.g. between National, Provincial and District governments). The SPARC project was often referred to as the “SPARC Model” in reference to the multi-sectoral and multi- level approach to examining issues and finding effective and innovative shared responses incorporated into Action Plans for implementation by Provincial and District staff.
Tag: Communication Implementation Modality Coordination |
| 34. | The “SPARC Model”, highly touted by government staff involved in the project, provides a very relevant approach to CCA needed in remaining villages within NTT and villages in the other 32 Provinces of Indonesia Tag: Effectiveness Implementation Modality |
| 35. | At the village level community members expressed a general lack of development opportunities and identified a variety of impacts of climate change on subsistence agriculture, health, water supply and infrastructure. The formation of community groups, such as Kemas Proklim to more effectively manage the negative impacts of climate change and the technical and financial assistance provided for CCA and AIG activities were identified as SPARC project activities that were highly relevant to their needs. Tag: Agriculture Crop production Climate Change Adaptation Relevance Technical Support Vulnerable |
| 36. | Effectiveness
The outputs identified above in Section 2.2 are reported to have been achieved in the final Project Implementation Report (PIR) prepared in June, 2018. These results are substantiated by project results presented in the Adaptation Monitoring and Assessment Tracking (AMAT) tool for CCA projects prepared in June 2018 and the Quarterly Monitoring Reports (QMR) and Internal Project Assurance Reports (IPAR) prepared for the SPARC project. Project documentation provides reasonable evidence of the achievement of all SPARC outputs and in large measure the evaluation mission confirmed achievement of SPARC. Nonetheless, interviews conducted during the evaluation mission showed in some cases the quality and sustainability of some outputs could be enhanced (see the evaluation of Outcome 2 outputs below and the evaluation discussion of sustainability). Tag: Effectiveness Implementation Modality Monitoring and Evaluation Quality Assurance Sustainability |
| 37. | In regard to the outputs of Outcome 1, field mission visits with 16 government staff representing national, provincial and district government offices support results reported by the project, particularly the recognition of the need for and benefits from multi-stakeholder dialogue and planning to address community CCA. At the provincial and district levels this has been institutionalized through the establishment of multi-stakeholder planning committees, the incorporation of CCA into Medium Term Development Plans at the provincial and district (RPJMD), village (RPJMDes) levels and the development of Action Plans that provide direction for activities implemented annually.
Tag: Climate Change Adaptation Effectiveness Local Governance |
| 38. | In regard to the outputs of Outcome 2, the formation of community groups facilitated by project staff working with community members, was identified by implementation staff and also by community members as one of the most challenging implementation actions of the SPARC project and also the most significant positive outcome in terms of the mobilization of community members to identify and confront CCA. For facilitators there was the issue of building trust with the community and encouraging community members to recognize the benefits of working collaboratively through group facilitation meetings. Community members indicated there had become the habit of people working individually, pursuing personal or household activities to achieve economic benefits. |
| 39. | SPARC built on The Climate Village Program (Kemas ProKlim), initiated by the MoEF as an effort to strengthen local initiatives related to climate change. Through Kemas ProKlim, MoEF gives recognition to the active participation of communities that are engaged in integrated climate change mitigation and adaptation efforts at the local level. Funding for Kemas ProKlim initiatives was provided by SPARC and is also available through the VDF and District government. SPARC targeted capacity building of Kemas Proklim leaders and members further enhanced the CCA capabilities of project communities. Community mobilization was and continues to be an extremely important foundational step supporting the success of Outcome 2 outputs noted below. Tag: Climate Change Adaptation Civic Engagement Programme Synergy |
| 40. | Output 2.1 set a target for 300 communities to develop a “community based climate risk information system”. Supporting this output, some communities participated in technical training and formed “climate risk information centres” to record and disseminate agricultural-tailored climate information to local farmers. One participating Kemas ProKlim leader reported sending posting regular reports to BKMG and sharing feed from BKMG to local farmers via smartphone messaging. It is not evident from project documents or from the field evaluation that 300 communities developed a formal “community based climate risk information system” to be used by communities as an ongoing risk assessment system. In reality the ongoing work of Kemas Proklim such as, CCA leadership, community discussions, future scaling up of existing CCA activities and the exploration and adoption of new CCA activities, may be considered an important part of a community’s climate risk information system.
Tag: Climate Change Adaptation Disaster Risk assessments Civic Engagement Data and Statistics |
| 41. | Output 2.2. set a target for 150 communities to adjust subsistence farming practices. Agriculture is generally the dominant activity in target communities. SPARC project documents and field evaluation interviews with farmers confirmed a strong interest in and benefits from training provided, technical support in climate smart agriculture such as biochar, the provision of infrastructure such as irrigation and plastic mulch, and the provision of climate adapted / drought resistant seed varieties. The June 2018 PIR noted that 175 community groups adopted more resilient agricultural practices. Tag: Agriculture Crop production Rural development Climate Change Adaptation Resilience building Effectiveness |
| 42. | Output 2.3 set a target of 100 communities diversifying income and becoming less sensitive to climate change, which was well exceeded based on a the 2018 PIR that reports 253 community groups participating in SPARC promoted Alternate Income Generating (AIG) activities. These were largely represented by providing direct grants to purchase inputs for existing income generating activities (e.g. livestock rearing, seaweed farming, fishing, traditional hand weaving, etc.), as well as in some cases training and grants for new income generating activities (e.g. bokashi fertilizer production, fiber-boat production, mattress production, etc.) and through the establishment of credit unions and cooperatives. Some AIG activities promoted by SPARC may not make communities “less sensitive to climate change”, for example if households develop a heavy dependence on seaweed farming this may put families at risk due the potential impact of climate change through sea temperature change, sea level rise, and extreme storm events. SPARC project documents do not outline how or if AIG activities were assessed to ensure they met the criteria of “sources of income less sensitive to climate change”. Tag: Climate Change Adaptation Resilience building Jobs and Livelihoods |
| 43. | Output 2.4 set a target of 50 communities with improved water infrastructure and resource management. This target was exceeded with 90 community groups reported to have benefited from SPARC activities. SPARC strategies targeting severe water scarcity for human use and for rain-fed agriculture included: the installation of solar powered pumps with piping bringing water to reservoirs for distribution to the community; the digging of infiltration wells to enhance groundwater recharge; the construction of water catchment dams; and the provision of training and infrastructure for water efficient agriculture and drip irrigation, etc. During the field evaluation, feedback from government staff and community members expressed a clear recognition of a long standing water resources problem, which is now exacerbated by climate change and the ability to improve water resource management through interventions implemented by the SPARC project. Tag: Agriculture Agriculture water resources Climate Change Adaptation Water resources |
| 44. | Output 5.1 identified the need for a coordinated and integrated approach to confront CCA and DRR resilience building. An integrated approach would also better utilize government and non- government staff and funding resources available to address the needs of CCA and DRR. With support from SPARC a workshop was organized and a report titled Climate Change Adaptation and Disaster Risk Reduction (CCA-DRR) Convergence was released in December 2018. The report provides a comprehensive assessment of the current CCA and DRR policy and implementation frameworks and makes recommendations for an integrated CCA-DRR planning approach as shown in the figure taken from the report. The report outlines the policy, institutional and funding instruments required and a set of indicators to measure the achievement of integrated CCA-DRR development planning. Implementing the recommendations of the CCA-DRR Convergence report will require strong leadership and significant effort by a committed government agency. Unfortunately the role of a leading sector or coordinating ministry/agency has yet to be established to guide all relevant ministries and agencies related to CCA-DRR. Tag: Climate Change Adaptation Disaster Risk Reduction Resilience building Public administration reform Policy Advisory |
| 45. | Outputs 5.2 established a target to develop relevant maps and data resources to enable application of CCA-DRR convergence initiatives in 6 villages in 3 districts. Training on converging CCA-DRR for provincial and district government Disaster Management Agency (BPBD) officials and partner NGOs was conducted in three districts. Participants were trained to develop climate- related risk/disaster data and in using Global Positioning System (GPS) to show the areas that are prone to climate related disaster at village level. The information is intended for use in the development planning process.
Tag: Climate Change Adaptation Disaster risk management Disaster Risk Reduction Data and Statistics Civil Societies and NGOs |
| 46. | Output 5.3 established a target to provide local NGOs with training to allow them to facilitate communities in the development of CCA-DRR measures. Local NGOs participated in multiple SPARC training sessions and they were assigned to each Kemas Proklim to provide ongoing capacity building support. Despite the training provided to local NGO staff, not all of those interviewed during the evaluation mission demonstrated a comprehensive understanding of CCA, DRR and the convergence of these. Tag: Climate Change Adaptation Disaster Risk Reduction Capacity Building Civil Societies and NGOs |
| 47. | Efficiency
The SPARC project is considered to have been very efficient, based on the project’s ability to achieve the intended outputs of Outcomes 1 and 2 and given the fact that the project underwent a substantial expansion at the midterm from 21 villages in three districts in phase I to a proposed additional 12 villages in three districts in phase II and an additional 13 villages in a new, fourth district. It should be noted that during the evaluation some concerns were raised regarding the addition of villages at the midterm, in particular the fact that while the first 21 villages benefited from five years of support and interaction with the SPARC project, the additional 25 villages were involved in the SPARC project for only two years and the limited support and interaction may reduce the sustainability of the outcomes. Tag: Efficiency Implementation Modality Programme/Project Design |
| 48. | It is difficult to evaluate the cost effectiveness of the SPARC project; however, a conservative assessment suggests there are likely considerable financial benefits to individual households and communities resulting from SPARC support of AIG. For example, during the field evaluation one household reported an income of approximately 100 M IDR (US$7,000) per year from seaweed cultivation which was re-introduced to the village by the SPARC project. Another household reported an annual income of approximately 4 M IDR (US$300) per year from the sale of traditional weaving, some of which may be attributed to support from the SPARC project which encouraged a revitalization of traditional weaving. Income generation was also reported from vegetable gardening, livestock rearing, biochar production, and fisheries, many of which were new sources of income generation for communities. Additional incremental household income may also be inferred from reported increased yields of crops using improved seeds and agricultural methods, time saving resulting from improved water supplies that reduce the time required to obtain water, and improved health as a result of improved nutrition suggesting reduced health care costs. Tag: Water resources Effectiveness Health Sector Nutrition Jobs and Livelihoods Poverty Reduction |
| 49. | If a conservative estimate of 5M IDR generated from AIG each year is used for each of the 9,800 households that participated in SPARC, the total cash benefits derived would be equivalent to 49,000M IDR or approximately $3.5 M USD per year. This would suggest the $5M UNDP GEF investment provided over five years to the SPARC project is likely to be recovered in less than two years based on the future cash benefits to participating households. Tag: Effectiveness Global Environment Facility fund Jobs and Livelihoods |
| 50. | Country ownershipSuccess of the SPARC project can be attributed in large measure to the commitment and endorsement by national, provincial and district government staff. A high level of country ownership is reflected in the adoption the SPARC model in policies, planning documents and action plans promoted by all levels of government. Indonesia’s Village Law (Law No.6 of 2014) and the associated VDF provides vital ongoing government financial commitment to SPARC actions that will continue to support the communities directly involved in the project and which can support actions in new communities through scaling up initiated by district and provincial governments. Further, district BAPPEDA staff interviewed indicated that relevant CCA VDF proposals submitted by communities to the district government would receive approval. Tag: Ownership Country Government |
| 51. | SPARC also inspired support form Bank NTT, which for the first time utilized their Corporate Social Responsibility fund to provide co-financing to a UNDP supported project. Tag: Partnership Private Sector |
| 52. | SPARC also supported an initiative by the University of Nusa Cendana (UNDANA) resulting in the establishment of a post-graduate level elective program titled “Climate and Development” within the Environmental Science Department. To date 15 students have already been accepted into the course, many of whom are government staff who will return to utilize an important new understanding of CCA in government programs. Tag: Climate Change Adaptation Education Civil Societies and NGOs |
| 53. | Mainstreaming
The SPARC project selected NTT within Indonesia as a province characterized by higher poverty and high climate change vulnerability resulting from a challenging communication network, infrastructure which is generally underdeveloped and a vulnerable subsistence agricultural economy associated with the 111 islands which make up much of the remote archipelago of the province. Further, the SPARC project selected districts and communities within NTT based on the village vulnerability and climate risk index which is based on higher poverty levels and climate risks. In this way the SPARC project can be said to have made a substantial contribution to poverty alleviation and the prevention and recovery from natural disasters to communities that are most in need within Indonesia. Tag: Disaster Recovery Disaster Risk assessments Natural Disaster Impact Poverty Alleviation |
| 54. | The successful achievement of Outcome 1 has made a substantive contribution to improved governance of village communities through engagement, capacity development and the adoption of a multi-sectoral governance model at the national and sub-national level. The governance model adopted enabled identification of climate risks and vulnerabilities and helped collaboratively implement locally appropriate CCA. At the national level MoEF as the implementing partner for SPARC, will take on the responsibility for replication and scaling up of CCA in new provinces and districts using a multi-sectoral approach. At the sub-national level NTT province and the four district governments will continue to implement multi-sectoral planning for CCA through their Medium Term Development Plans. SPARC also supported research and actions to converge CCA–DRR programs to enhance government services to communities. And with support from SPARC a new “climate and development” graduate program at the University of Nusa Cendana in Kupang, NTT is training government staff and others in the field of multi-sectoral planning for CCA and DRR. Tag: Climate Change Adaptation Disaster risk management Impact Rule of law Education |
| 55. | SPARC promoted gender equality by ensuring the involvement of both men and women in the decision-making and planning processes that developed community proposals for adaptation actions. Within family member groups that benefited directly from the SPARC project, 20,607 members are female (50% out of a total of 40,972 direct beneficiaries). Women members of family groups were empowered through project activities such as: increased time availability to undertake income generating activities (enhanced vegetable gardening for nutrition and income generation, mattress production, livestock rearing, traditional ikat weaving and sewing); a variety of trainings to improve knowledge and skills; improved access to resources (land and water); and improved networking to access district and provincial government resources as well as financial resources.
Tag: Gender Equality Gender Mainstreaming Women's Empowerment Jobs and Livelihoods Poverty Alleviation Capacity Building |
| 56. | During the TE field mission where new water distribution systems were introduced by the project, the women interviewed reported significant time saving, given the amount of time previously required to reach the water source. One women reported that previously it was necessary to wake up at 3:00 am to fetch water, now with water available within the community she has more time for weaving (an income generating activity). In another case women reported having water available meant they were able to establish a vegetable garden within the community. In one community women have taken the lead to initiate expansion of the water supply system installed by SPARC, creating new additional water supply points for households. Where drip irrigation was introduced women reported that in previous years they grew only maize and “salad”, whereas now they grow so many different kinds of crops including, tomato, eggplant, chili, watermelon and long bean which can be sold in the local market for income.
Tag: Women's Empowerment Jobs and Livelihoods Poverty Alleviation |
| 57. | The inclusion of children and youth groups was not a focus of the SPARC project. Based on the evaluators experience reviewing other project in Asia there are good opportunities working with children and youth as agents of change, they are more receptive to new ideas, they are innovative, they become great advocates of CCA and DRR and as the generation that will face more serious CC impacts there is tremendous value including children and youth in all CC projects.
Tag: Climate Change Adaptation Disaster Risk Reduction Youth |
| 58. | SustainabilityThe evaluation of sustainability is complex due to the fact that it is dependent on a large number of factors, some of which are within the control of a project and others beyond the control of the project. In addition, unforeseen changes can dramatically impact potential sustainability. The sustainability of the SPARC project outcomes is assessed below in the context of key stakeholders and beneficiaries at the following four scales:
Tag: Sustainability National Rural Local Governance Sustainability |
| 59. | The sustainability of the SPARC outputs in the 456 project communities is considered “likely” based on the following observed and reported results:
Tag: Sustainability Sustainability |
| 60. | Key Risks to Sustainability
Tag: Natural Disaster Water resources Sustainability Sustainability |
| 61. | The sustainability of SPARC outputs in the four project districts is considered “moderately likely” based on the following observed and reported results:
Tag: Climate Change Adaptation Sustainability Local Governance Sustainability |
| 62. | Key Risk to Sustainability
Tag: Sustainability Risk Management Strategic Positioning Sustainability |
| 63. |
Key Risks to Sustainability
Tag: Sustainability Local Governance Risk Management Sustainability |
| 64. | Key Risk to Sustainability
Tag: Sustainability Human and Financial resources Risk Management Sustainability Country Government |
| 65. | In summary, the sustainability of water security, food security and livelihoods within the 46 communities where SPARC operated is likely. The sustainability of similar initiatives replicated in new villages, districts and provinces through future CCA planning and scaling up of multi- sectoral planning is moderately likely. This is due to the risks identified for national, provincial and district governments which impair their ability to undertake the necessary scaling up and replication of the SPARC Model beyond the 46 communities, three districts and one province reached by the SPARC project. Tag: Food Security Water resources Sustainability Sustainability Jobs and Livelihoods |
| 66. | The SPARC project could have more extensively assisted responsible stakeholders (district, provincial and national government staff) in the development of implementation strategies, action plans and/or road maps intended to replicate SPARC in new communities after the end of the project (December 2018). The needs identified to strengthen scaling up / replication (i.e. sustainability) include the following:
Tag: Sustainability Resource mobilization Strategic Positioning Sustainability |
| 67. | Impact
Based on the LogFrame for SPARC the following table outlines a Theory of Change (TOC) used to undertake an analysis of project impacts. The impact noted in the table is based on the project objective identified in the ProDoc and is considered equivalent to the Global Environmental Benefit which may be derived from the project. A second impact, considered implicit though not articulated in the project design, has been added based on the role of MoEF as the implementing partner in the project and their responsibility as Directorate of Climate Change and Adaptation. Tag: Impact Programme/Project Design Results-Based Management Theory of Change |
| 68. | Qualitative assessment of SPARC’s TOC is presented in the table below along with the following ratings which are based on desktop and field investigations of the TE: Not achieved (0) - the TOC component was not explicitly or implicitly identified by the project, and/ or very little progress has been made towards achieving the TOC component, and the conditions are not in place for future progress Poorly achieved (1) there are no appropriate mechanisms set out to achieve the TOC component after SPARC’s UNDP GEF funding has ended, and/ or very little progress has been made towards achieving the TOC component, but the conditions are in place for future progress should new support be provided to complete this component. Partially achieved (2) the TOC component is explicitly recognized and the mechanisms set out to achieve it are appropriate but insufficient (e.g. there is no clear allocation of responsibilities for implementing the mechanisms after SPARC UNDP GEF funding ends). Moderate and continuing progress was and is being made towards achieving the TOC component, although there is not yet a strong basis assuring the eventual delivery of the intended impact (Global Environmental Benefits). Fully achieved (3) the TOC component is explicitly recognized and appropriate and sufficient mechanisms to achieve it are apparent (e.g. specific allocation of responsibilities and financial and staff support is available after SPRAC UNDP GEF funding ends), and/ or substantial progress has been made towards achieving the TOC component and there is strong assurance of eventual delivery of the intended impact (Global Environment Benefits). Tag: Effectiveness Theory of Change |
| 69. | The SPARC project has also demonstrated improvements in ecological status and reductions in stress on ecological systems. Most notable of the positive environmental impacts of the project are:
Tag: Agriculture water resources Energy Impact |
| 70. | In the original project plan the Village Vulnerability and Climate Risk Index and subsequent participatory community consultations would lead to the selection of locally appropriate CCA actions. Upon reflection the PMU recognized this would result in a long gap between project launch and the provision of benefits to communities. Concerned about diminishing local interest and ownership if no concrete actions are demonstrated for such a long period of time, the project team agreed with the MoEF and BAPPEDA to identify 25% of target communities and move ahead with “no-regret” community investments. This would ensure that local stakeholders continue to observe tangible actions from the first year of the project, which is an important factor for uninterrupted community buy-in. Tag: Climate Change Adaptation Disaster Risk assessments Implementation Modality Project and Programme management Vulnerable |
| 71. | The sustainability of SPARC outputs within NTT province is considered “moderately likely” based on the following observed and reported results:
Tag: Climate Change Adaptation Sustainability Local Governance Sustainability |
| 72. | The sustainability of SPARC outputs nation-wide is considered “moderately likely” based on the following observed and reported results:
Tag: Sustainability Sustainability |
| 73. | ASSESSMENTSThe four principal evaluation questions – Relevance, Effectiveness, Efficiency, and Sustainability in the TOR were expanded and examined during the evaluation under twenty questions and issues, and are reported below. The TE examined all 18 activities across the planned outcomes/components, with comparisons between the initial plans, the recommendations made by the MTR with the results at the end of 2015 together with the perceptions of key stakeholders. This is described in tables 7 and 8. RELEVANCEThe project is in full consonance with the main objectives of the GEF focal area and the priorities of enhancing national ownership of climate change activities and to strengthen countries’ capacities. The project goals are also highly congruent with the global agenda on climate change mitigation and to the environment and development priorities at the local and national levels as declared by the RMI in multiple policy statements and goals. The project enabled a small South-South cooperation54, through the exchange of resources, and knowledge with SPC during the implementation of the SPV installations and in the installation of the wind tower. The Marshall Islands remains heavily reliant on imported fossil fuels, harming its environment and weakening its economy. It has reiterated its goal to become free of fossil fuels and with good planning and implementation, this could be possible, with also reduced costs for energy and increased local employment. Its relevance is judged to be high. EFFECTIVENESSThe TE examined all 18 activities across the planned outcomes/components, with comparisons between the initial plans, the recommendations made by the MTR with the results at the end of 2015 together with the perceptions of key stakeholders. This is described in tables 9. It has already been noted that the progress missed all milestones on timeliness. It has also been noted that the evaluation concurs with the MTR that there were too many components and too many ill-defined activities. The evaluation found that after 2012 with the appointment of the second full time project manager the progress improved and many of the activities planned were undertaken. |
| 74. | EFFICIENCYThe project implementation was highly inefficient. The project was designed for a period of five years and is yet to be completed after 8 years. Inefficiencies and delays were encountered at multiple levels in project management, fund disbursements, launching and completion of activities. They were largely due to institutional factors and not attributable to any individual. Similarly, or due to the above reasons, financial planning and management were weak. Supervision, guidance and technical backstopping remained problematic throughout the project. There were the processes of regular PIR, yearly progress and financial reports; monitoring reports and activity reports as specified. But there was inadequate qualitative information and feedback, and active processes used to adjust the project activities and outputs, overcome challenges and make a systemic contribution to the larger goal of improved capacity in RMI to assess and use RE more effectively. SUSTAINABILITYThe congruence of RMI needs in RE and the existence of multiple donor resources focused on RE support the long term continuation and enhancement of the efforts promoted by ADMIRE. On the other hand, the narrow and project related sustainability of the solar home systems provided by donors for rural electrification in outer islands remain in question at the end of ADMIRE (and North-REP), given the lack of solutions for the poor collections of maintenance fees, and the poor maintenance of equipment. Tag: Efficiency Sustainability |
| 75. | GENDERThe project design, implementation and monitoring have been found to have taken into consideration, gender issues and the role of women. The ADMIRE Prodoc does not specifically mention gender as an issue for the project and no special gender considerations were noted. At the same time, a goal of gender equality is to promote an active and visible policy of mainstreaming a gender perspective in all policies and programmes so that, before decisions are taken, an analysis is made of the effects on women and men. While the project has not been focused on analytical work, it should be noted that its focus on SPV for the electrification of rural households would have a number of positive benefits for women and girls (as well as men and boys) because of the reduced use of kerosene for lighting with health and financial benefits and also with an increased ability to use the light in the evening for studies, leisure and improved productivity. ADMIRE also worked with the national NGO for women, WUTMI to enhance productivity through the use of solar driers. It’s approach to gender dimensions, as not narrowly focused on women, could be quite appropriate for a small country, where the President, the Secretary of MRD, the Head of Energy and the Project Manager are all women. But it is recommended that in any future work and drawing the project conclusions, the RMI and UNDP, keep the issue in mind and ensure that the project does consider the potentials for differential impacts by gender. |
| Recommendations | |
|---|---|
| 1 | Project activities ensuring scaling up should be clearly articulated at project design and with actions included in annual plans intended |
| 2 | Incorporate participatory monitoring and evaluation in project design, whereby beneficiaries choose targets and indicators relevant to them, beneficiaries monitor and report of indicators, and identify adaptive strategies to adjust/modify CCA or AIG actions |
| 3 | Supporting existing or new income generating activities should be accompanied by appropriate market chain analysis |
Management Response Documents
1. Recommendation:
Project activities ensuring scaling up should be clearly articulated at project design and with actions included in annual plans intended
to ensure the success of scaling up project outcomes/outputs.
Management Response: [Added: 2019/06/20] [Last Updated: 2020/12/26]
Development of scale-up strategy for a pipeline climate smart agriculture (CSA) intervention in NTT Province that emphasize on increasing the effectiveness and coverage of agricultural
support services.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Inclusion of scale-up strategy in the pipeline CSA programme interventions in NTT Province
[Added: 2019/06/20] [Last Updated: 2020/02/10] |
UNDP CO Indonesia | 2020/01 | Completed | The scale up strategy in the pipeline CSA programme interventions In NTT province has been included. It has been reference for the development of new pipeline. History |
2. Recommendation:
Incorporate participatory monitoring and evaluation in project design, whereby beneficiaries choose targets and indicators relevant to them, beneficiaries monitor and report of indicators, and identify adaptive strategies to adjust/modify CCA or AIG actions
Management Response: [Added: 2019/06/20] [Last Updated: 2020/12/26]
Development of public, private and civil society partnerships strategy for the implementation and monitoring of CSA input for selected crops in NTT Province. The interventions will be
executed through strengthening the capacity of Farmer Producer Organizations (FPOs) to promote more institutionalized and effective farming activities.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Inclusion of PPP strategy in the pipeline CSA programme interventions in NTT Province
[Added: 2019/06/20] [Last Updated: 2020/02/10] |
UNDP CO Indonesia | 2020/01 | Completed | The action was implemented in 2019. History |
3. Recommendation:
Supporting existing or new income generating activities should be accompanied by appropriate market chain analysis
Management Response: [Added: 2019/06/20] [Last Updated: 2020/12/26]
Conduct value chain analysis ranging from the stage of production, postharvest, distribution (including market) and consumption for specific agricultural commodities in NTT province. The VCA
is used to identify gaps in all stages and its specific interventions
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Finalization of Feasibility study on 6 agriculture commodities in NTT.
[Added: 2019/06/20] [Last Updated: 2020/02/10] |
UNDP CO Indonesia | 2020/01 | Completed | Finalization of feasibility study was conducted and has being reference to similar interventions History |