- Evaluation Plan:
- 2017-2021, Jamaica
- Evaluation Type:
- Final Project
- Planned End Date:
- 07/2017
- Completion Date:
- 10/2017
- Status:
- Completed
- Management Response:
- No
- Evaluation Budget(US $):
- 20,000
Final Evaluation of the Strengthening the operational and financial sustainability of the National Protected Area System project
Share
Document | Type | Language | Size | Status | Downloads |
---|---|---|---|---|---|
![]() |
report | English | 1300.37 KB | Posted | 1535 |
![]() |
tor | English | 809.74 KB | Posted | 1198 |
Title | Final Evaluation of the Strengthening the operational and financial sustainability of the National Protected Area System project | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Atlas Project Number: | 00059298 | ||||||||||||||
Evaluation Plan: | 2017-2021, Jamaica | ||||||||||||||
Evaluation Type: | Final Project | ||||||||||||||
Status: | Completed | ||||||||||||||
Completion Date: | 10/2017 | ||||||||||||||
Planned End Date: | 07/2017 | ||||||||||||||
Management Response: | No | ||||||||||||||
Focus Area: |
|
||||||||||||||
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021) |
|
||||||||||||||
SDG Goal |
|
||||||||||||||
SDG Target |
|
||||||||||||||
Evaluation Budget(US $): | 20,000 | ||||||||||||||
Source of Funding: | Project | ||||||||||||||
Evaluation Expenditure(US $): | 34,000 | ||||||||||||||
Joint Programme: | No | ||||||||||||||
Joint Evaluation: | No | ||||||||||||||
Evaluation Team members: |
|
||||||||||||||
GEF Evaluation: | Yes
|
||||||||||||||
Key Stakeholders: | UNDP, GEF, National Environment and Planning Agency | ||||||||||||||
Countries: | JAMAICA |
Lessons | |
---|---|
1. | Without a proactive and open participation of relevant national stakeholders in the design process, especially from those agencies which would be implementers, design will not be relevant nor feasible. Without all national counterparts being proactive and straightforward about their technical capacities, their expected involvement in implementation, the relevance of any part of proposed output/outcome within a country’s national context design fails and often needs to be adjusted if project is to be implemented at all.
|
2. | The mere production of plans, studies, and processes as outputs does not automatically translate into results. Studies, reports, documents and processes need to be accompanied by clear mechanisms that promotes knowledge assimilation, knowledge sharing, and clear-cut mechanisms to inform and promote policy processes for adoption of outputs. |
3. | The roles of different stakeholders cannot be underestimated and as early as possible they should be clearly defined and assimilated from the onset, especially the roles of those stakeholders and institutions that should provide strategic direction (such as project director, committees, board members, management unit). |
4. | For a complex intervention, the project management unit needs to be strong, well-funded and adequately staffed. Staff needs to have individual capacity to manage such a project, and also have needed management and technical expertise for such a unit. |
5. | As it relates to UNDP, a lesson learned is that its guidance and supportive role cannot be taken lightly. When UNDP collaborates with the implementation process through whatever means (such as log frame reform, changes in decision – making structures, bilateral meetings with implementing agency, and so on) implementation improves. |
Findings | |
1. | ANALYSIS OF LFA/RESULTS FRAMEWORK (PROJECT LOGIC /STRATEGY; INDICATORS) Tag: Effectiveness Sustainability Programme/Project Design Project and Programme management |
2. | THREATS, BARRIERS, AND UNDERLYING CAUSES SUSTAINING PROJECT LOGIC/STRATEGY
Barriers were also specifically identified at project design. Among these were inadequate policy instruments related to efficient and effective management (including financial management) of protected areas; limited capacity for management –institutional, individual, operational and policy related management--; an inadequate funding base and resources to finance management instruments, general limited political and public support of PA s as well as weak understanding of the socio – economic benefits of protected areas. Insufficient as well as inadequate coverage of protected areas throughout the country was also identified as a barrier. The Project’s logic and strategy therefore was to confront these issues through specific outputs and expected outcomes that would, plausibly, deal with threats and barriers for adequate protected area management in a sustainable development context in Jamaica. Therefore, in terms of overall logic and strategy the design responded to an adequate rationale and it was designed as a strategic intervention. Tag: Climate change governance Climate finance Biodiversity Site Conservation / Preservation Wildlife Conservation Challenges Programme/Project Design Project and Programme management |
3. | LOGFRAME AND INDICATORS Tag: Climate finance Challenges Effectiveness Impact Relevance |
4. | Overall, the indicators (and underlying these the overall logic of the intervention) link outputs, products, results with the expected outcomes and objectives to a degree. In retrospect, however, some sections there could have been better calibration of the outputs to the outcomes. For instance, in some cases it is not clear how the products will or would lead to effects, impacts or outcomes. As an example, it is not clear how the products such as management plans or business plans link to improved management since it is not explicit how they would be implemented to have effects or impacts. That is, there are weak linkages between products and expected outcomes, or how outputs/products would result as outcomes in some cases. The indicators are in most cases, therefore, output or product indicators and do not attempt to tally outcome or effect. Tag: Effectiveness Impact |
5. | Throughout the results framework alterations, however, no changes were made at the objective and outcome levels. Nevertheless, some stakeholders considered that –even with this reform—the expectations were still deemed too high considering resources, capacity, and time scope of the NPAS Project. That is, although in general the revised indicators in the log frame were more attuned to what realistically could be achieved at the product level, still in general the project was deemed too ambitious in what it was expected to achieve at the outcome level and to the possibilities to lead change so deep as the Project aimed at given the Jamaican policy context. Tag: Challenges Impact |
6. | DESIGN FORMULATION Tag: Challenges Programme/Project Design |
7. | Although most consulted stakeholders who did participate in the design process indicated that they provided inputs to some degree, several made a few sorts of appreciations. First, that the national stakeholders (both government and non – government) needed to be more assertive and have more inputs as to what were the national capacities and eventual commitments that such a project would entail and second that the project should have linked better with national agencies’ timelines and work plans. Furthermore, it is considered that the design process was rushed since it was carried out in a shorter period of time than initially prearranged, which meant that consultations were hasty and not as thorough as needed. Tag: Programme/Project Design Project and Programme management |
8. | The timing of the expected results was not specifically set up at design and at formulation. It is assumed by several stakeholders that if a specific timeline would have been set up, the project’s outputs could have had a greater effect and could have been more effective. For instance, several baseline studies, plans, and documents were drafted at the very end of the NPAS Project, while interventions at the field level that dealt with some of the aspects of these outputs were done before these documents were finalized. Therefore (in part due to a lack of a planned clear sequential pattern and timeline for the outputs, in part due to the delays in implementation) baseline studies/documents were made available to implementers after the specific onsite interventions took place dealing with the issues in these outputs. Tag: Challenges Effectiveness |
9. | Lastly, budgeting as planned in design formulation was deemed inadequate. First, the planned budget for management (specifically for the Project Management Unit) was insufficient. Second, the costing of outputs (primarily consultancies) was also deemed inadequate to meet with the expected products and to attract the quality needed for these products. This was identified throughout the course of the Project, but there were no attempts to make changes. Tag: Challenges Project and Programme management |
10. | PLANNED STAKEHOLDER PARTICIPATION Tag: Challenges Effectiveness Programme/Project Design Project and Programme management |
11. | REPLICATION APPROACH Tag: Challenges |
12. | UNDP COMPARATIVE ADVANTAGE UNDP’s country office in Jamaica has had a key role in the country in developing and managing capacity building programmes and technical assistance projects for natural resource management, climate risk management and adaptation as well as for other more general environmental issues (at the larger project levels as well as with the Small Grants Programme). UNDP’s capital of information, access to expertise, knowledge management capabilities as well as its regional and global positioning and development of similar projects was also part of the agency’s comparative advantage. The experience in human resources development, integrated policy support, institutional strengthening, and participation is further enhanced by UNDP’s ability to draw on experts (regionally and inter – regionally) to propel capacity in protected area management when is not found in – country. Tag: Climate change governance Natural Resouce management Effectiveness Impact Human and Financial resources UN Country Team Capacity Building |
13. | MANAGEMENT ARRANGEMENTS Given that protected areas fall under the realm of several different national agencies, these institutions were also partners of the Project, not only at the decision – making process (such as in the Project Screening Committee and eventually in the ad hoc technical committees set up in the latter parts of the implementation process) but also in direct implementation of activities and products. The Project Steering Committee at start-up was quite large, which made the decision-making process slow and cumbersome. As part of its adaptive management strategies, a restructuring of the PSC took place at approximately mid-point of the implementation process where the number of members of the PSC was reduced. Tag: Project and Programme management |
14. | The design also provides guidelines for the functioning of the Project Management Unit (PMU) as well as staffing guidelines. The original management arrangements indicated that the PMU would be responsible for directing, supervising and coordinating implementation and that it be located within NEPA. The PMU was to be composed of a project manager, an administrative assistant and a part-time accountant. By all analysis these arrangements have been deemed as inadequate in order to manage such a complex and substantial project as NPAS. This, together with the percentage of funds allocated and budgeted for staffing, has been one of the causes for implementation issues which will be seen further along this report. Tag: Challenges Programme/Project Design Project and Programme management |
15. | ADAPTIVE MANAGEMENT (CHANGES TO THE PROJECT DESIGN AND PROJECT OUTPUTS DURING IMPLEMENTATION) Tag: Challenges Programme/Project Design |
16. | PARTNERSHIP ARRANGEMENTS (WITH RELEVANT STAKEHOLDERS INVOLVED IN THE COUNTRY/REGION) Tag: Challenges Impact Relevance Programme/Project Design Bilateral partners Country Government Coordination |
17. | Also, partnership arrangements were similarly maintained with other non – state actors involved in protected area management in the country. This was particularly the case with non – governmental organizations charged with the mandate of managing particular sites. Furthermore, the mentioned government agencies as well as other non – governmental organizations and other relevant government institutions partnered in the governance and guiding of the Project through the Project Steering Committee and the ad-hoc technical committees. PSC participation was positive, with some rotation in representation throughout the life span of the project. Attendance and involvement was proactive for the most part. At times, there was delegation and rotation without bringing new participants up to date. Tag: Natural Resouce management Bilateral partners Country Government Coordination |
18. | Partnership and linkages were also maintained at the regional level. A key collaboration took place with the Caribbean Biodiversity Fund (CBF) given that this organization will be the umbrella institution for the financing mechanism that would originate out of the NPAS Project. Several stakeholders have indicated that it would have been desirable to link bilaterally with other countries, particularly in the Caribbean, that have implemented or are implementing these sorts of project and also with those countries that are part of the CBF initiative. Although exchanges have taken place with these countries, it would have benefited the NPAS project to have had deeper linkages, in particular to learn how the issues of protected area management and financial sustainability was dealt with in each case. South-south cooperation would have benefitted the Project through sharing of knowledge, experiences and of best practices. Tag: Challenges Sustainability Bilateral partners International Financial Institutions |
19. | FEEDBACK FROM MONITORING AND EVALUATION ACTIVITIES USED FOR THE ADAPTIVE MANAGEMENT Tag: Challenges Change Management Project and Programme management |
20. | The earlier implementation stage was characterized by a stagnant delivery, and little or no adaptive management. Therefore, feedback from monitoring activities was not used for adjusting management issues. After the mid-point of the project a strong usage of adaptive management instruments took place (as indicated in other relevant section of this report). Much of this was based on monitoring and evaluation activities. The findings from monitoring and evaluation activities were used as a base to redirect whatever needed to be redirected in order to achieve results, albeit in the latter implementation period of NPAS. Other issues that hindered monitoring reside in the fact that UNDP did not have a specific Monitoring and Evaluation position at the Jamaican Country Office until 2014 and, therefore, did not have a specific area to aid the Project in monitoring. Tag: Challenges Programme/Project Design Project and Programme management |
21. | PROJECT FINANCE Tag: Global Environment Facility fund International Financial Institutions |
22. | Based on the ProDoc revision in 2015, the project’s budget was reduced to USD2,020,585 due to the reallocation of USD750,000 to the Caribbean Biodiversity Fund. At the end of 2016, the total expenditure from GEF funds was USD1,898,391. The balance at the end of 2016 was USD122,193. Notwithstanding, the approved budget and the approved spending limit for 2017 was USD247,700, of which USD228,377was expended. This reflects an over expenditure of $106,184. This was an oversight from three sides: the project’s implementing partner, NEPA, UNDP Country Office and the UNDP GEF Unit in Panama. The various shifts in budget, including direct payment to CBF, may have led the team to lose track of the actual amount available and to overstate the available budget, given the initial slow rate of delivery. While there were reductions in the co-financing amounts by specific donors (e.g., government, TNC), the overall amount was increased to USD6,178,554 due to an increase in KFW’s contribution to the CBF. It is important to note that co-financing went directly to the CBF from the donors. Tag: Challenges Global Environment Facility fund Multi Donor Trust Funds Donor |
23. | MONITORING AND EVALUATION: DESIGN AT ENTRY AND IMPLEMENTATION Tag: Impact Monitoring and Evaluation |
24. | The implementation of the Monitoring and Evaluation framework has followed, to a large degree, the M & E plan. The Inception Workshop and Report were set up early in the implementation process (two months after project signature). The mid-term review took place six months before it was assumed it would take place. Given the early signs of implementation and governance stand stills present at that time of the project mid-term review, this was positive since it gave the Project leeway in designing and implementing adaptive management. Other monitoring tools were also applied as planned, yet some had delays. Stakeholders indicate that application of the monitoring plan was difficult and intricate at times since, in addition to the reporting to be done according to the Project’s M & E plan, national agencies’ reporting was added. This generated extra effort for the PMU and duplication of efforts. Tag: Effectiveness Efficiency Monitoring and Evaluation Risk Management |
25. | IMPLEMENTING PARTNER AND UNDP IMPLEMENTATION / EXECUTION COORDINATION, AND OPERATIONAL ISSUES Tag: Challenges Impact Programme/Project Design Project and Programme management Country Government |
26. | The Project established a Project Management Unit hosted in the offices of NEPA while a representative from this institution acted as Project Director. The PMU had several personnel rotations throughout the implementation period, including three different project managers and a period without a project manager where NEPA staff assumed a proactive management role. These rotations hindered continuity of the implementation process. Therefore, this issue can, to some degree be associated with some aspects of the critical delays that the Project suffered throughout its operation. Staffing of the PMU was deficient in many respects. First, since this sort of GEF – funded UNDP-implemented projects management units are set up following guidelines which establish a rather small staffing arrangement, it was assessed that the arrangements were not effective. Considering the periods where this staffing arrangement was not in place (for instance due to leaves of absence), even the minimal arrangement established at design following UNDP/GEF guidelines was many times not met. Furthermore, the salary caps (i.e. percentage of a project’s budget that can be used for salaries) resulted in limitations to the salaries that could be paid. Considering Jamaican salary ranges it was deemed that the limited amount available for salaries did not attract personnel with the needed or desired technical capacities and management capabilities required for managing such a large and intricate project as NPAS. This coupled with limited national capacities and lack of responsiveness from government to meeting in due time with the required procurement and decision making for management greatly hindered implementation. Tag: Challenges Effectiveness Procurement Project and Programme management |
27. | Given that the Project was implemented in a NEX/NIM modality, the role of NEPA was key as it regards to management and operative matters, not only as they relate to the PMU but also in all terms of operational issues (including contracting, procurement, and approvals needed to execute work plans). NEPA has a complexed and long procedure for processing procurement matters which –in turn—caused significant delays as well as set- backs in project implementation. Up to five different administrative processes had to be implemented for decision making and procurement, just from NEPA. It was estimated that hiring of consultants took about eight months on average. Tag: Challenges Procurement Project and Programme management |
28. | Overall, the above-mentioned deficiencies and issues related to management were some of the causal effects of the very slow start-up, such as not meeting with timelines, substantial postponements in execution, very low delivery until the last stages of the project implementation process. These were also the reasons why two extensions had to be obtained to finalize the project, even while the project had a planned implementation span of six years, and this was also the rationale for the review to the results framework. The latter, although commendable in some respects since it overtook some design issues, the reforms to the log frame, also responded to the fact that –due to implementation issues—the target results were abridged to have a greater likelihood to meeting with them, at the product/output levels. Tag: Challenges Project and Programme management |
29. | In summary, the project had several issues regarding implementation processes and severe delays in delivery at its start up. However, adaptive management and an intensified and purposefully accelerated delivery in the last year of implementation resulted in achievements, albeit mostly at the product level. Therefore, as a composite, these aspects are deemed Moderately Satisfactory (MS) since there were moderate shortcomings in the achievement of Project results due to management issues. Tag: Challenges Effectiveness Impact Project and Programme management |
30. | Working in tandem with UNDP’s processes was also difficult since the two administrative procedures overlapped. Notwithstanding these issues, coordination between UNDP and the national implementing partner was positive throughout the operational process. UNDP’s role throughout implementation varied. There was a more proactive role after the mid-point. Especially when full realization was made by all parties that the project had a very low delivery rate and was in a serious risk of coming to a standstill, and that UNDP had to provide a more active role if the project would continue to be implemented. UNDP also had a strong role in procuring expertise when technical proficiency was not available in-country. Therefore, the overall quality of UNDP implementation was Satisfactory (S) given shortcomings identified in particular in the achievement of effectiveness and efficiency. The shortcomings identified, by many stakeholders, deal with the matter that UNDP could have played a more active role in implementation and execution (evidently without interfering with the country’s national implementation matters) early on the implementation process in order to avoid or moderate the serious delays that the Project had, particularly in its initial stages of implementation. Tag: Effectiveness Efficiency Project and Programme management |
31. | OVERALL RESULTS Tag: Impact |
32. | Although this output (1.1) is somewhat completed, the National Conservation Trust Fund of Jamaica (NCTFJ) is still awaiting final formalization of this Trust Fund with the Caribbean Biodiversity Fund. Formalization would be completed with the signing of the partnership agreement. It is expected that this would take place by December 2017. The establishment of the NCTFJ, if fully operational and capitalized, would be a major achievement of the NPAS Project leading to the consolidation of a financing mechanism for protected areas in Jamaica, a key expectation out of this project as well as key issue in protected area management within a developing country context. The establishment of the fund had a long series of debates within the Project and with key stakeholders at the national, regional, and international levels. After governance issues of such a Trust Fund were modified to fit national priorities, and when formal registration issues were at last carried out, the Trust Fund was formally recognized at the national level. The signing of the official partnership agreement with the Caribbean Biodiversity Fund would be the last formalization step that would need to be taken to formalize the Trust Fund. However, a considerable matter to be contemplated is the issue of co – funding. Although co – funding from the country’s Tourism Enhancement Fund was expected throughout the NCTF deliberations, the likelihood of drawing from those funds is deemed doubtful from several key stakeholders at the time of the evaluation. First due to budget limitations, second due to a lack of presentation of robust and solid proposals. Working on having the NCTFJ functioning could be a key aspect of immediate follow up for the NPAS Project. Tag: Natural Resouce management Protected Areas Multi Donor Trust Funds Project and Programme management |
33. | At the output level, within Output 1.2, the preparation of model site level business plans was achieved. However, several problems were identified with these business plans. First their phasing. The business plans were completed at the very end of the Project. Yet, interventions at field sites that specifically dealt with issues related to the business plans (such as upgrading of facilities or merchandising for the generation of income for the protected areas) were implemented before the plans were available. Therefore, there was no possibility of implementing activities with the guidance of the respective business plans. Other issues with business plans was their quality and their approach. Several business plans were not of the expected quality, and they had to be redrawn. Furthermore, several attempts at drafting business plans were abandoned given that the consultants engaged did not have any knowledge of business planning for protected areas, which is a rather specific area of expertise where not any sort of business plan is applicable. Lastly, all stakeholders consulted regarding the quality of these products agreed that those who drafted several of the plans had no knowledge of the national or even regional context or –if they did—this was not reflected fully in the products. That is, that they had no knowledge of “what it takes” to draw and establish a business plan in a developing country context such as in Jamaica (with its particularities regarding protected area policies and its socio-economic context) and that the proposals in the plans were highly inapplicable in this framework. At the time of the terminal evaluation one business plan was being used for seeking further financing for PA management. Tag: Challenges Business Model |
34. | Although the baseline for Output 1.3 is inaccurate given that there were documented revenue generation mechanisms in protected areas, the output was completed in several products given that there were activities implemented (such as upgrading of infrastructure) which could theoretically increase revenue generation. It is indicated that this is theoretical since at the time of the evaluation no increased revenue had been achieved yet. Furthermore, when local stakeholders who were not protected areas managers were engaged in order to generate mechanisms for income diversification, these have not been sustained and they have been abandoned. Again, many of these activities would have had a linkage with the business plans (see above), but as indicated above, the business plans were finalized after the revenue generation mechanisms were tested or piloted, precluding interchange between these highly linked outputs. Tag: Challenges Business Model Project and Programme management Infrastructure |
35. | Output 1.4 completed only at the product level, as the report from which the above chart is extracted indicates, no effect or outcome can be determined. There is no evidence of beginning implementation of these financial plans at the time of the evaluation. One of the main outcomes expected out of the NPAS Project was to streamline protected area management policy in the country, in particular through this specific output/outcome (2.1). A new overarching policy was drafted, also Drafting Instructions for an Overarching Protected Area Legislation (Protected Areas Act for Jamaica), yet they had not been fully adopted by the time of the evaluation. Some stakeholders indicate that this was one of the design failures given that the expected output/outcome was not relevant for Jamaica, it was not feasible nor applicable. Nevertheless, in addition to the already myriad of policies dealing with different protected areas recommendations for drafting instructions for a new protected area Act were drafted. Although it has been placed in the legislative agenda, stakeholders are uncertain as to whether it would be adopted. Tag: Protected Areas Challenges Programme/Project Design |
36. | The expansion of the PA network as such was not achieved within the Project (as expected for Output 2.2). The base documents for declaring the mentioned areas (Pedro Cays and Black River Protected Area Landscape) have been drafted however. Yet, the process of declaration itself is expected to commence after the Project ended. Although some stakeholders do indicate that this culmination of declaration of new PAs might take place in the near future, there is no indication of how this will be brought about as a result of the project. Tag: Challenges Project and Programme management |
37. | Management plans were completed for the targeted protected areas (Output 3.1). Some of these were innovative or pioneering since some areas had no management plans at all. Other areas updated already existing plans. Stakeholders from some protected areas indicate that they are using the new or updating plans for day to day management, which is indicative of effect and likelihood of sustainability of these products. Tag: Efficiency Sustainability Business Model Project and Programme management |
38. | This output is linked to Output 1.3. As above, the baseline is inaccurate given that there were documented conservation-based economic development activities in protected areas. The output was completed in several areas given that there were activities implemented (such as upgrading of infrastructure) which could theoretically increase conservation-based economic development activities. It is indicated that this is theoretical since at the time of the evaluation no conservation – based economic activities based on the project are reported. Again, as in Output 1.3, many of these activities would have had a linkage with the business plans (see above), but as indicated above, the business plans were finalized after the conservation – based economic development activities would have been piloted, precluding meaningful interchange between these highly linked outputs. Tag: Challenges Business Model Project and Programme management |
39. | Although increased stakeholder awareness regarding PAs in all protected areas amongst stakeholders and increased stakeholder feedback and involvement in PA activities is not measured as an outcome just as an output (as expected as of Output 3.4), there have been some activities carried out. However, the effectiveness of these activities is not being measured. The Project also had a series of issues with communications consultants, and partly based on those problems a full awareness campaign was never implemented as planned. Just some isolated posters and signage. Although the report states that “NPAS webpage has been linked to the NEPA website and now being maintained by NEPA” this is incorrect. The webpage presence is very weak, it only contains an outdated Project Description (Information Sheet) and the Project Document. The workshops and events with communities were very weak methodologically. For the most part, also, the Project and its products did not have thorough visibility. Tag: Challenges Communication Strategic Positioning |
40. | Furthermore, very key issues were not taken up by the Project in the way it was expected by an intervention that was to be development oriented (perhaps due to this international consultancy modality or because the NPAS Project decided not to recognize these issues). For instance, the matter indicated in the planning documents that opportunities for communities to realize the potential social and economic benefits accruing from biodiversity are lost or that there are no formal state policies to facilitate mutual benefit opportunities between conservation and tourism (which is one of the main drivers and employment generation activity in Jamaica and which – in the country – is tourism based on natural areas). The opportunities to link protected areas with the country‘s socio-economic development priorities (also indicated in the project planning documents) were basically lost within the implementation process and when attempts were made to include livelihood aspects for communities that use protected areas, these were rather weak and imprecise. Tag: Challenges |
41. | Furthermore, when results are analysed in comparison to targets or indicators that would supposedly be met at the end of the project, several issues arise. First of all, as indicated in the section on log frame and results indicators, many of these do not deal with outcomes or results but only with outputs. Therefore, whatever analysis that can be made of achievement or not of results, outputs, or outcomes struggles with this issue. At the output level, it can be said that NPAS has achieved several products. Studies, plans, improvement of infrastructure in protected areas, background information for declaring new areas and other such products have been drafted and overall –at the product level—the planned outputs have been achieved for the most part. Furthermore, the basis for what can be a sustained financing mechanism for protected areas have also been laid out. Unfortunately, due to excessive delays and management problems, most of these products were achieved at the very end of the project. Given this, and given the prevailing vision that the Project would generate those outputs with little or no foresight of achieving outcomes, effects and even impacts, the latter were not achieved. A few products are being used for management of protected areas in Jamaica (for instance, the management plans). That is, for the most part the products were achieved but their use, effect and potential impact has not been fully reached and there are some doubts whether these would after all be done in the near future if there is no sustained impulse to do so. This is the case with policy adoption and the trust fund, for example. Tag: Challenges Impact |
42. | COMMUNICATIONS AND VISIBILITY Tag: Challenges Communication |
43. | RELEVANCE Tag: Relevance Global Environment Facility fund Project and Programme management |
44. | EFFECTIVENESS AND EFFICIENCY Tag: Effectiveness Efficiency |
45. | Regarding effectiveness, the Project has been fairly effective in achieving outputs/products and less effective in achieving outcomes. As some stakeholders indicate, and a valorisation with which this evaluation agrees, the project had issues in harnessing effects of most of the products achieved due to a fragile implementation and due to weak adoption of these outputs, and in resolving issues with the implementation of the products. Although some products are producing outcomes (for example, it can be inferred that the management plans by being applied increasing the effectiveness of protected areas management towards improved operational mechanisms), as a whole not much effect is being determined. Although some key stakeholders indicate that this is the case because the project design lacked guidance on the implementation of products, this evaluation disagrees with that statement since the premise of the project has always been in obtaining outcomes and meeting with its general objective. This project, as all of these kind, was results - oriented and it is implicit and explicit that it should have sought effects. Tag: Effectiveness Efficiency Results-Based Management |
46. | The efficiency analysis of NPAS Project requires that it be divided into two periods of analysis, keeping in mind that all of the analysis contained in the terminal evaluation is for the whole implementation process. That is the scope of the evaluation is the Project as a whole, from beginning to its conclusion. First an analysis is made from the Project’s initial implementation stages to its relative midpoint and then a second phase from midpoint to its conclusion, including the periods in the extension periods. The first stage of implementation was moderately unsatisfactory given the very significant shortcomings experienced indicating slow delivery (basically deficits in producing outputs, products and outcomes but also deficits in having the NPAS Project coalesce as a project). Essentially, this first stage was characterized by a very low level of delivery and even for total lack of mechanisms to implement several of the Project components. The second tranche of the Project was relatively more efficient and therefore more satisfactory, yet many activities, processes and products had to implement in a short period most of what was not delivered in the first years of operation. This undoubtedly has impacted on the efficiency of outputs and on the attaining or not of outcomes. Therefore, a composite ranking of efficiency for the full scope of implementation is MS (Moderately Satisfactory). Tag: Challenges Effectiveness Efficiency Implementation Modality |
47. | COUNTRY OWNERSHIP Tag: Challenges Impact Country Government Capacity Building |
48. | MAINSTREAMING Tag: Challenges Impact Global Environment Facility fund UN Country Team Mainstreaming, Acceleration and Policy Support (MAPS) |
49. | SUSTAINABILITY Tag: Sustainability Global Fund Vertical Trust Funds |
50. | Socio-economic risks to sustainability: When analysing socio economic risks to sustainability, an examination is made of the potential social or political risks that may jeopardize sustainability of project outcomes. The level of stakeholder ownership, as seen in the narrative of this report, is somewhat strong in some areas and weak in others, and this poses some socio-economic risks to sustainability. Although government does indicate that it is in their interest that the project’s benefits continue to accrue, other risks are still identified. For instance, the lack of buy-in from several sectors of Jamaica regarding the true need to sustain management of protected areas vis-à-vis sectors of the economy which are deemed more important (such as tourism and extractive industries). Therefore, the ranking for socio – economic sustainability is Moderately Likely (ML), given that, although there are moderate risks, there are also expectations that at least some outcomes at different levels would be sustained. Tag: Challenges Sustainability Risk Management Sustainability |
51. | Institutional framework and governance risks to sustainability: At the time of the final evaluation there are no clear institutional and governance changes identified that would indicate the probability of governance sustainability. There are no clear-cut legal frameworks, policies, governance structures and processes in place attributable to the NPAS Project. Although some policies are expected to be in place (for instance, the declaration of new areas) in the medium term, the adoption of other policy which was impelled by the Project is still doubtful, there have been political changes in the latter part of the implementation process without a full progression of impelling and petitioning to the new political authorities regarding the reforms needed at the policy level to adopt the new acts, etc. Furthermore, the low general buy-in to these reforms, although understandable given Jamaica’s situation regarding protected area management processes, are indications that it is not perceived as if all of the generated and proposed policy would be adopted in the near future. Therefore, the ranking for this sort of sustainability is Moderately Likely (ML) given that there is substantial risk that outcomes will not materialize in a manner attributable to the Project or will not carry on after project closure, although some outputs and activities should carry on. Tag: Challenges Sustainability Rule of law Project and Programme management Sustainability |
52. | Environmental risks to sustainability: Environmental risks to sustainability conflicts are identified regarding natural resource management and regarding climate change. Regarding the former, conflicts arise out of productive activities that do threaten management of protected areas. Some of these were identified at design and remain relevant to date (encroachment of tourism endeavours, conflicts with extractive industries, etc.). Furthermore, the weather patterns of the Caribbean due to climate change continue to affect protected areas, as well as –of course—the rest of the country. Issues such as increased forest fires, increase in severity of weather events such as hurricanes, coral bleaching and others have been identified as threats and increased environmental risks that can and do increase environmental risks and issues associated to climate change vulnerabilities and, in turn, associated to the management of protected areas. Therefore, given the moderate risks faced, the ranking for environmental sustainability is Moderately Likely (ML). Tag: Climate change governance Natural Resouce management Sustainability Sustainability |
53. | Environmental risks to sustainability: Environmental risks to sustainability conflicts are identified regarding natural resource management and regarding climate change. Regarding the former, conflicts arise out of productive activities that do threaten management of protected areas. Some of these were identified at design and remain relevant to date (encroachment of tourism endeavours, conflicts with extractive industries, etc.). Furthermore, the weather patterns of the Caribbean due to climate change continue to affect protected areas, as well as –of course—the rest of the country. Issues such as increased forest fires, increase in severity of weather events such as hurricanes, coral bleaching and others have been identified as threats and increased environmental risks that can and do increase environmental risks and issues associated to climate change vulnerabilities and, in turn, associated to the management of protected areas. Therefore, given the moderate risks faced, the ranking for environmental sustainability is Moderately Likely (ML). Tag: Climate change governance Natural Resouce management Sustainability Sustainability |
Recommendations | |
---|---|
1 | RECOMMENDATIONS FOR FUTURE PROGRAMMING |
2 | 18. Beginning at the design stage, all national counterparts should thoroughly acknowledge what specific commitments they are making, as well as link any of these to a particular agency or institution’s work plans. There should be factual capacity assessment of national partners, not only their ability to manage financial aspects of a project but also assess their technical and managerial capacities. If weaknesses are identified, capacity should be built – in at institutional levels in order to generate enough competence to adequately implement complex projects. |
3 | 19. Given that the mere production of plans, studies, and processes as outputs does not automatically translate into results, these should be sought as part of a results-based project. Studies, reports, plans, documents and processes need to be accompanied by clear mechanisms that promote knowledge assimilation, knowledge sharing, and clear-cut mechanisms to inform and promote policy processes for adoption of outputs. Design and inception should state (and implementation should follow) a ‘road map’ where not only the achievement of outputs and products are indicated but the timing of such achievements needs to be specified in order to avoid generating most outputs at the end of a project, and consequently not impelling the achievement of outcomes and effects. Furthermore, timing of products indicating which should be produced before others should reflect how they interrelate. A project needs to establish clear links between studies, products or outputs and the expected outcomes (such as policy generation and adoption, policy commitments, public – private partnerships, investments, etc.). |
4 | 20. The budgeting sections of a project should be realistic at design and should budget sufficiently so that there is enough staff to manage a project and to draw the capacity(ies) needed for technical and administrative inputs. It should include a proper realistic financial plan with adequate costing of management (management personnel) and technical aspects (including needed technical staff as well as consultancies). Attention in budgeting should be paid to the true number of personnel needed to run such a project as well as the number and type of consultancies needed for developing products. National costs should be accurately acknowledged in the budget of a project in order to realistically budget all proposed personnel, products, processes, outputs and investments. |
5 | 21. The roles of different stakeholders within a project should be clearly defined and assimilated from the onset, generating buy – in, especially the roles of those stakeholders and institutions that should provide strategic direction (such as boards and committees with the appropriate representation and decision-making capabilities of members). If this does not occur, then there should be adequate flexibility to impel buy-in or change mechanisms, including other stakeholders that can assume a proactive role. |
6 | 22. Risks within a project should not be underestimated, and a risk management framework should be drawn at design and reviewed continuously. Once properly established, risks should be continuously monitored in order to promote whatever mitigation measures need to be implemented. |
7 | 23. Design as well as project inception and planning stages need to be precise and defined in order to guide the implementation process as well as obtaining achievements, outcomes and overall results. Design needs to be realistic, and log frame tools need to be credibly developed in order to guide implementation and tally achievements, not only to determine accomplishments but to correct the course of implementation when needed. Indicators as well as output to outcome processes need to be determined and robust measures for seeking results needs to be imbedded from the design and inception stages. |
8 | 25. In order for these sorts of projects to truly be developmentally – oriented they should be thorough in their work with communities, not only donating materials in the aim to improve their livelihoods but also to attend to these communities so that they can truly adopt whatever development aim the project is seeking. When working with local community groups and local communities a project needs to be aware of the skills, knowledge, and institutional capacity these organizations have. Interventions at the local level should also receive technical support in order for local communities to be able to implement demonstration piloting. |
9 | 26. UNDP needs to work with and assist the countries where interventions take place in order to aid them in applying processes that support projects’ technical and implementation capabilities (specially support project implementation and efficient decision – making capacities) and in applying procurement systems to increase capacity to efficiently implement projects aiding in the fulfilment of a project’s objective. UNDP should provide information on project management, financial reporting and other such project requisites in order to avoid misunderstandings as well as to generate capacity for implementation. |
10 | 28. When rotation as well as political and institutional turnovers take place, projects should have mechanisms in order to provide transfer of knowledge and information so that institutional knowledge and capacity transfer is assured. New political ministerial personnel need to be brought up to date regarding what a project has achieved and how to sustain results in a new political setting. |
11 | 29. A project’s communication strategy should be an ongoing process that generates buy – in, generates knowledge about the issues a project deals with as well as acknowledge its visibility. A communication strategy needs to be accompanied by clear inputs where the different partners are identified (funders, implementing agency(ies), UN agencies involved). A communication strategy should document and communicate issues, achievements, and challenges. |
12 | 30. When situations indicate that in – country knowledge base and expertise is not sufficient for generating outputs and there is a need for harnessing expertise from outside of the country, all efforts should be made to generate local capacity as well as introduce national issues in the resulting products. All key stakeholders should have clear inputs into the calls (terms of reference, etc.) for expertise in order to have products that, first of all, reflect national issues and, second, are useful for the country. International consultants should be partnered with national consultants in order for the outputs to reflect national issues and to transfer capacity to national experts. Also, mechanisms (training materials, workshops, etc.) should be promoted in order that effective generation of capacity takes place in relation to the production of studies or reports. |