Evaluation of the 2015-2017 Saemaul Initiative towards Inclusive and Sustainable New Communities

Report Cover Image

Share

Document Type Language Size Status Downloads
Download document SI Evaluation TOR Final.pdf tor English 718.52 KB Posted 523
Download document ISNC final report November 2018.docx report English 487.22 KB Posted 399
Title Evaluation of the 2015-2017 Saemaul Initiative towards Inclusive and Sustainable New Communities
Atlas Project Number: 00079025
Evaluation Plan: 2018-2021, Bureau for Policy and Programme Support
Evaluation Type: Final Project
Status: Completed
Completion Date: 11/2018
Planned End Date: 11/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.1.2 Marginalised groups, particularly the poor, women, people with disabilities and displaced are empowered to gain universal access to basic services and financial and non-financial assets to build productive capacities and benefit from sustainable livelihoods and jobs
Evaluation Budget(US $): 30,000
Source of Funding: Ministry of Foreign Affairs of the Republic of Korea
Evaluation Expenditure(US $): 25,800
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Birgitte Woel birgbirgitte.woel@gmail.com
GEF Evaluation: No
Key Stakeholders: • UNDP • Country Office • Local Authorities • Relevant ministries • Republic of Korea
Countries: GLOBAL
Lessons
Findings
1.

6.1.1 Type A countries

UNDP is the acknowledged multilateral system entry point for innovative territorial partnerships, including decentralized cooperation and local level South-South and triangular cooperation modalities. Therefore, the ISNC project utilized UNDP’s ‘Strategy to Support Localization of the Sustainable Development Goals (SDGs),’which aims to improve the quality of life for local residents and building resilient state-society relationships at the local level. An important element of the project methodology is to help all local actors discover the benefits and modalities of effective and continuing co-operation, linking communities with other public and private actors in the local and subnational arena[1].

In brief the key components were as follows:

  • Community participation and ownership with emphasis on social inclusion of women, youth and disadvantaged groups (such as persons with disabilities;
  • Co-financing principle for community projects: 20 percent of ISNC contribution would be matched with 30 percent government cost-sharing and 50 percent in-kind contributions of labour and services mobilized by the communities;
  • Performance-based incentives may be introduced to stimulate the drive and agency of the pilot communities for demonstration effect.

 


[1] Summary of chapter 3.1, Implementation Guidance


Tag: Relevance Gender Mainstreaming Partnership Strategic Positioning Resilience South-South Cooperation Agenda 2030

2.

6.1.2 Type B countries

The implementation focused on policy advices and contributed knowledge to the centres of excellence. UNDP tapped into national existing funds allocated for projects to create and redistribute productive resources to benefit micro and small enterprises to ensure the sustainability of the ISNC implementation along with existing enabling national polices. Additionally, UNDP utilized the ART Initiative (Articulation of Territorial Networks for Sustainable Human Development), stimulating dialogue between territories to face the global development challenges[1].

It was acknowledged that the small-scale investment from the ISNC programme was not likely to add noticeably to the country. Hence it was viewed as a comparative advantage to build on existing programmes to develop policy advices, contribute knowledge to the model through centres of excellence, and promote South-South cooperation (SSC).

It was important to find policy entry points to inform change at the national level. This involved defining national policy priorities which were included in the agreements between UN and the given country.  Tentatively, finding converging entry points would not only help in selecting critical sites,  but could also create policy impact and ownership at the national level increasing the likelihood of sustainability in the long run.

 


[1] Implementation Guidance, p.39


Tag: Sustainability Knowledge management South-South Cooperation National Institutions

3.

6.1.3 Characteristics of the process scorings of the two types of countries

The figure below illustrates the scorings of respectively type A and B country experiences with the ISNC planning and implementation process[1]

The blue colour shows the number of positive experiences with a given project implementation process, the orange shows the number of negative experiences while the grey indicates the number of proposed changes to the same process area. The latter indicates the satisfaction with applied processes.


Tag: Effectiveness Impact Sustainability

4.

6.1.4 National level framework

In countries with decentralized systems of governance and advanced local development policies and institutions (e.g. Uganda and Rwanda), ISNC supported strengthening of existing institutions, mechanisms and practices so as to reinforce their linkage with the local level and enable them to provide better services to communities and respond to their development needs.

Local governments at lower levels are often weak at functional level and little engaged in local development process. Hence the bottom-up planning process aimed at reinforcing the district government commitment by building governmental capacity to mainstream local plans and connect them to resources. Additionally, capacities of local governments below district level were enforced in planning, financing, implementation and monitoring of inclusive and sustainable local development. Hence, in East Africa, this approach aimed at responding to the implementation of African Charter on the Values and Principles of Decentralisation, Local Governance and Local Development.

Additionally, finding a high-level political champion for the ISNC model can be critical for changing attitude of government agencies and communities and making development projects sustainable. Fostering an effective and committed high-level political champion may determine the long-term sustainability and national government ownership[1].

 


[1] As above


Tag: National Local Governance Institutional Strengthening

5.

6.2.1 How relevant are the Saemaul Undong articulated theories of change to the expected outcomes and mandate of UNDP within the context of the ISNC?

There was no declared Theory of Change for the ISNC project since the project aimed at testing the actual applicability of SMU principles rather than aiming at pre-defined changes.  This chapter will therefore only asses the relevance of the expected outcomes and mandate of UNDP within the context of ISNC.The objective was to:

Update, integrate and scale up elements of the Saemaul Undong (SMU) and its application into an exemplary systematic approach and effective platform for development cooperation and to create a critical mass of support to localizing the Sustainable Development Goals (SDGs).


Tag: Effectiveness Relevance Project and Programme management Theory of Change South-South Cooperation Agenda 2030

6.

6.2.2   How relevant were the implementation mechanisms and processes for achieving   the ISNC outcomes and institutional effectiveness results in each of the six countries?

How effective have the implementation mechanisms and corporate strategies been in supporting achievement of the ISNC outcomes and results?

Since the two questions overlap, the analysis of both is merged in the following.

The implementation and mechanisms and processes can be divided into:

  • Planning and design approach
  • Implementation approach and processes
  • Monitoring and reporting design and processes

This section provides country examples of experiences with the project planning and implementation and no full accounting of the experiences of each country, since experiences are multiple and often diverse. Hence the full account is found in templates in the annexes. The report writing was based on prior data analysis, which determined the focus and structure of the report enabling use of highlights of the relevance and effectiveness of project implementation processes.

 


Tag: Effectiveness Relevance Monitoring and Evaluation Programme/Project Design Project and Programme management Strategic Positioning

7.

6.2.2.1 Planning and design[1]

Before the project period (2015-2017) UNDP Headquarter (HQ) had a one-year inception phase to have a robust background for selecting applicable SMU principles and to define criteria for selection of countries. Thus, the design was well considered.

Being a pilot project rather focusing of the relevance of principles than focusing on a specific end-result, the design and plans varied from those of development projects. However, it is only possible to test whether certain principles work, if also measuring results/desired changes for each of the principles. With a timeframe of two years the project could not be expected to have significant and widespread achievements at outcome and impact level. Without a clear line from principle, to approaches applied for each principle and clear definition of level and type of achievements, it is not possible to conclude, which of the principles would be replicable in given contexts.


Tag: Effectiveness Relevance Programme/Project Design Project and Programme management

8.

6.2.2.2 Implementation approach and processes[1]

With regard to administrative processes the scoring of Viet Nam covers the views of all countries by stating that: “Timelines, procedures and funds/budget versus project activities strictly followed the Harmonized Program and Project Management Guidelines (HPPMG), which was approved by the UN Resident Coordinator and the Government of Viet Nam and applied for all UNDP/UNFPA/UNICEF projects in Viet Nam. UNDP HQ support and coordination of information exchange and learning among 6 countries play an important role in defining project interventions/ activities”. All countries had positive scores on administrative procedures.


Tag: Effectiveness Implementation Modality Project and Programme management Capacity Building South-South Cooperation Women and gilrs

9.

To what extent does the ISNC position UNDP respond to the 2030 Agenda and the new development landscape – especially within the context of localizing the Sustainable Development Goals?

Apart from feeding into national policies, strategies and plans, the project was to operate the frame of the 2030 agenda and thus deliver to specific Sustainable Development Goals.

Only Viet Nam (CO) and Uganda (Local Government) account for which SDGs the project delivered to, while Lao accounts for the national policies to which the project delivered. Since all projects were designed to deliver to national policies, strategies and programmes they will implicitly have delivered to the SDGs as policies and intervention in all donor supported countries are to some extent aligned to the 2030 agenda.

All countries refer to which areas they wanted to focus on, e.g. hunger, but without mentioning which SDGs such focus will deliver to.

Conclusions:

  • Narratives in global and national progress reports show that all countries deliver to one or more of the SDGs without accounting for which and how.
  • Hence there were no recorded efforts of how each country localised the principles for SDG related achievements.

Tag: Relevance Strategic Positioning Agenda 2030

10.

6.2.4 To what extent and how were stakeholders involved at design level?

A range of stakeholders from central to local government, trade associations, CSOs, community level leadership arrangements, research institutions, KOICA and others were involved from planning to implementation and monitoring. The respective partners participated in the levels of planning and implementation that were relevant for the role they played. This saw central government, CSO head offices, KOICA and research institutions participate at the more overall level of planning involving alignment with national policies and strategies and establishment of criteria for the implementation, while local governments, partner CSOs and communities participated in the detailed design of the implementation of the community prioritised activities.


Tag: Relevance Ownership Programme/Project Design Bilateral partners Country Government

11.
  1. Effectiveness

Establishment of effectiveness presupposes regular, results-based monitoring with recording of results at output, outcome and impact level. Effectiveness can only be assessed if having full record of global achievements in this case and also knowing the level of achievement.

Moreover, being a testing of the applicability of SMUs principles with regard to SDG relevant achievements, such reporting should have been an integral part of all global reporting. Since these reports were more narrative than results-based and had no account of the experienced applicability of the applied principles, the analysis of the effectiveness will rather assess the reported development achievements at national level and less the effectiveness of the learning.

Since the monitoring plays a role for the full evaluation of the ISNC project it was viewed as relevant to have a small assessment of the limitation of the monitoring, maybe mainly the reporting, which was used. It should be noted that there has been no opportunity to assess the monitoring tools and practices at national level for which reason the analysis mainly concerns design and reporting.


Tag: Effectiveness Impact Country Government Agenda 2030

12.

Monitoring and reporting

6.3.1.1 Monitoring design

Most monitoring designing begins with development of results or logical framework organising the logic of all monitoring including formulation of indicators, source of data and in which period given data can/shall be collected. Not all activities may run throughout a project period.

The logical framework shall as a minimum have a logical arrangement of activities, SMART[1] indicators, suggestion of how/where to get the required data, information etc.

The ISNC global framework did not have such details. It is acknowledged that such detailed framework could not be developed until the countries had decided which SMU principles they wanted to apply for which SDG activities in conjunction with decisions on which approaches and activities that would be used for each of the chosen principles. There is no structured framework for this which may also be the reason why there is no uniform, national reporting feeding into an analytical global report focusing on the learning aspects. The project objective intentions were to:  

…… update, integrate and scale up elements of the Saemaul Undong (SMU) and its application into an exemplary systematic approach…..

To be exemplary there need to be conclusions drawn on the contextual applicability. The global reports were the media for this.


Tag: Relevance Monitoring and Evaluation

13.

6.3.2 How effective[1] has UNDP been in achieving the expected results of the ISNC?

As indicated above most achievements were at output level while the project intended to achieve results at impact level despite having only two years of implementation as highlighted in the following:

  1. Identify proven approaches and policy options for inclusive and sustainable local development, drawing on the expertise of SMU and other relevant solutions from development partners, including those from the South
  2. Achievement of policy impact at both local and national levels through applying the systematic approach of ISNCs to three countries where earlier application of SU experiences has achieved initial results;
  3. South-South and Triangular knowledge exchange and cooperation as well as global advocacy through support to already existing centres of excellence that share evidence-based results and experiences from the initial applications, with the aim of achieving impact at the regional and global levels.

 


Tag: Effectiveness Impact Knowledge management Policies & Procedures Education Women and gilrs Youth

14.

To what extent can UNDP be expected to contribute to development impacts at   the country level through the achievement of the ISNC outcomes?

Chapter 6.3.2 provides good evidence that UNDP stands a very good chance of moving the level of achievements from output to outcome and impact level in all of the six involved countries should the project have a phase II.

The involved governments have all engaged in different ways both at central and local level and have accepted changes in hitherto procedures and practices. Hence there is a foundation for continued introduction of SMU principles at national level. It should be noted that the life quality aspect, which is the aim of the SMU principles, is not achieved, which could make devolved level participants be hesitant and activities at this level fall apart. There is urgent need to demonstrate the value of the principles. The involved governments may need technical and financial support for some years to generate stable systems that can ensure the desired sustainability.


Tag: Challenges Effectiveness Impact Knowledge management Programme/Project Design

15.

6.4 EfficiencyEfficiency in the DAC development context should be understood as “Value for money” – or did the project achieve the planned results within the provided resources.  In this SMU piloting and testing context the principles were tested, but not concluded on and made exemplary. As concluded earlier the project delivered impressively to result 1 and 2, while achievements for result 3 were predominantly at activity level. Since result 3 concerns sharing of knowledge at national, regional and global level, there need to be something tangible to share. Since most achievements were at activity and output level there was little tangible to share in terms of proven effective approaches generating sustainably improved life quality. Hence the project did not deliver the expected value.


Tag: Efficiency Human and Financial resources Operational Efficiency Ownership

16.

6.4.2 To what extent have UNDP and KOICA resources been used efficiently in contributing to the outcomes and results outlined in the ISNC project document?

At the time of planning the direct funding in US Dollars was as indicated below. The two major donors were:

The Republic of Korea with                       USD $5,066,361.00

UNDP with                                     USD $63,577.00

All countries spent all of their resources.  Uganda and Rwanda requested non-cost extensions in 2018 in order to deliver on several activities. This was communicated to the donor and approved, and funds were spent during the first quarter in 2018.  Additionally, some resources were put aside for 2018 at HQ to conduct the evaluation and the hiring of consultant to ensure the final execution of some activities of the project as both staff members managing the project at HQ have left to other jobs.


Tag: Efficiency Human and Financial resources Operational Efficiency Project and Programme management

17.

6.4.3 How efficient was the coordination and collaboration, specifically management arrangements at the global, regional and country levels, in supporting the implementation and results achievements of the ISNC?

The coordination was three-pronged including UNDP HQ to CO coordination, regional/SSC coordination and national coordination linking producers with the policy level. Concerning coordination of project implementation from UNDP HQ to COs there is little reporting. A few examples of CO experiences with HQ coordination is given in chapter 6.2.2. All countries had positive scores on the management procedures between national and UNDP HQ level finding the systems and structures straight forward and easy to use.

The project was managed by the Development Impact Group in the Bureau for Policy and Programme Support (BPPS) under the SSC team.


Tag: Efficiency Implementation Modality Bilateral partners Country Government Coordination South-South Cooperation

18.

6.5.1 Gender Mainstreaming

On gender mainstreaming UNDP’s Implementation Guidance explains the necessity of gender mainstreaming interventions as follows:

No new global agenda can be achieved if it is not adopted at all levels of society. One of the key lessons we learned from the Millennium Development Goals (MDGs) is the importance of national and local actions for high-level impacts. By localizing the SDGs, we aim to reach women and youth as well as the poorest and most marginalized people. These groups often face additional burdens of discrimination – based on age, gender, ethnicity, indigenous status, disability, place of residence, HIV status or other factors. They typically have the least resources and remain the farthest behind. If development is not carried out locally, it will never benefit everyone.

Therefore, a local approach in pursuit of the proposed SDGs, involving communities in the transformation they want while empowering them to become transformative will universalize the development agenda as it localizes its implementation[1].


Tag: Sustainability Gender Mainstreaming Agenda 2030 Women and gilrs Youth

19.

6.5.2 Environmental measures

The environmental sustainability had two dimensions addressing respectively productive and residential areas. UNDP’s intentions were:

…to also reflect the integration of the three pillars of sustainable development – economic growth, social development and environmental sustainability. The ISNC model aims to promote sustainable productions that involve the community in economic activities that are environmental friendly and deliver social benefits[1].


Tag: Effectiveness Sustainability Risk Management Inclusive economic growth

20.

6.6.1 Are the results of the ISNC programme in the individual countries likely to be sustained?With the earlier described range of achievements, national anchoring and stakeholder commitments it would be easy to conclude that the project would be nationally sustainable. It should be noted, though, that the achievements are hardly internalised due to the often-late roll-out and short time of implementation. The stakeholders in all countries gave their judgement on the likely sustainability. It shows below[1]:


Tag: Sustainability Human and Financial resources Monitoring and Evaluation Project and Programme management Strategic Positioning Bilateral partners Civil Societies and NGOs National Institutions

21.

As villagers gained more confidence in their ability and the basic infrastructure necessary to improve agricultural productivity, Saemaul Undong shifted its focus, and income-generation projects were gradually initiated while the scope and size of each living condition improvement project increased. In the last phase, the focus was shifted towards capacity-building and attitudinal changes, while the scope of the projects became broader. Activities in urban areas, factories and corporations became more common, which changed Saemaul Undong into a national campaign[2].


Tag: Relevance Sustainability Resource mobilization Theory of Change Capacity Building Agenda 2030

22.

The inception period provided details suggesting a linking of the project to national policies and strategies and to other relevant project/programme activities in the respective countries. This resulted in involvement of all or some key stakeholders in all the six countries e.g. relevant central and local government ministries and institutions, CSOs, other types of organisations and KOICA. The three type A countries mainly involved key stakeholders at devolved level, while the three type B projects mainly involved stakeholders at central level. In the five countries the selection of communities was guided by key stakeholders, while Uganda CO seems to have played a central role with regard to community selection.


Tag: Relevance Health Sector Programme Synergy Project and Programme management Country Government Jobs and Livelihoods National Institutions

23.

Type B countries:

COUNTRY

EXAMPLES OF CAPACITY BUILDING INITIATIVES

Myanmar

  • Launch of report of Myanmar’s good practice and learning from mainstreaming conflict sensitivity into local and community development;
  • Establishment of inter-ministerial study group on conflict sensitivity and the first indicator framework on conflict sensitivity in Myanmar. It served as a mechanism for information sharing and dialogue

Rwanda

  • Comparative analysis report on Saemaul projects vis-a-vis Rwanda home-grown solutions highlighting the nine best practices.
  • Online portal was developed, and stakeholders were trained on the use of online centre of excellence for knowledge sharing

Viet Nam

  • Development of handbook on community leadership and facilitation skills got community leaders, village heads, farmer group leaders, civil social organizations/associations, etc.
  • ISNC Festival piloted the first time at national level;
  • Passing on one resolution and four Decisions in the Assembly;
  • For the first-time gender mainstreaming became a cross-cutting issue in all national poverty reduction interventions

Table 4 - Examples of capacity building, type B countries


Tag: Effectiveness Knowledge management Project and Programme management Education Capacity Building

24.

6.3.1.2 Organisation of the monitoring

The monitoring was well organised and appreciated by all countries.  There were clear lines for data handling, compilation and analysis – the latter at country board level. A couple of countries notice some challenges at field level where partners may have insufficient experience with handling, analysing and using the data.At the HQ level, the SSC team in BPPS undertook all upstream monitoring and evaluation measures, which were in accordance with UNDP policies and procedures, to ensure the proper implementation of the ISNC global project.


Tag: Effectiveness Monitoring and Evaluation Project and Programme management Agenda 2030

25.

Table 5 – Examples of achievements, Result 1

Examples of achievement under Result 2:

COUNTRY

EXAMPLES OF ACHIEVEMENTS

Rwanda

  • A national forum on ISNC was established for policy dialogue

Viet Nam

  • ISNC spirits and approach have been institutionalized into official program document of NTP-SPR (2016-2020) and poverty reduction related policies
  • Passing of 1 Resolution and 4 Decisions
  • For the first time 3-5-year budget for poverty reduction
  • For the first-time gender became cross-cutting issue in all poverty reduction
  • ISNC institutionalised in the National Targeted Programs resulting in budget and a range of changes in government approach

 


Tag: Effectiveness Policies & Procedures Poverty Reduction

26.

Table 6 - Examples of achievements, Result 2

Inputs to this result should mainly be expected from type B countries. Myanmar did not report on any achievements under this result.

Examples of achievement under Result 3:

COUNTRY

EXAMPLES OF ACHIEVEMENTS

Bolivia

  • Produced video showcasing SU/ISNC and SSC. The Government of Ecuador shows interest in rolling out the SU/ISNC model

Uganda

  • Partnership with Uganda National SMU Centre and National Farmers Leadership Center (NFLC).

Because of limited planning for dissemination of knowledge, best practices and experiences of the ISNC and because some of the key project interventions were commenced in the final quartile of the project lifetime the knowledge sharing was incomplete. Further, not all the stakeholders had access to the Best Practices and Experiences of the ISNC

Myanmar

  • Because of the research and training, DRD proposed the World Bank to provide conflict sensitivity training to their staff of National Community Driven Development project (NCDD). International Alert is now providing conflict sensitive facilitation training to CDD staff of DRD with the financial support of the World Bank since May 2018 until now.
  • The Netherland embassy and many INGOs invited Alert to share our research findings especially at their organization strategic planning workshop and partners meetings.
  • Alert also conduct conflict sensitivity workshop with local partners of the Netherland embassy and provide technical support to integrate conflict sensitivity in the strategy of Netherland embassy

Rwanda

  • The communication about the Centre of Excellence did not have the biggest outreach during the project.
  • Organization of regional conference on Lessons Learned sharing best practices on rural and community development in Africa

Viet Nam

  • ISNC has been institutionalized in the two National Targeted Programmes on Sustainable Poverty Reduction (NTP-SPR) and New Rural Development (NTP-NRD) for the period 2016-2020 (particularly on production support and capacity building), circulars, the M&E framework and handbooks for NRP-SPR implementation

Table 7 – Examples of achievements, Result 3

 


Tag: Effectiveness Knowledge management Capacity Building South-South Cooperation

27.

Nationally the coordination was three-pronged consisting of (i) coordination of project functions, (ii) linking national-local efforts and (iii) coordination of the sharing of knowledge sharing.

The project coordination was spearheaded by ISNC project boards headed by UNDP CO. The board comprised key partners inclusive of KOICA.

Coordination at national level scores thinly[2]. The efficient coordination is mainly seen in the scale of achievements which, in all countries, required substantial coordination. Examples of direct coordination are few, while the establishment of systems and structures linked to the intended national-local linkage for type A project (understood as utilising existing policies and strategies to have better livelihoods at community level), and the local-national linkage for type B projects (understood as learning from best practices to enhance national policies and strategies) are plenty.


Tag: Efficiency Coordination South-South Cooperation

28.

Bolivia and Uganda note untimely roll-out of the project as neither national budgeting period, nor agricultural production seasons were adhered to. In Bolivia seeds were therefore delivered out of season. In Uganda the roll-out came during an election campaign. Obviously, such incongruity in timing of activities could have had cost efficiency implications the size of which cannot be established. In the evaluation scorings Bolivia and Rwanda notes that the funds were insufficient to support the activities proposed by communities and associations[1].With regard to achievements among type A countries[2], the three across the entire spectre of activities from planning, to introduction, implementation and establishment if systems and structures supporting the SMU principles and further to knowledge products in terms of e.g. published story book about best ISNC practices, video on the same case studies on gender mainstreaming. Systems and structures supporting sustainability include e.g. saving and credit associations (Uganda), establishment of milling facilities and similar (Uganda), Livelihood support centres and Gender Information Rooms (Lao) and Leadership School for women and Youth in Bolivia.


Tag: Efficiency Human and Financial resources Operational Efficiency Project and Programme management

29.

A review of country documents and evaluation scorings show that there was no systematic gender mainstreaming as defined by UNDP, but rather measures to have equality. Here, as in many other projects/programmes, gender seems to be understood as the equal right of both male and females to participate in same initiatives. However, gender mainstreaming is wider and is defined as follows:


Tag: Effectiveness Gender Mainstreaming Capacity Building Women and gilrs Youth

30.

The trilateral and early involvement approach had the potential to generate a high level of sustainability, if the new capacity and subsequent attitude change had had the necessary time to take full shape and root.

The envisaged changes at national level were immense at output but were to a large extent not achieved at outcome and impact level. As mentioned by Rwanda most time was spent on implementation (input/output) and less on establishing systems and structures supporting a continuation. Left as they are most activities seem not to be sustainable with only Viet Nam having institutionalised the approach into various policies and programmes. The likelihood of sustainability in Viet Nam is significantly higher than in the other countries.

The environmental implications of the new productions are not known. Neither is the potential benefits from environmentally sound practices and localisation of the same.

Conclusions:

  • When analysing the scale of sustainability of achievements, the project efficiency is substantially less than when measuring efficiency as (output level) achievements against budget. This tallies well with stakeholder observations that time was too short and that the full focus on implementation/achievements may not be ideal.
  • Analysis of the sustainability of the use of SMU principles was not monitored and reported on. The many achievements indicate that, if given the required time, the use of localised SMU principles would facilitate a sustainable move towards the SDGs.
  • The gender and environmental sustainability was not recorded wherefore valuable learning about and sharing of best practices may have been lost.

Tag: Effectiveness Sustainability Gender Mainstreaming

31.

With regard to logistics the country experiences were many and diverse as the examples below show[2]:

On a positive note the scorings were:

  • Bolivia noted that delegation of execution of funds, project design and procurement authority from UNDP HQ to the CO helped in moving the work and resources faster. Other countries noted positive experiences fully in line with this.
  • Across the countries the strong involvement and use of local/national systems and structures (policies, programmes, strategies, and capacities) was highly appreciated and found effective.

On a negative note the experiences were the following:

  • Project duration was noted as being (very) short in all countries either by COs or by partners. For some the haste with which the project was implemented was also caused by late start of the implementation due to late funding. Only one CO notes that time was adequate, but that the period was very busy. Despite this, all find that achievements were good not least taking conditions into account.
  • UN procedures perceived as lengthy by all countries whether COs, partners or KOICA. Specifically, the procurement procedures were mentioned as they delayed the progress. It was suggested that UNDP HQ transfer a lump sum to the COs, e.g. quarterly, to have match project pace and mode of operations.  In some recipient countries government procedures at all levels were lengthy, too, which added to the challenge faced by the short project duration.
  • Frequent change of personnel across all stakeholders (COs and partners) necessitated repetition of i.e. training and information, which disrupted and implicitly delayed the work

 


[1] The details describing applied mechanisms and processes at country level are found in annex 6

[2] Full details in process scorings, annex 6


Tag: Effectiveness Relevance Policies & Procedures Procurement Programme/Project Design Project and Programme management

32.

UNDP’s core mandate is:

To support developing countries in designing and implementing national policies for sustainable human development with a focus on poverty reduction and the Sustainable Development Goals (SDGs).

Since the ISNC intention was to test SMU principles with the view to assess their relevance for localising the implementation of SDGs, the mandate of UNDP is thus very relevant within the context of the ISNC project.


Tag: Relevance Sustainability Project and Programme management Agenda 2030

Recommendations
1

It is overall strongly recommended that the project should be immediately extended incorporating observations and lessons learned from the evaluation.

It is recommended that the HQ/RO/CO roles are clear with no gaps or overlaps:

  • HQ should be the office overseeing and coordinating at global level without direct implementation engagement and little participation in project activities, apart from global activities;
  • HQ should be the entity conducting the research part of such test/pilot project documenting and sharing the learning of the research/test/piloting part of the project;
  • RO should be the office coordinating regional activities and supervising on day-today issues being familiar with the cultural and political context;
  • RO should be the convener of/or support regional meetings and participate in these;
  • CO should be the overseeing and coordinating office at national level supervising partners with regard to collaboration requirements and procedures and supporting the process as necessary.
  • Sufficient funds and time be allocated for on-site review and evaluation of a possibly second phase.
2

For planning and design, it is recommended that:

  • Projects aiming at changing government and local systems and structures and in addition changing the attitude of a population should plan for a 10 (-13) year project. The full period could be broken down into 3-year phases with clear defined types and level of achievements for each phase as they are designed.
  • Timing of the launch of activities should reflect seasonal activities in government and among stakeholders.
  • Stakeholders suggest:
  • Government institution to lead the implementation of all activities and disburse the project cost to Government or partner organization
  • Reinforced use of knowledge into the planning
  • Capacity building is a key activity when introducing SMU principles.

The DAC definition of capacity building is far wider than just knowledge.There is need for application of a holistic approach encompassing development/adjustment of:

  • Systems
  • Structures
  • Knowledge
  • Skills
  • Adequate number and type of staff
  • Equipment/facilities
  • Psychologically conducive work environment.

A range of methods support such changes:

  • Training/knowledge
  • Hands-on experience/skills
  • Peer Learning
  • Mentoring
  • Twinning
  • Provision of technical staff and secondments – also at devolved and community level.
  • There is strong need for ONE overall log frame when involving several countries, not least when testing principles, as there is need for documenting applicability of each principle and the context in which they are applicable. Systematic reporting on achievements for outputs, outcome and impact for each principle would generate learning making replication be easy.
  • Equipment/facilities
  • Psychologically conducive work environment.

A range of methods support such changes:

  • Training/knowledge
  • Hands-on experience/skills
  • Peer Learning
  • Mentoring
  • Twinning
  • Provision of technical staff and secondments – also at devolved and community level.
  • There is strong need for ONE overall log frame when involving several countries, not least when testing principles, as there is need for documenting applicability of each principle and the context in which they are applicable. Systematic reporting on achievements for outputs, outcome and impact for each principle would generate learning making replication be easy.
3

For implementation the following is recommended:

  • The evaluation scorings showed a significant need for detailed communication strategy

The strategy should define:

  • Who should know what – and the purpose
  • Media for sharing (TV, radio, conferences, reports, well-designed printed briefs, short breakfast meetings etc.)
  • Frequency of sharing with each type of stakeholders
4

Recommendations for project exit include:

  • The negative effect of lack of a project exit strategy was the concern of several stakeholders.  Irrespective of the timeframe there is need for an exit strategy preparing the mind-set of all stakeholders that this is the time for a take-over.

Documents, whether policies, instructions or others cannot replace a planned exit, which is more practical than intellectual.

1. Recommendation:

It is overall strongly recommended that the project should be immediately extended incorporating observations and lessons learned from the evaluation.

It is recommended that the HQ/RO/CO roles are clear with no gaps or overlaps:

  • HQ should be the office overseeing and coordinating at global level without direct implementation engagement and little participation in project activities, apart from global activities;
  • HQ should be the entity conducting the research part of such test/pilot project documenting and sharing the learning of the research/test/piloting part of the project;
  • RO should be the office coordinating regional activities and supervising on day-today issues being familiar with the cultural and political context;
  • RO should be the convener of/or support regional meetings and participate in these;
  • CO should be the overseeing and coordinating office at national level supervising partners with regard to collaboration requirements and procedures and supporting the process as necessary.
  • Sufficient funds and time be allocated for on-site review and evaluation of a possibly second phase.
Management Response: [Added: 2019/01/08] [Last Updated: 2021/01/05]

Management appreciates the recommendation highlighting the need for clear management and coordination structures and a well thought through project design. The evaluation report findings and this recommendation will be taken into consideration in the development of a second phase of the project, should UNDP pursue this, subject to internal capacity and continued availability of donor funding.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 UNDP to consider pursuing a second phase of the project, subject to continued availability of donor funding, and to organizational capacity to resource a project team.
[Added: 2019/01/08] [Last Updated: 2020/01/31]
TBD – subject to situation of SSC portfolio in the new organizational structure (currently being redesigned) 2019/12 No Longer Applicable [Justification: No second phase of the project will be persuaded, the evaluation lessons are noted for any future South-South S Cooperation programmes. ]
History
2. Recommendation:

For planning and design, it is recommended that:

  • Projects aiming at changing government and local systems and structures and in addition changing the attitude of a population should plan for a 10 (-13) year project. The full period could be broken down into 3-year phases with clear defined types and level of achievements for each phase as they are designed.
  • Timing of the launch of activities should reflect seasonal activities in government and among stakeholders.
  • Stakeholders suggest:
  • Government institution to lead the implementation of all activities and disburse the project cost to Government or partner organization
  • Reinforced use of knowledge into the planning
  • Capacity building is a key activity when introducing SMU principles.

The DAC definition of capacity building is far wider than just knowledge.There is need for application of a holistic approach encompassing development/adjustment of:

  • Systems
  • Structures
  • Knowledge
  • Skills
  • Adequate number and type of staff
  • Equipment/facilities
  • Psychologically conducive work environment.

A range of methods support such changes:

  • Training/knowledge
  • Hands-on experience/skills
  • Peer Learning
  • Mentoring
  • Twinning
  • Provision of technical staff and secondments – also at devolved and community level.
  • There is strong need for ONE overall log frame when involving several countries, not least when testing principles, as there is need for documenting applicability of each principle and the context in which they are applicable. Systematic reporting on achievements for outputs, outcome and impact for each principle would generate learning making replication be easy.
  • Equipment/facilities
  • Psychologically conducive work environment.

A range of methods support such changes:

  • Training/knowledge
  • Hands-on experience/skills
  • Peer Learning
  • Mentoring
  • Twinning
  • Provision of technical staff and secondments – also at devolved and community level.
  • There is strong need for ONE overall log frame when involving several countries, not least when testing principles, as there is need for documenting applicability of each principle and the context in which they are applicable. Systematic reporting on achievements for outputs, outcome and impact for each principle would generate learning making replication be easy.
Management Response: [Added: 2019/01/08] [Last Updated: 2021/01/05]

UNDP overall concurs with the need for a long-term perspective – including continued funding over extended periods of time – in order to help ensure sustainable outcome level results and to achieve change in terms of perception, culture and systems. To create high quality projects - based on analysis and evidence - UNDP has made guidance available to all programming units on how to develop rigorous theory of changes to identify key risks, solution pathways, stakeholders, based on the comparative advantages of UNDP. Lessons learned from evaluations such as this one also helps feed into the project formulation.

UNDP takes note of the need to strengthen a holistic approach to capacity building, a core objective of the project.

UNDP also concurs with the statement that an overall log frame is needed to help guide the project in achieving its goals and objectives, and to establish efficient monitoring and reporting structures. The project results framework will help assist the definition of the project results to support the planning, management and monitoring of development activities. This is another lesson learned which will be considered during project formulation should there be a second phase.

Should UNDP pursue a second phase of the project, subject to funding availability and capacity to implement, UNDP will review the implications of this recommendation and reflect them accordingly into the results framework(s), as applicable.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Should a second phase of the project be pursued, UNDP to reflect the implications of this recommendation into a new results framework(s for the project.
[Added: 2019/01/08] [Last Updated: 2020/01/31]
TBD – subject to situation of SSC portfolio in the new organizational structure (currently being redesigned) 2019/12 No Longer Applicable [Justification: No second phase of the project will be persuaded, the evaluation lessons are noted for any future South-South S Cooperation programmes. ]
History
3. Recommendation:

For implementation the following is recommended:

  • The evaluation scorings showed a significant need for detailed communication strategy

The strategy should define:

  • Who should know what – and the purpose
  • Media for sharing (TV, radio, conferences, reports, well-designed printed briefs, short breakfast meetings etc.)
  • Frequency of sharing with each type of stakeholders
Management Response: [Added: 2019/01/08] [Last Updated: 2021/01/05]

BPPS welcomes the recommendation and concurs with the importance of having a clear communication strategy. This will be considered together with the other recommendations when defining the roles and responsibilities, and how to communicate results. A communication strategy will need to be carefully discussed with the implementing stakeholders, national Governments and the Project Board.

UNDP would like to note that implementation of this recommendation is contingent on a continuation of the project.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Should a second phase of the project be pursued, UNDP to reflect the implications of this recommendation into a new results framework(s for the project.
[Added: 2019/01/08] [Last Updated: 2020/01/31]
TBD – subject to situation of SSC portfolio in the new organizational structure (currently being redesigned) 2019/12 No Longer Applicable [Justification: No second phase of the project will be persuaded, the evaluation lessons are noted for any future South-South S Cooperation programmes. ]
History
4. Recommendation:

Recommendations for project exit include:

  • The negative effect of lack of a project exit strategy was the concern of several stakeholders.  Irrespective of the timeframe there is need for an exit strategy preparing the mind-set of all stakeholders that this is the time for a take-over.

Documents, whether policies, instructions or others cannot replace a planned exit, which is more practical than intellectual.

Management Response: [Added: 2019/01/08] [Last Updated: 2021/01/05]

BPPS appreciates this recommendation and strongly supports the importance of having a clear exit strategy to ensure sustainability of results developed as part of a project.  The exit strategy should be built into the design of the project but remain flexible enough to be modified throughout the project cycle. UNDP will consider this recommendation in line with the established policy and guidance on programme and project management (PPM), and subject to a decision to continue the project.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Should a second phase of the project be pursued, UNDP to reflect the implications of this recommendation into a new project document that will incorporate findings from the evaluation on the exit strategy.
[Added: 2019/01/08] [Last Updated: 2020/01/31]
TBD – subject to situation of SSC portfolio in the new organizational structure (currently being redesigned) 2019/12 No Longer Applicable [Justification: No second phase of the project will be persuaded, the evaluation lessons are noted for any future South-South S Cooperation programmes. ]
History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org