- Evaluation Plan:
- 2017-2022, Montenegro
- Evaluation Type:
- Final Project
- Planned End Date:
- 12/2021
- Completion Date:
- 12/2021
- Status:
- Completed
- Management Response:
- Yes
- Evaluation Budget(US $):
- 20,000
PAR project evaluation
| Document | Type | Language | Size | Status | Downloads |
|---|---|---|---|---|---|
 OP-31 X 21 PAR Evaluation Report.pdf |
related-document | English | 723.66 KB | Posted | 418 |
 ToR Project Evaluation_PAR_June11.docx |
tor | English | 83.92 KB | Posted | 412 |
| OP-31 X 21 PAR Evaluation Report.pdf |
report | English | 723.66 KB | Posted | 419 |
| Title | PAR project evaluation | ||||||
|---|---|---|---|---|---|---|---|
| Atlas Project Number: | 00110139 | ||||||
| Evaluation Plan: | 2017-2022, Montenegro | ||||||
| Evaluation Type: | Final Project | ||||||
| Status: | Completed | ||||||
| Completion Date: | 12/2021 | ||||||
| Planned End Date: | 12/2021 | ||||||
| Management Response: | Yes | ||||||
| Focus Area: |
|
||||||
| Corporate Outcome and Output (UNDP Strategic Plan 2018-2021) |
|
||||||
| SDG Goal |
|
||||||
| SDG Target |
|
||||||
| Evaluation Budget(US $): | 20,000 | ||||||
| Source of Funding: | Government | ||||||
| Evaluation Expenditure(US $): | 14,000 | ||||||
| Joint Programme: | No | ||||||
| Joint Evaluation: | No | ||||||
| Evaluation Team members: |
|
||||||
| GEF Evaluation: | No | ||||||
| Key Stakeholders: | Ministry of Public administration, EU, State institutions, municipalities | ||||||
| Countries: | MONTENEGRO |
| Lessons | |||
|---|---|---|---|
| 1. | Through synergetic layering of the groundwork, trust-building and piloting of different activities and approaches, the project was catalytic in furthering UNDP and public administration reform programming and in inviting strong donor interest The project achievements also provided the basis for exploring higher-level policy options including the opportunities for improving the policy making process related to interconnectivity between the eGovernance and key areas of the public administration reform. |
||
| 2. | The capacity development of national partners at the systemic, organisational and individual levels should be seen as an investment in both future programming and as a means of maximising partnerships. The capacity development should be seen as a long-term effort that needs to be embedded in broader change processes that are owned and driven by those involved, that are context-specific and that are as much about changing values and mindsets through incentives, as they are about acquiring new skills and knowledge. Also, UNDP should make a distinction between functional and technical capacities and support the development of both. The exact mixture of capacities to be addressed through a capacity development response will depend on the outcome of a capacity assessment. This will further contribute towards the long-term sustainability of the project results. |
||
| 3. | The project requires standardised mechanisms for learning, in particular from its monitoring and evaluation efforts that can be reflected both in the project implementation, as well as fed into the CPD programmatic cycle. The next project cycle should factor in a midterm evaluation as a way to (re)adjust the course of action during the project implementation towards transformative results. |
||
| 4. | The project should develop a systematic mechanism to capture lessons learned and preserve the project results to the extent possible. This includes preparation of analysis, case studies, lessons learned reports, document gathering etc. |
||
| 5. | As a result of the analysis and assessments undertaken a more informed project can be designed, which is realistic and attainable given the operational realities, the opportunities, and constraints on the ground. A small set of high quality, measurable indicators should be developed and reviewed to ensure that the indicators are not only clearly defined but are also representative, reliable, and feasible. A greater use of qualitative indicators that measure perceptions and behaviours at the outcome level, as opposed to quantitative indicators that measure activities at the output level, will likely better capture project progress and results, as well as contributions towards the outcomes and impact. |
||
| Findings | |||
| 1. | The project design reflects UNDP positioning UNDP has a strong comparative advantage within the Public Administration sector and is recognized by the national partner as the only viable partner in the Public Administration Reform Sector, particularly in relation to eGovernance, human resource management including performance management, accountability and transparency of services. That position should be leveraged and capitalized on in the next programming cycle. The understanding of the importance of the mainstreaming of the position of the UNDP as the key partner of Government of Montenegro in the public administration reform sector for the next programmatic cycle should inform the investment of CORE funds. In addition to a clearly defined thematic focus UNDP should strategically use the existing momentum to elevate from a stand-alone project to a programmatic platform. |
||
| 2. | Strategic Prioritisation The evaluator finds that the project has a clear results framework, based on a coherent narrative and profile that is also linked to the CPD, and is attractive to national partners, donors, and other development partners. As stated, the Project successfully mitigated the suboptimal implementation of the optimisation plan at the local level (output 1) and facilitated the interconnectivities between key state registers via SISEDE (output 2). The evaluator also finds the there is space for some programmatic revision and strategic re-focusing in terms of reformulating future outputs and the rationale behind the project, in order to ensure that the outputs are delivered and contribute to higher-level outcomes in a coherent manner, and that the project is successfully implemented. Indicators should be revised to better capture the achievements of the project and their contribution to higher-level outcomes. There should be greater emphasis on qualitative indicators, which capture the voice of people, and in particular the most vulnerable, which would provide a clear pathway between activity – output – outcome – impact, as well as show progress towards change. In view of the fact that the project is constantly evolving to needs and context, it is recommended that the project set tailor made outcome level indicators, that would provide the goalposts for the project, while the output level indicators could be more flexible and adapted to circumstances as they change. It is recommended to revise the project efficiently so that the Project Board could approve the revisions, and in the most cost-effective and efficient manner. |
||
| 3. | Communication of Results The evaluator finds that communication with the national partners and donor took place on a regular, even daily basis. Crucially, the regular and open communication did allow for sufficient flexibility in terms of adapting the project activities in a substantial manner during the project implementation period. This allowed UNDP to align the project with the changing local political, legal and institutional context and design additional activities that were more suited and more achievable given the realities of the operational context. The evaluator finds that communication mechanisms are very strong, however, communication could be further strengthened, through improving dialogue and the exchange of information, to ensure the quality assurance of the project results. UNDP needs to be continuously involved in dialogue with the project partners proactively addressing the challenges and limitations. A regular and active exchange with partners, with transparent communication of both positive and negative experiences will lead to sustainable partnership relations. The evaluator finds that UNDP has successfully integrated the project into its wider portfolio, however it should go beyond just the communication of results and develop a strategic communication plan to raise the partnership profile. |
||
| 4. | Project Expertise National partners highly valued the project staff’s capacity, expertise, experience as well as commitment. The technical knowledge and skills of the project staff are assessed as excellent and fit for the task at hand. The project staff is also recognized as the ones that encourages innovative thinking and provide space for creative input and that value input from stakeholders and partners across the sector. In addition to the support provided in implementation of the project activities the national partners find that the project staff were always ready to assist above and beyond in the development of policies, leadership skills, partnering abilities and institutional absorption capabilities in order to achieve and sustain development results. At the same time the evaluator finds that the external experts did not always provide the needed technical support at the right level. As per the national partners’ view the external experts did not always have sufficient local contextual knowledge. Consequently, this created situation where some of the analytical findings and recommendations were not fully implementable. |
||
| 5. | Project Approach The evaluator recognises that all project partners are satisfied with the flexibility and responsiveness of UNDP to their needs. This has positioned UNDP as a preferred partner. However, the downside of this approach is that the assistance provided can be perceived as fragmented and developed on an ad hoc basis. It is perceived among some partners that UNDP has not fully maximised strategic cooperation and partnerships. This can potentially dilute UNDP’s positioning as a policy partner since it prevents the adequate strategic profiling of the organisation, meaning that UNDP is sometimes perceived as an organiser of events or provider of a logistical support while it is striving to be recognised as a leader, or master in public administration reform in Montenegro. Further, the changing normative, institutional, and political context within which the project was implemented required implementing partners to keep funding modalities that allowed for a more responsive, flexible and adaptive approach to programmatic activities. While the project has already taken steps towards developing a more partner-orientated focus, in the next phase the project should move this a step beyond. This should include building a higher degree of trust, adequately communicating and managing expectations and limitations, seizing opportunities and creating solutions. |
||
| 6. | Sustainability The evaluator finds that the project has very strong sustainability potential though the project duration was originally projected for 24 months (May 2018 - May 2020) and then extended until October 2021. The ensured long-term relevance related to UNDP support to the Government of Montenegro in developing the new PAR strategy 2022 - 2026, generated systemic institutional buy-in and operationalisation of the EU Sector Budget Support financing instrument present strong evidence of the achieved sustainability.
|
||
| 7. | The 2030 Agenda While the project is highly relevant and tailored to the national context, including the EU Accession process and in particular political criteria related to the public administration reform, there is potential to position the project more strategically within the national and global sustainable development context. Although the EU accession process provides the key reform agenda for Montenegro, there are additional international commitments, which Montenegro is obligated to fulfil. The evaluator finds that the project could have been more closely aligned to the 2030 Agenda and the Sustainable Development Goals, which provides UNDP’s thematic niche for strategic positioning as a policy partner. Montenegro has pledged itself internationally to meet the goals and targets of the 2030 Agenda, and any future programming should be positioned in such a way to assist Montenegro in fulfilling these international commitments. Particular reference should be made to SDG 16 and in particular targets 16.6 - Develop effective, accountable and transparent institutions at all levels; and 16.7 - Ensure responsive, inclusive, participatory and representative decision-making at all levels; and their corresponding indicators. |
||
| Recommendations | |
|---|---|
| 1 | It is recommended to explore linkages and synergies with other projects so that this project can be elevated as a programmatic platform. The evaluator recommends that in future programming, UNDP improve the strategic linkages between the project components and partners through designing it in an innovative manner as a “platform project“, where UNDP is gathering strategic partners around the same goal, without being at the centre of the project design. This project design is particularly applicable in complex thematic areas, such as public administration reform, which has multiple stakeholders and multiple development challenges. This offers on the one hand the possibility for donors to access and cover more partners within the same framework and on the other hand facilitates horizontal cooperation among national partners. However, the complexity of partnerships requires both individual tailor-made approaches towards certain project components but also a strategic overview of the entire intervention. There is a risk that although the projects are within the same framework they operate as silos. |
| 2 | It is recommended that the project review its strategic direction allowing for some prioritisation of activities, while retaining its ability to be flexible and opportunistic. In order to provide some strategic direction, the evaluator recommends that the future programming use eGovernance as an enabler for programming in the area of public administration reform. However, using eGovernment as the enabler requires going beyond the IT systems, tools and Infrastructure, because this does not automatically result in the streamlining and re-organization of government functions and transformation of roles and responsibilities. This needs to be part of a larger transformation process that underpin the human resource management, transparency and accountability. At the same time during the next programmatic cycle a more holistic approach, that aims to balance a focus on internal reform with a concern for the changing relations between public administrations and citizens and business, will be needed. Functional assessments, as a tool of structural reform, need to provide the analytical basis and guide for programming and implementation. That includes identifying: a) redundant functions; b) duplication between and within institutions at central and local levels; c) missing functions as well as the review of the match between the function and the staff engaged in performing. |
| 3 | The communication between the project and partners should reflect work processes and achieve complementarity and increase programmatic synergies. The evaluator also recommends that project staff take a strategic, solution-orientated approach, which will further position the project and move from the communication of activity results towards communication of the strategic impact of results. |
| 4 | In the future special attention should be given to the process of selecting external consultants. The external consultants should be carefully selected with appropriate local and regional expertise and experience so they could provide expertise calibrated to the local context. |
| 5 | UNDP needs to enhance its strategic approach in future programming in addressing partners needs and institutional priorities though adaptive programming, based on the experiences gained during the implementation of the project. This highlighted the need for responsive, flexible, and adaptive implementation based on the changing context and as such, UNDP should consider a move towards even more adaptive programming. This will allow for the shifting of priorities and resources where results are not being achieved. This approach will require strong and measurable system-based indicators and rigorous monitoring, including a regularly updated risk management framework to make adjustments to programming on a regular basis, but will allow for the potential of better results with the same resources. These should be systematically programmed and budgeted to allow for review, reflection and adaptation as required. The next phase of the project should be looking more at incubating and seeding in the public administration reform sector. This means the project is investing resources in multiple, concurrent small level pilots that may be based on successful approaches from within Montenegro or from external sources with similar contexts, and then to see which are able to be adapted to work in this country and in this political system. It is also recommended that during incubating and seeding UNDP further explore the cooperation with civil society organisations that could be instrumental in testing certain solutions and providing real time data that could be feed in into the project implementation. |
| 6 | In the next programmatic phase, it is strongly recommended that the issues of sustainability and national ownership are looked at in a more comprehensive way from the planning stage. UNDP should strive to synchronise capacity development process with the Government strategic and planning processes. It may be feasible to incorporate the capacity development process at the level of programmes/projects, across the formulation, implementation and review. Approaching capacity development through this process lens provides a rigorous and systematic way of supporting it; improves the consistency, coherence and impact of efforts; and helps promote a common frame of reference for a programmatic response to capacity development and the sustainability prospects. |
| 7 | It is recommended to use the 2030 Agenda as a cohesive element in exploring and maximizing organic synergies between the project components and partners. This would also contribute to raising the profile of future project activities and results at the national level and assist Montenegro in meeting its international obligations, as well as strengthening UNDP’s strategic position as the key partner. |
1. Recommendation:
It is recommended to explore linkages and synergies with other projects so that this project can be elevated as a programmatic platform. The evaluator recommends that in future programming, UNDP improve the strategic linkages between the project components and partners through designing it in an innovative manner as a “platform project“, where UNDP is gathering strategic partners around the same goal, without being at the centre of the project design. This project design is particularly applicable in complex thematic areas, such as public administration reform, which has multiple stakeholders and multiple development challenges. This offers on the one hand the possibility for donors to access and cover more partners within the same framework and on the other hand facilitates horizontal cooperation among national partners. However, the complexity of partnerships requires both individual tailor-made approaches towards certain project components but also a strategic overview of the entire intervention. There is a risk that although the projects are within the same framework they operate as silos.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the new UN Sustainable Development Cooperation Framework (UNSDCF) and CPD 2022-2026 formulation process. In the new CPD programme cycle (2022-2026), UNDP will focus on designing Public Administration Reform as a ‘’platform project’’. By mainstreaming of the position of the UNDP as the key partner of Government of Montenegro in the public administration reform sector, UNDP will use the existing momentum to elevate from a stand-alone project to a programmatic platform.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| New CPD/programming developed and validated with national partners
[Added: 2021/12/15] |
RR/Team Leaders | 2022/12 | Initiated |
2. Recommendation:
It is recommended that the project review its strategic direction allowing for some prioritisation of activities, while retaining its ability to be flexible and opportunistic. In order to provide some strategic direction, the evaluator recommends that the future programming use eGovernance as an enabler for programming in the area of public administration reform.
However, using eGovernment as the enabler requires going beyond the IT systems, tools and Infrastructure, because this does not automatically result in the streamlining and re-organization of government functions and transformation of roles and responsibilities. This needs to be part of a larger transformation process that underpin the human resource management, transparency and accountability.
At the same time during the next programmatic cycle a more holistic approach, that aims to balance a focus on internal reform with a concern for the changing relations between public administrations and citizens and business, will be needed. Functional assessments, as a tool of structural reform, need to provide the analytical basis and guide for programming and implementation. That includes identifying: a) redundant functions; b) duplication between and within institutions at central and local levels; c) missing functions as well as the review of the match between the function and the staff engaged in performing.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the new programming cycle. The strategic re-focusing will be made in terms of reformulating future outputs and the rationale behind the project, in order to ensure that the outputs are delivered and contribute to higher-level outcomes in a coherent manner, and that the project is successfully implemented.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Both recommendations will be addressed during next CPD programme cycle
[Added: 2021/12/15] |
Responsible unit Team Leader and Programme Management | 2026/12 | Initiated | History |
3. Recommendation:
The communication between the project and partners should reflect work processes and achieve complementarity and increase programmatic synergies. The evaluator also recommends that project staff take a strategic, solution-orientated approach, which will further position the project and move from the communication of activity results towards communication of the strategic impact of results.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the next programming cycle. Communication will be further strengthened, through improving dialogue and the exchange of information, to ensure the quality assurance of the project results. UNDP will be continuously involved in dialogue with the project partners proactively addressing the challenges and limitations. A regular and active exchange with partners, with transparent communication of both positive and negative experiences will lead to sustainable partnership relations. UNDP will go beyond just the communication of results and develop a strategic communication plan to raise the partnership profile.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Develop a strategic communication plan to raise the partnership profile and strengthening communication beyond communicating results
[Added: 2021/12/15] |
Responsible unit Team Leader and Programme Management | 2026/12 | Initiated |
4. Recommendation:
In the future special attention should be given to the process of selecting external consultants. The external consultants should be carefully selected with appropriate local and regional expertise and experience so they could provide expertise calibrated to the local context.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued immediately.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Ensure that external experts always provide the needed technical support at the right level
[Added: 2021/12/15] |
Responsible unit Programme Management | 2027/12 | Initiated |
5. Recommendation:
UNDP needs to enhance its strategic approach in future programming in addressing partners needs and institutional priorities though adaptive programming, based on the experiences gained during the implementation of the project. This highlighted the need for responsive, flexible, and adaptive implementation based on the changing context and as such, UNDP should consider a move towards even more adaptive programming. This will allow for the shifting of priorities and resources where results are not being achieved. This approach will require strong and measurable system-based indicators and rigorous monitoring, including a regularly updated risk management framework to make adjustments to programming on a regular basis, but will allow for the potential of better results with the same resources. These should be systematically programmed and budgeted to allow for review, reflection and adaptation as required.
The next phase of the project should be looking more at incubating and seeding in the public administration reform sector. This means the project is investing resources in multiple, concurrent small level pilots that may be based on successful approaches from within Montenegro or from external sources with similar contexts, and then to see which are able to be adapted to work in this country and in this political system.
It is also recommended that during incubating and seeding UNDP further explore the cooperation with civil society organisations that could be instrumental in testing certain solutions and providing real time data that could be feed in into the project implementation.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the new programming cycle.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Building a higher degree of trust, adequately communicating and managing expectations and limitations, seizing opportunities and creating solutions.
[Added: 2021/12/15] |
Responsible unit Team Leader and Programme Management | 2023/12 | Initiated |
6. Recommendation:
In the next programmatic phase, it is strongly recommended that the issues of sustainability and national ownership are looked at in a more comprehensive way from the planning stage. UNDP should strive to synchronise capacity development process with the Government strategic and planning processes. It may be feasible to incorporate the capacity development process at the level of programmes/projects, across the formulation, implementation and review. Approaching capacity development through this process lens provides a rigorous and systematic way of supporting it; improves the consistency, coherence and impact of efforts; and helps promote a common frame of reference for a programmatic response to capacity development and the sustainability prospects.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the new programming cycle. The exit strategy will specify the transition arrangements and sustain and/or scale-up results, as relevant. It will describe how national capacities would be strengthened and monitored as relevant.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Develop the exit strategy
[Added: 2021/12/15] |
Responsible unit Team Leader and Programme Management | 2023/12 | Initiated |
7. Recommendation:
It is recommended to use the 2030 Agenda as a cohesive element in exploring and maximizing organic synergies between the project components and partners. This would also contribute to raising the profile of future project activities and results at the national level and assist Montenegro in meeting its international obligations, as well as strengthening UNDP’s strategic position as the key partner.
Management Response: [Added: 2021/12/15]
The recommendation is accepted and will be pursued through the new programming cycle. Particular reference will be made to SDG 16 and in particular targets 16.6 - Develop effective, accountable and transparent institutions at all levels; and 16.7 - Ensure responsive, inclusive, participatory and representative decision-making at all levels; and their corresponding indicators.
Key Actions:
| Key Action | Responsible | DueDate | Status | Comments | Documents |
|---|---|---|---|---|---|
| Align future programming more closely to the Agenda 2030
[Added: 2021/12/15] |
Responsible unit Team Leader and Programme Management | 2023/12 | Initiated |