United Nations Trust Fund for Human Security Canton 10 Project Final Evaluation Report

Report Cover Image
Evaluation Plan:
2015-2019, Bosnia and Hercegovina
Evaluation Type:
Final Project
Planned End Date:
12/2016
Completion Date:
03/2017
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
15,000

Share

Document Type Language Size Status Downloads
Download document UNTFHS Canton 10 Evaluation Expert ToR.pdf tor English 431.24 KB Posted 203
Download document RA C10 Final evaluation report FINAL.pdf report English 4113.38 KB Posted 218
Download document RA C10 Rapid Assessment FINAL.pdf report English 1498.91 KB Posted 148
Title United Nations Trust Fund for Human Security Canton 10 Project Final Evaluation Report
Atlas Project Number: 93068
Evaluation Plan: 2015-2019, Bosnia and Hercegovina
Evaluation Type: Final Project
Status: Completed
Completion Date: 03/2017
Planned End Date: 12/2016
Management Response: Yes
Focus Area:
  • 1. Democratic Governance
  • 2. Cross-cutting Development Issue
  • 3. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.1. National and sub-national systems and institutions enabled to achieve structural transformation of productive capacities that are sustainable and employment - and livelihoods- intensive
Evaluation Budget(US $): 15,000
Source of Funding: Project budget/donor resources
Evaluation Expenditure(US $): 18,945
Joint Programme: Yes
Joint Evaluation: Yes
  • Joint with UN Agencies
  • Joint with UNDP, UNHCR, UNICEF and IOM
Evaluation Team members:
Name Title Email Nationality
Richard Allen Evaluation consultant richard@richardallen.eu
Richard Allen Evaluation Consultant richard@richardallen.eu
GEF Evaluation: No
Key Stakeholders: Canton 10 of the Federation of Bosnia and Herzegovina, Municipalities Drvar, Bosansko Grahovo, Glamoc, Livno, Kupres, Tomislavgrad, Social Welfare Centers, Civil Society Organizations
Countries: BOSNIA AND HERCEGOVINA
Comments:

Due date of the evaluation is December 2016. The Evaluation Report is uploaded while the Management response is expected to follow within the prescribed three weeks, in January 2017. 

Lessons
1.

An inception period and report are essential: The absence of an inception report created difficulties, particularly at the beginning of the project, and this is seen in the overall slow rate of implementation in the first year. The inception period should also have provided critical baseline data, on which to make more informed assessments about project impact and effectiveness, and a monitoring and evaluation plan. Neither of these were produced, and this had a profound detrimental effect on the project;


2.

Project coordination is a full-time task: In general, considering the project budget and the range of activities, the project was under-staffed.


3.

An effective team needs interpersonal as well as technical skills: Poor project performance at the beginning of the project was attributed to the substandard interpersonal skills of the project coordinator.


4.

Project level monitoring is essential: For joint projects, project level monitoring and evaluation needs to be supported by a single budget, against a single plan, and with one agency responsible for leading and coordinating monitoring activities. An alternative might be to place budget and responsibility with the office of the Resident Coordinator, if the resources are available;


5.

Common activities require unified budgets;


6.

Quality project management needs quality assurance; Joint projects need joint project management procedures. A joint project also needs to have appropriate quality assurance procedures, in which key project outputs are properly scrutinised and approved. At the same time, there should not be duplication of effort in which a team member has to write, for example, a report for his/her own agency and another report for the joint project. Hence some effort needs to be invested in advance to work out the most sensible and efficient arrangements. Ideally, project management procedures would be backed up by an effective methodology and IT system to assist in the production, storage and access of project documents.


7.

More frequent reporting is needed for good project management; More frequent reporting at the project level is needed to identify issues, approve changes and to be assured that the project is on track. The reporting also needs to provide a record of the actions and decisions taken, particularly when it concerns any major changes to scale or scope.


8.

A clear lesson is that well before the end of the project, the question should have been asked whether there was a need and a possibility for such a development framework as in Canton 10 to continue, and if so, in what form.


Findings
1.

The planned inception period did not achieve the required outputs. Despite an inception report being a required deliverable, this was not produced, and the project continued based on the original budget and workplan. Since this workplan and budget were produced over a year earlier than the start of the project, there were some key changes in circumstances that required adaptation of the scale and scope of the plan and budget;

 

 

 

 


2.

The project coordinator was allocated to the task for only 50% of his time, and in total there were only around 4 full-time equivalent staff. This was not sufficient for project of this size and complexity. A project coordinator for a USD 2.5 million project needs to be 100% dedicated to the work and especially during the start-up phase. Consequences of this include the slow implementation rates at the beginning of the project, and the insufficient project management documentation throughout.


3.

Appointment of team members – especially in a complex environment such as this one in which diplomacy is required in relations between UN agencies and with local stakeholders – needs to be on the basis of their interpersonal skills and abilities in working with a team as much as their technical knowledge.


4.

The project did not have an overall monitoring plan, leaving the job up to individual agencies. This meant there were no standard reporting instruments except for the annual report format, and there were insufficient planned and budgeted activities to obtain required data for monitoring. Last-minute improvisation was able to get some information about the impact of the income generating activities, but this would have been more reliable and systematic if it had been properly planned and budgeted. Disputes between the agencies meant that the key monitoring tool, the planned Human Security Impact Assessment (HSIA) did not take place either at the beginning nor at the end of the project, and so the project’s overall impact on human security could not be reliably assessed.


5.

Promotional activities were originally planned and budgeted as separate activities of each of the implementing agencies. This was found to be highly inefficient, since there was no overall promotional plan and the amounts of money with each agency were relatively small. In the second year, these amounts were pooled and with the help of the UN expert with responsibility for public relations, a single plan was produced. The resulting communication efforts were more effective and able to promote the project as a whole (including the Trust Fund for Human Security ‘branding’).


6.

The project’s management procedures were particularly weak. Since there was a reliance on individual agencies’ own procedures, there were no coherent processes for planning, budgeting, reporting and coordination. These were instead ad hoc, and dependent on the time and initiative of the project team. Joint projects need joint project management procedures. At the same time, there should not be duplication of effort in which a team member has to write, for example, a report for his/her own agency and another report for the joint project. Hence some effort needs to be invested in advance to work out the most sensible and efficient arrangements.The absence of an inception report is a good example of a case in which a required output was not produced, and there was no quality assurance to flag that this was the case. Another example is the failure to produce the critical HSIA survey. Its absence was known by the project team, but there was no paper trail to explain why such a critical part of the project had not been produced, nor to flag the consequences for the project and wider donor accountability. Project management procedures need to be able to document key changes to activities and outputs, in order to justify decisions made, and to inform senior management.


7.

Reporting for at the project level was required only annually; other reporting was defined according to the internal procedures of each agency. This means that there were only two opportunities for the project management to see how the project was progressing throughout the 3 years and 3 months of project implementation – following year 1, and following year 2 when reports were compiled. This is not sufficient for project oversight. The reporting also needs to provide a record of the actions and decisions taken, particularly when it concerns any major changes to scale or scope. The annual reports available for this evaluation (years 1 and 2 only) were only partially able to do this. This gave the misleading impression that all was as planned; however, as we see from the actual expenditures against budget (see p 72), all was not as planned.

In development programmes, it is understood that things will change; activities are more or less expensive than budgeted, some things work better, some apparently good ideas don’t work at all, and opportunities arise that are too good to be missed. This is understood. What is needed is a way of reporting these that does not seek to blame, but rather seeks to inform and to learn. Better reporting should acknowledge that the plans will change, and to provide ways in which the (agreed) changes to plans can be incorporated, with justifications, and then clearly reported.

It is especially important to acknowledge when things are not working as early as possible, so that remedial action can be taken quickly. Leaving activities undone without reporting and identifying appropriate alternatives (such as the Human Security Impact Assessment) means that significant gaps arise that are then a surprise to senior management and donors much later.


8.

Time to exit or time to continue?

The project succeeded in many ways. One of the major successes was to establish a relationship and a framework in which international actors – in this case the UN country team – work closely with local authorities to achieve significant development results.

As the project now comes to a close, this relationship and framework will also come to a close. However, the development needs of Canton 10 are not resolved by any means: there are still major issues of exclusion and discrimination faced by minority returnees, slow economic growth painfully affects those of whichever ethnicity living in the remote rural areas and small towns.

While it is understood that there will not be any further support from this particular funding source – the UN Trust Fund for Human Security – it is not impossible to see other major donors providing the resources for future development work.


Recommendations
1

Continuing area-based development in Canton 10 and beyond. The project has established a solid basis for supporting future development in Canton 10. It has demonstrated a successful model of area-based development which is well-adapted to the context of Bosnia and Herzegovina. This includes effective mechanisms for planning and supporting implementation of development projects and institutional reforms. It also includes a relationship of trust between the UN country team and the local authorities; the UN Country Team has shown its capabilities to steward resources carefully and in the interests of the Canton as a whole, and this is appreciated by the local authorities.

The UN Country Team should consider building on this successful project and seek additional funds to continue the support to Canton 10, and apply a similar model in other parts of the country with extremely vulnerable populations – and in particular vulnerable minorities.

The EU’s Instrument for Pre-Accession Assistance (IPA, now IPA II) in other accession countries has financed similar area based development projects as a means to prepare local authorities for EU membership and the competences required to manage structural funds. The UN Country Team should consider opening a dialogue with the EU delegation in Bosnia and Herzegovina and the national authorities with responsibility for planning EU funding support to explore the potential for funding continued support for Canton 10, and other parts of the country.

2

Follow-up is necessary: The absence of an exit strategy for the project as a whole is a concern. The UN Country Team should consider formulating a follow-up plan for the achievements in Canton 10. There were many activities carried out in the last year of implementation, for which there will not be adequate follow up. Follow up is needed in key areas such as:

  1. Ensuring and advocating for continued funding for the mobile teams
  2. Tracking the progress and implementation of the law on free legal aid
  3. Ensuring continued appropriate use of communal assets such as the milk-collectors/coolers
  4. Assessing the extent to which the Canton Development Agency is working, and if any further support is needed. In particular, ensure that the governing model for the agency will provide appropriate safeguards for it to address needs of all the Canton’s citizens, not just one ethnic group.
  5. Following up the continuing work on violence prevention in schools.

Before the end of the project, the UN Country Team should consult with local authorities, and meet together to formulate what action is necessary, and possible within the available resources. The result should be a follow up action plan for an ‘exit’ phase of around 1 year.

3

Project management methodologies and documentation. A key weakness identified in the project was the absence of project management systems and procedures for joint UN projects. While each agency has its own internal project management procedures, when they work together, there is no clarity on what the reporting and management requirements are. As a consequence, there was inadequate management reporting and oversight of project deliverables.

The UN country team – and possibly their respective headquarters – should look at how to improve the methodologies, IT support, and documentation of joint projects. In the meantime, any future joint UN project should develop its own set of project management requirements, including, for example, inception reports and project logic revision, regular quarterly reports and management meetings, annual reporting against progress indicators, and minuted management and steering committee meetings.

4

Monitoring and Evaluation. The poor quality of technical aspects of project design that support monitoring and evaluation – notably the objectives and indicators – as well as the absence of a M&E plan, undermined the ability of the project to assess its results and impact. The UN Country Team should consider strengthening the capacities of personnel in project design and M&E, and at the same time, consider strengthening the role and involvement of the UN Country Team to ensure the quality of project design and M&E in all joint UN projects.

5

Decision-making in joint projects: Joint projects are complicated beasts, and they often require considerable negotiation between UN Agencies. In the case of the Canton 10 project, there was no formal procedure for decision-making agreed in advance. It was reported that under the MDG fund joint projects, it was a requirement that the Resident Coordinator had a formally agreed final decision-making power in case of disputes between individual agencies. The TFHS did not have such a requirement, and therefore there was some lack of clarity, hard decisions were avoided, and this resulted in tasks not being completed as planned (e.g. the HSIA survey). For future joint projects, the UN Country Team should establish clear decision-making procedures formally agreed in advance, including ways of rapidly resolving disputes between agencies.

6

Projects need adequate staff time: The Canton 10 project staffing was inadequate for a project of this size and scope. A coordinator allocated for only 50% of his time, and a total of 4 full time equivalent staff, was not sufficient to achieve all that was needed. This is not necessarily a question of the budget; there was sufficient funds available, but it is not clear that all staff were able to dedicate sufficient time to the project because of other work assignments competing for their time. That so much was achieved is a credit to the coordinator, the team, and their dedication. The UN Country Team should ensure that in future projects of this size there is at least one person full-time in the project coordinator role, and sufficient project staff to manage implementation. The link between the project coordination budget and the actual time spent by the project team members needs to be carefully monitored. To this end, project team members – particularly those not dedicated full time to the project – should complete weekly timesheets indicating the actual time spent on each project to which they are assigned.

1. Recommendation:

Continuing area-based development in Canton 10 and beyond. The project has established a solid basis for supporting future development in Canton 10. It has demonstrated a successful model of area-based development which is well-adapted to the context of Bosnia and Herzegovina. This includes effective mechanisms for planning and supporting implementation of development projects and institutional reforms. It also includes a relationship of trust between the UN country team and the local authorities; the UN Country Team has shown its capabilities to steward resources carefully and in the interests of the Canton as a whole, and this is appreciated by the local authorities.

The UN Country Team should consider building on this successful project and seek additional funds to continue the support to Canton 10, and apply a similar model in other parts of the country with extremely vulnerable populations – and in particular vulnerable minorities.

The EU’s Instrument for Pre-Accession Assistance (IPA, now IPA II) in other accession countries has financed similar area based development projects as a means to prepare local authorities for EU membership and the competences required to manage structural funds. The UN Country Team should consider opening a dialogue with the EU delegation in Bosnia and Herzegovina and the national authorities with responsibility for planning EU funding support to explore the potential for funding continued support for Canton 10, and other parts of the country.

Management Response: [Added: 2017/09/19]

This recommendation is valid and will be considered for implementation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Canton 10, given the underdeveloped status, remains in the focus of the UNCT. The recommendation will be reviewed further in the context of the UNDAF review planned for 2017 Q4 where the possibilities for further joint interventions in Canton can be discussed. UNCT has also initiated dialogue with the EUD to promote the idea that any further financial support from EU funds is pre-conditioned with adherence to methodologies and processes developed through this project.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNDP 2017/09 Completed Some project components are already encompassed and will be further developed by other ongoing projects working in Canton 10, as elaborated under Key Action 2. The UNDP Project Integrated Local Development Project III (ILDP III) continues to work with the Cantonal Government and the Cantonal Development Agency on further enhancement of their capacities with regards to planning and implementation of strategic priorities in Canton 10 and respective municipalities. History
2. Recommendation:

Follow-up is necessary: The absence of an exit strategy for the project as a whole is a concern. The UN Country Team should consider formulating a follow-up plan for the achievements in Canton 10. There were many activities carried out in the last year of implementation, for which there will not be adequate follow up. Follow up is needed in key areas such as:

  1. Ensuring and advocating for continued funding for the mobile teams
  2. Tracking the progress and implementation of the law on free legal aid
  3. Ensuring continued appropriate use of communal assets such as the milk-collectors/coolers
  4. Assessing the extent to which the Canton Development Agency is working, and if any further support is needed. In particular, ensure that the governing model for the agency will provide appropriate safeguards for it to address needs of all the Canton’s citizens, not just one ethnic group.
  5. Following up the continuing work on violence prevention in schools.

Before the end of the project, the UN Country Team should consult with local authorities, and meet together to formulate what action is necessary, and possible within the available resources. The result should be a follow up action plan for an ‘exit’ phase of around 1 year.

Management Response: [Added: 2017/09/19]

This recommendation is valid and will be considered for implementation

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Follow-up activities will be carried out through other ongoing projects implemented by the UN agencies in BiH, such as: -(Integrated Local Development Project (ILDP): Swiss-funded project implemented by UNDP, providing support for further development and institutionalization of the Cantonal Development Agency and the Cantonal Development strategy; -Access to Justice, funded by the BPPS - Bureau for Policy and Programme Support, working on implementation of the newly adopted Law on Free Legal Aid and establishment of the Cantonal Free Legal Aid Center; -Via Dinarica: A Platform for Sustainable Tourism Development and Local Economic Growth, a joint initiative by USAID and UNDP; -GED (Green Economic Development) project, funded by SIDA, working on institutionalization of energy management and implementation of infrastructural measures in the field of energy efficiency. -UNICEF Country Programme of cooperation and 2017-2018 JWP in ECD, Education and Child Protection
[Added: 2017/09/19] [Last Updated: 2018/03/30]
UNCT, UNDP, IOM, UNICEF, UNHCR 2018/03 Completed -IOM continues active lobbying for institutionalization of mobile teams in public budgets. -As part of its operational plan 2017-2018, UNHCR will continue providing individual legal aid assistance when required and based on a proper vulnerability/needs assessment. UNHCR will also closely monitor development in terms of free legal aid in Canton 10, improvement of the legislation and implementation at cantonal/entity/national levels. -UNDP continues actively working in Canton 10, through other projects that will cover the activities b, c and d, including continued monitoring of appropriate use of communal assets. -UNICEF remains engaged in Canton 10. As a follow up to the end of the project, UNICEF and the Cantonal authorities agreed a number of priority actions in Canton 10 as part of the 2017-2018 JWPs, particularly in the ECD, Education and Child Protection sectors. History
Meeting with mayors and representatives of C10 held and priorities for future priorities reviewed in line with the available strategies. The conclusions of the meeting presented at the project closure conference.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNCT, UNDP 2016/12 Completed As a follow up to the end of the project, UNICEF and the Cantonal authorities agreed a number of priority actions in Canton 10, as part of the 2017-2018 JWPs, particularly in the ECD, Education and Child Protection sectors. History
3. Recommendation:

Project management methodologies and documentation. A key weakness identified in the project was the absence of project management systems and procedures for joint UN projects. While each agency has its own internal project management procedures, when they work together, there is no clarity on what the reporting and management requirements are. As a consequence, there was inadequate management reporting and oversight of project deliverables.

The UN country team – and possibly their respective headquarters – should look at how to improve the methodologies, IT support, and documentation of joint projects. In the meantime, any future joint UN project should develop its own set of project management requirements, including, for example, inception reports and project logic revision, regular quarterly reports and management meetings, annual reporting against progress indicators, and minuted management and steering committee meetings.

Management Response: [Added: 2017/09/19] [Last Updated: 2017/09/19]

Agreeable in the part concerning improvements to JP implementation modalities. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNCT Guidelines for implementation of joint programmes, in line with the UNDG guidance note on JP, have been adopted to improve management processes of the joint projects.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNCT 2017/05 Completed History
4. Recommendation:

Monitoring and Evaluation. The poor quality of technical aspects of project design that support monitoring and evaluation – notably the objectives and indicators – as well as the absence of a M&E plan, undermined the ability of the project to assess its results and impact. The UN Country Team should consider strengthening the capacities of personnel in project design and M&E, and at the same time, consider strengthening the role and involvement of the UN Country Team to ensure the quality of project design and M&E in all joint UN projects.

Management Response: [Added: 2017/09/19]

This recommendation is partly agreeable 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
M&E capacities of UNDP and UNCT projects will be improved with the involvement of the M&E Specialist at CO level that will provide assistance to the projects.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNCT, UNDP 2017/06 Completed UNDP has revised its standard project document template to include Quality Assurance and M&E Frameworks. History
5. Recommendation:

Decision-making in joint projects: Joint projects are complicated beasts, and they often require considerable negotiation between UN Agencies. In the case of the Canton 10 project, there was no formal procedure for decision-making agreed in advance. It was reported that under the MDG fund joint projects, it was a requirement that the Resident Coordinator had a formally agreed final decision-making power in case of disputes between individual agencies. The TFHS did not have such a requirement, and therefore there was some lack of clarity, hard decisions were avoided, and this resulted in tasks not being completed as planned (e.g. the HSIA survey). For future joint projects, the UN Country Team should establish clear decision-making procedures formally agreed in advance, including ways of rapidly resolving disputes between agencies.

Management Response: [Added: 2017/09/19]

This recommendation is valid and will be considered for implementation

Key Actions:

Key Action Responsible DueDate Status Comments Documents
UNCT will ensure that clear decision-making hierarchy is clarified at any future joint project.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNCT 2017/09 Completed Guiding Note on Joint Projects was adopted in May 2017 and is in support of this recommendation History
6. Recommendation:

Projects need adequate staff time: The Canton 10 project staffing was inadequate for a project of this size and scope. A coordinator allocated for only 50% of his time, and a total of 4 full time equivalent staff, was not sufficient to achieve all that was needed. This is not necessarily a question of the budget; there was sufficient funds available, but it is not clear that all staff were able to dedicate sufficient time to the project because of other work assignments competing for their time. That so much was achieved is a credit to the coordinator, the team, and their dedication. The UN Country Team should ensure that in future projects of this size there is at least one person full-time in the project coordinator role, and sufficient project staff to manage implementation. The link between the project coordination budget and the actual time spent by the project team members needs to be carefully monitored. To this end, project team members – particularly those not dedicated full time to the project – should complete weekly timesheets indicating the actual time spent on each project to which they are assigned.

Management Response: [Added: 2017/09/19]

This recommendation is valid and will be considered for implementation

Key Actions:

Key Action Responsible DueDate Status Comments Documents
C10 was an exception in terms of the management arrangements applied. All future joint projects will continue the practice of having a full-time project coordinator and fully-involved project staff, which will be adequately reflected in both project document and project budget.
[Added: 2017/09/19] [Last Updated: 2017/09/26]
UNCT 2017/05 Completed Guiding Note on Joint Projects was adopted in May 2017 and is in support of this recommendation. History

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org