Mid-term Governance Outcome Evaluation

Report Cover Image
Evaluation Plan:
2015-2019, Timor-Leste
Evaluation Type:
Outcome
Planned End Date:
12/2018
Completion Date:
12/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Title Mid-term Governance Outcome Evaluation
Atlas Project Number: 96725,86475,72017,82979,14955
Evaluation Plan: 2015-2019, Timor-Leste
Evaluation Type: Outcome
Status: Completed
Completion Date: 12/2018
Planned End Date: 12/2018
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.1.1 Capacities developed across the whole of government to integrate the 2030 Agenda, the Paris Agreement and other international agreements in development plans and budgets, and to analyse progress towards the SDGs, using innovative and data-driven solutions
Evaluation Budget(US $): 50,000
Source of Funding: UNDP, Cost Sharing
Evaluation Expenditure(US $): 23,000
Joint Programme: Yes
Mandatory Evaluation: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Charlemgane Sphia Gomez Consultant
GEF Evaluation: No
Key Stakeholders:
Countries: Timor-Leste
Comments:

by 2020, state institutions are more responsive, inclusive,accountable and decentralized of service delivery and improved service delivery and realization of rights, particular of the most excluded group.

Lessons
1.

Lessons Learned 1: Timorsation of the process is positive and is well received by all stakeholders. Projects which are cofunded by the government are particularly welcomed and ownership of the projects and its results are strongest in these types of projects.


2.

Lesson learned 2: A mixed approach to team composition is the most cost effective and efficient, this can include either a national or international project manager, at least 50% of Timorese staff and UNVs. Individual consultants can be called upon for specific areas of the projects.


3.

Lesson learned 3: Common language of understanding is essential and project managers where possible should be able to speak and understand Tetum and be familiar with the Timorese context. Nonetheless, international comparative experience is equally important in order to ensure a comprehensive approach to project management and technical inputs.


4.

Lesson learned 4: The use of graphic design tools and different formats of messaging is important to increase visibility and to better inform about projects´ success and provide an overview of the institution/area being supported.


5.

Lesson learned 5: Access to sufficient funds is important, nonetheless the approach in which the project is managed and executed is equally important to ensure more efficient and effective results.


6.

Lesson learned 6: Periodic reviews/evaluation should take place for all projects in order to ensure accountability, transparency and responsiveness of the projects themselves.  Mid-term reviews can guide projects and review their modus operandi if they are not on track to achieve their results
 


7.

Lesson learned 7: Indicators that tend to be only quantitative do not denote change and therefore make it difficult to ascertain the true progress of the project.


8.

Lessons learned 8: Indicators alone do not necessarily reflect the success of the project.


9.

Lesson Learned 9: The lack of horizontal linkages between the projects has weakened the ability of the projects to collectively impact on the UNDAF and CPAP outcomes.


10.

Lesson learned 10: Individual UNDP projects are very unlikely to impact on the UNDAF and CPAP outcomes as desired.


11.

Lessons learned 11: More formalised coordination and collaboration is needed to make a greater impact on the UNDAF and CPAP outcomes


12.


 Lesson learned 12: Lack of an exit strategy for key support to institutions weakens the projects ability to be sustainable.


Findings
1.

Conclusion 1: Each of the six projects are very relevant to both the UNDAF outcome and the CPAP outcome and align with national priorities.
 


2.


 Conclusion 2: The projects fulfil the needs of the beneficiaries and are demand driven rather than supply driven.


3.

Conclusion 3: National ownership of the projects and the results are very strong, this is especially pertinent for projects which are co-funded by the government.


4.

Conclusion 4: The projects have been effective to varying degrees. If we look towards the individual projects, a number of them have already reached their designated targets and are on track to fulfilling the outcome goals set by the individual projects. Nonetheless, the current political crisis has impacted on all projects in a number of different ways, resulting in some of the activities being delayed and in some cases implementation being almost halted.
 


5.

Conclusion 5: The design of the projects render it almost impossible for the governance outcomes to be achieved at the level of UNDAF and CPAP. This is due to a poor integral strategy which crosses over all six projects as well as the lack of a theory of change approach to designing the projects under evaluation.


6.

Conclusion 6: The lack of a robust and comprehensive programme M & E framework and the fact that the majority of the projects have not been evaluated during the period under assessment has made it very difficult to assess with accuracy the successes of the individual projects to achieve the governance outcomes.
 


7.

Conclusion 7: The lack of SMART indicators and relevant targets means that the projects are unable to demonstrate the potential change that could be attributed to the individual projects.


8.

Conclusion 8: There is no integral comprehensive strategy to achieve the outcome goals and projects tend to be isolationist and designed to work towards outputs rather than outcomes.


9.

Conclusion 9: UNDP is seen as a reliable and transparent partner and its support to government institutions is deemed to add value by the institutions and counterparts.


10.

Conclusion 10: UNDP and the country office in particular have experienced a lull in funds especially for governance.  This trend is slightly different at the regional level whereby funds for governance are increasing.  At the world level, governance is seen as the second most important portfolio of UNDP.


11.

Conclusion 11: Projects are considered to be efficient, especially given the access to such a restricted amount of funds.


12.

Conclusion 12: Programme management is seen as efficient, however a mixed approach of having experienced persons at the head of the project, combined with a heavy Timorese component and specific areas covered by UNVs seems to be ideal.  Individual consultants can be called for specific needs when they arise.


13.

Conclusion 13: M & E framework needs to be more robust. To this end, it is essential that an internal review takes place of the current framework in order to ascertain its relevance and effectiveness to track progress of the outcomes and not just the outputs.


14.

Conclusion 14: The projects which were delayed in activities due to the political situation, should be extended in order to ensure that they are able to complete the activities in the timespan allotted.


15.


 Conclusion 15: Capacity building is deemed as having an impact on individuals, however service delivery has been prioritised over institution building.


16.

Conclusion 16: Institution building needs the buy-in of all key actors, and although systems have been developed, a number of these systems have not been integrated into the daily work of the institutions


17.

Conclusion 17: There tends to be over reliance on UNDP and its projects, and while appreciated, more strategic approaches towards medium and longterm transfer of knowledge is needed in order to improve impact and eventual sustainability of the results achieved. 
 


18.

Conclusion 18: While gender was included in some of the prodocs, there does not exist a comprehensive and inclusive strategy to ensure that the project incorporates gender at all levels of the project. Isolated activities with specific focus are important, however projects should have gender and human rights at the core of their strategy.
 


19.

Conclusion 19: While conscious efforts were afforded to ensure the inclusion of women and gender aspects in the projects to a certain extent. Youth and children were not captured in the same light. This is especially important considering that over half of Timorese population is under the age of 25.


Recommendations
1

Recommendation 1: Conduct an internal review of the current M & E framework for the CPAP and the projects under governance in order to ascertain their relevance and effectiveness.  SMART indicators are essential for appropriate and effective tracking of progress.

2

Recommendation 2: Design and implement base line surveys for the governance sector in order to feed into the next CPAP and UNDAF as well as the future projects under Governance. This can be an initiative of the Governance Unit itself and should be implemented within the last six months of the current projects. 

3

Recommendation 3: Adequate mapping of the governance sector is essential in order to guide future projects as well as being utilised as a tool to comprehend the gaps, duplication and possible areas of collaboration and cooperation with different actors working on governance. 

4

Recommendation 4: Explore ways on how to formalise collaborative and cooperative efforts with other UN agencies, especially UN Women, UNFPA and UNICEF.  Seeking advice of other agencies is not enough, working together on key issues is more efficient, effective and will inevitably have a greater impact.

5

Recommendation 5: Ensure that project activities are more based on effective research, and where possible, ensure a multi stake holder approach. IE UN Agencies, Government Institution and CSOs.

6

Recommendation 6: Consider conducting training on gender responsiveness projects for the governance unit and the projects themselves.  This can also involve the key counterparts.  Training should comprise a training on what gender responsiveness is and the different ways all projects can make their projects more gender responsive.  

7


 Recommendation 7:  Ensure that gender, youth and children are incorporated into project activities and subsequent results and that relevant indicators are included at all levels of implementation.

8

Recommendation 8: Ensure that project design, implementation and management takes on a human right based approach to development.
 

9

Recommendation 9: Develop ways of ensuring better visibility and outreach for each of the projects.  Use of graphic design tools to illustrate progress as well as to inform the general public about the progress and work of individual institutions is key.
 

10

Recommendation 10: Ensure the extension of the projects whose implementation were impacted by the 2018 early elections.

11

Recommendation 11: In order to measure the transfer of knowledge, relevant indicators should also be established at all levels on the success of capacity building/institution building.

1. Recommendation:

Recommendation 1: Conduct an internal review of the current M & E framework for the CPAP and the projects under governance in order to ascertain their relevance and effectiveness.  SMART indicators are essential for appropriate and effective tracking of progress.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

This recommendation is partially relevant and an assessment of the indicators will be conducted with the support by the Governance team of the Regional Hub. However, the CPAP is only going to be valid for one more year, therefore the assessment will not influence the current framework, but could be used for inputs to the new one.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Arrange for a detailed assignment or a meeting.
[Added: 2019/01/16] [Last Updated: 2019/01/30]
UNDP Governance 2019/03 Initiated UNDP has been implementing the necessary changes for the UN reform, and been leading some inter-agency groups (UNCG) and participating to the rest of them (youth, gender). Currently working together with all agencies on International Women Day
1.2 Review the M&E CPAP framework based on their effectiveness and relevance.
[Added: 2019/01/16] [Last Updated: 2019/01/30]
UNDP Governance 2019/03 Not Initiated UNDP has just submitted a Joint Programme proposal with UNFPA, UN Women and IOM under the coordination of UNRC.
2. Recommendation:

Recommendation 2: Design and implement base line surveys for the governance sector in order to feed into the next CPAP and UNDAF as well as the future projects under Governance. This can be an initiative of the Governance Unit itself and should be implemented within the last six months of the current projects. 

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. This is an interesting proposal, which the Unit is gladly going to try to develop together with the Project Managers and Chief Technical Advisors.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Creating a baseline survey with inputs by the Project Managers and CTA
[Added: 2019/01/16] [Last Updated: 2019/02/13]
PMs and CTA of Governance 2019/04 Initiated The person supporting this work has postponed her travel to April, and therefore this will be done then.
2.2 Interview the national counterparts through the contacts by the projects
[Added: 2019/01/16]
Project Staff of projects under Governance and current national counterparts 2019/03 Not Initiated
2.3 Summarize the inputs from the national counterparts into a common input for the new UNDAF and CPAP.
[Added: 2019/01/16]
UNDP Governance 2019/04 Not Initiated
3. Recommendation:

Recommendation 3: Adequate mapping of the governance sector is essential in order to guide future projects as well as being utilised as a tool to comprehend the gaps, duplication and possible areas of collaboration and cooperation with different actors working on governance. 

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. A mapping exercise is high on the to do list.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Create a mapping exercise of the governance sector in Timor Leste
[Added: 2019/01/16]
UNDP Governance 2019/04 Not Initiated
3.2 Share the mapping internally and institutionalize it in the shared folders.
[Added: 2019/01/16]
UNDP Governance 2019/04 Not Initiated
3.2 Share the mapping internally and institutionalize it in the shared folders.
[Added: 2019/01/16]
UNDP Governance 2019/04 Not Initiated
4. Recommendation:

Recommendation 4: Explore ways on how to formalise collaborative and cooperative efforts with other UN agencies, especially UN Women, UNFPA and UNICEF.  Seeking advice of other agencies is not enough, working together on key issues is more efficient, effective and will inevitably have a greater impact.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agree. The UN Reform will definitely support the developments of cooperation in that direction. The cooperation with those specific agencies has gradually increased in the last few months with concrete initiatives being implemented.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Follow the developments and constructive cooperation within the UN reform
[Added: 2019/01/16] [Last Updated: 2019/02/13]
UNDP 2019/05 Initiated UNDP is taking numerous actions for the delinking process, and is participating in inter-agency groups and cooperations on gender (international women day), youth, M&E, programme, communication, etc.
4.2 Take concrete steps to engage the mentioned agencies in creation of joint proposals in the future relying on their specific expertise
[Added: 2019/01/16] [Last Updated: 2019/01/30]
UNDP Governance 2019/03 Initiated
5. Recommendation:

Recommendation 5: Ensure that project activities are more based on effective research, and where possible, ensure a multi stake holder approach. IE UN Agencies, Government Institution and CSOs.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

The team partially agrees with this recommendation, as we do consider that all projects are based on research, but the combination of stakeholders could be strengthened.  

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Add a guidance for the engagement of stakeholders to the mapping of governance stakeholders exercise.
[Added: 2019/01/16]
UNDP Governance 2019/05 Not Initiated
6. Recommendation:

Recommendation 6: Consider conducting training on gender responsiveness projects for the governance unit and the projects themselves.  This can also involve the key counterparts.  Training should comprise a training on what gender responsiveness is and the different ways all projects can make their projects more gender responsive.  

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. This is one of the elements of the Gender Seal Certification process which UNDP is undertaking, however at the first stage only focused on UNDP staff. One of the outcomes of the process is that all future projects are programmed through a gender and youth lens.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 Develop an Action Plan for increasing long term gender equality results through the Gender Seal Certification for the project staff.
[Added: 2019/01/16] [Last Updated: 2019/01/30]
Gender Focal Team 2019/01 Completed
6.2 Implement the Action Plan for the Gender Seal, specifically the elements on delivering gender sensitive to the project staff.
[Added: 2019/01/16] [Last Updated: 2019/01/27]
Gender Focal Team 2019/12 Initiated
6.3 Evaluate and self-assess on the Gender Seal indicators
[Added: 2019/01/16] [Last Updated: 2019/01/30]
Gender Focal Team 2020/01 Not Initiated Gender Seal Focal Team is working on implementing and monitoring the Action Plan.
7. Recommendation:


 Recommendation 7:  Ensure that gender, youth and children are incorporated into project activities and subsequent results and that relevant indicators are included at all levels of implementation.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. This is one of the elements of the Gender Seal Certification process which UNDP is undertaking. In addition, the Youth Focal Point should add to the youth mainstreaming elements. One of the outcomes of the process is that all future projects are programmed through a gender and youth lens.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
7.1 Develop an Action Plan for increasing long term gender equality results through the Gender Seal Certification for Programme staff.
[Added: 2019/01/16] [Last Updated: 2019/01/30]
Gender Focal Team 2019/01 Completed
7.2 Implement the Action Plan for the Gender Seal and youth mainstreaming, specifically the elements on programming for future projects.
[Added: 2019/01/16] [Last Updated: 2019/01/27]
Gender Focal Team 2019/12 Initiated
7.3 Evaluate and self-assess on the Gender Seal indicators
[Added: 2019/01/16]
Gender Focal Team 2020/01 Not Initiated
8. Recommendation:

Recommendation 8: Ensure that project design, implementation and management takes on a human right based approach to development.
 

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. A training on human rights approach to development needs to be organized.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
8.1 Concept Note for a training on HR approach to development developed.
[Added: 2019/01/16] [Last Updated: 2019/02/14]
UNDP Governance 2019/03 Initiated We have already started taking courses and developing a Concept Note, to be coordinated with the resident Human Rights Advisor.
8.2 Trainer from the Regional Hub or from the Human Rights Resident Advisor recruited for assignment.
[Added: 2019/01/16]
UNDP Governance 2019/03 Not Initiated
8.3 Programme and leading project staff trained in human rights approach to development.
[Added: 2019/01/16] [Last Updated: 2019/01/28]
UNDP Governance 2019/04 Not Initiated
9. Recommendation:

Recommendation 9: Develop ways of ensuring better visibility and outreach for each of the projects.  Use of graphic design tools to illustrate progress as well as to inform the general public about the progress and work of individual institutions is key.
 

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. One of the projects is very successful in doing this, and the other projects will be encouraged to hire communication staff to develop products to summarize the results achieved, especially for those towards the end of their timeline.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
9.1 Agree on the resources required for the assignment
[Added: 2019/01/16]
Project Managers and CTA 2018/12 Completed
9.2 Create ToRs for those projects that do not have communication staff/consultants
[Added: 2019/01/16] [Last Updated: 2019/02/13]
Project Managers and CTA 2019/03 Completed The ToR has been developed for AC&D and the JSP projects. PNTL is developing one.
9.3 Hire communication staff/consultants
[Added: 2019/01/16] [Last Updated: 2019/02/14]
Project Managers and CTA 2019/03 Initiated Already recruited for LEARN, ZEESM, JSP, AC&D. PNTL still to recruit.
9.4 Create communication products for each project
[Added: 2019/01/16]
Communication staff / consultants 2019/03 Not Initiated
10. Recommendation:

Recommendation 10: Ensure the extension of the projects whose implementation were impacted by the 2018 early elections.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed. This has already been done for three of the six running projects, in addition to a budget revision and moving activities to the upcoming year for one of the projects.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
10.1 Request no cost extensions by Japan for LEARN, JSP and PNTL, as well as from the Republic of Korea SDG Trust Fund for LEARN and JSP. Ask for budget revision for Anti-corruption and Decentralization by KOICA for moving activities for next year.
[Added: 2019/01/16]
UNDP Governance 2018/11 Completed No-Cost extension was approved by Japan for: LEARN, JSP and PNTL. No-Cost extension was approved by RoKTF for LEARN and JSP. Budget revision was approved for AC&D by Koica.
11. Recommendation:

Recommendation 11: In order to measure the transfer of knowledge, relevant indicators should also be established at all levels on the success of capacity building/institution building.

Management Response: [Added: 2018/12/03] [Last Updated: 2019/01/16]

Agreed, but it is challenging to influence all creation of indicators in the future projects.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
11.1 M&E specialist to support the work of the Governance Programme
[Added: 2019/01/16]
SSRI PMU 2020/01 Not Initiated
11.2 Revise capacity building indicators of ongoing projects
[Added: 2019/01/16] [Last Updated: 2019/02/13]
SSRI PMU 2019/04 Not Initiated We need to make an assessment of all indicators first, in order to make this review. The assessment of all indicators is to take place in April.
11.3 Give recommendations for improvements of current capacity building indicators
[Added: 2019/01/16]
SSRI PMU, PSC, MSA 2019/03 Not Initiated
11.4 Create a guild for how to create measurable capacity building indicators
[Added: 2019/01/16]
SSRI PMU, MSA, UNDP CO 2019/04 Not Initiated
11.5 Conduct an internal training on using measurable capacity building indicators
[Added: 2019/01/16]
SSRI PMU, MSA 2019/05 Not Initiated

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org