Mid-Term Evaluation of FATA Governance project

Report Cover Image
Evaluation Plan:
2018-2022, Pakistan
Evaluation Type:
Mid Term Project
Planned End Date:
10/2021
Completion Date:
10/2021
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Document Type Language Size Status Downloads
Download document TOR_Subject Specialist Evaluator_MAGP_IC_210202_v3_ZM.pdf tor English 265.96 KB Posted 740
Download document MAGP-Evaluation QA Checklist- Final Draft[1].docx.pdf related-document English 664.89 KB Posted 651
Download document UNDP-MAGP- Evaluation Report- Final -Oct 2021.docx report English 20992.06 KB Posted 597
Title Mid-Term Evaluation of FATA Governance project
Atlas Project Number: 00108127
Evaluation Plan: 2018-2022, Pakistan
Evaluation Type: Mid Term Project
Status: Completed
Completion Date: 10/2021
Planned End Date: 10/2021
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2018-2021)
  • 1. Output 1.1.1 Capacities developed across the whole of government to integrate the 2030 Agenda, the Paris Agreement and other international agreements in development plans and budgets, and to analyse progress towards the SDGs, using innovative and data-driven solutions
SDG Goal
  • Goal 10. Reduce inequality within and among countries
  • Goal 16. Promote peaceful and inclusive societies for sustainable development, provide access to justice for all and build effective, accountable and inclusive institutions at all levels
SDG Target
  • 10.3 Ensure equal opportunity and reduce inequalities of outcome, including by eliminating discriminatory laws, policies and practices and promoting appropriate legislation, policies and action in this regard
  • 16.6 Develop effective, accountable and transparent institutions at all levels
  • 16.7 Ensure responsive, inclusive, participatory and representative decision-making at all levels
Evaluation Budget(US $): 50,000
Source of Funding: Donor
Evaluation Expenditure(US $): 45,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Syed Rizwan Mehboob Mr syedrizwanmehboob@gmail.com PAKISTAN
GEF Evaluation: No
Key Stakeholders: UNDP, Government of KP
Countries: PAKISTAN
Lessons
Findings
Recommendations
1

1.It is therefore recommended that the Project’s design is updated to reflect these new securities and political realities of the merged areas and Khyber Pakhtunkhwa.

2

2. Decentralized implementation of the Project’s interventions in the coming months, with units moved to divisional headquarters could be explored as a solid alternative to the “province-heavy” delivery mechanism, adopted so far; this would likely reflect the grass root impact and ownership of local administrative structures in a more meaningful and impactful manner.

3

3. The Project’s work on the production of legal drafts, instruments and bills for the KP Government seems to be limited in terms of the required linkages with relevant Government departments. It is therefore recommended that the work under this stream be passed over to the UNDP’s Amn o Insaaf Project, as that project is better suited to undertake this work

4

4. The VC/NC data could be used to calculate a composite index that makes use of the various variables in the dataset to arrive at a single numeric value that could be used to rank VC/NCs on their overall level of development and availability of facilities. This will be useful in identifying priority VC/NCs as well as priority regions within the merged areas

5

5. The datasets produced by the Project fill a vacuum in data available on the merged areas. However, dissemination of these datasets could be improved. For starters, advocacy should be carried out with the KP Government to encourage them to allow public access for all the datasets. Furthermore, dissemination events could be carried out in major universities across the country to encourage students as well as academia to use the data for their research projects. Similarly, the datasets could be used to create more infographics to disseminate through social media accounts of the Project

6

6. Given the issues with the recruitment of survey firms. It is recommended that a Long-Term Agreement (LTA) is signed with a survey firm to avoid the procurement delays in the onboarding process. Similarly, efforts should be made to give the selected firm a special status with regards to NOC requirements for conducting the survey. This is expected to result in reduced processing times for these two crucial bottlenecks

7

7. The dearth of data in merged areas is such that the robocall/SMS surveys provide a convenient option for data collection. However, to truly assess the reliability of these surveys, there is a need to map the catchment areas of the telecom providers, to highlight areas that are not accessible via this technology. Furthermore, an assessment could be carried out to gauge the technical abilities, such as the ability to read and respond to text messages and robocalls, of the population of the merged areas. It is essential to explore the limitations of this method, before more rounds of such surveys are carried out.

8

8. It is recommended that the LGA Program be extended with an expanded scope of work. Besides awareness raising the LGAs could also perform enumeration duties for surveys. Given the constraints in conducting the regular surveys as well as SMS/Robocall surveys, the LGA presence and mobility in the merged areas could be leveraged to conduct surveys as well.

9

9:-  In terms of outreach, social media tend to have a higher outreach in some tribal districts than TV. The Project should conduct an analysis of the overall engagement using social media analytics to understand the demographics of those groups that are being reached through the social media campaign. This will help with finetuning the campaign according to the response from the audiences. The Local Government Ambassador (LGA) Program can play a crucial role in increasing the Project’s outreach. It is therefore recommended that the program be extended with an extended scope of the program. On awareness raising the primary focus should be on decentralized service delivery and grass root empowerment. Furthermore, other awareness-raising needs such as that on COVID-19 as well as on the landmines awareness issue could also be considered and developed in consultation with the Government of Khyber Pakhtunkhwa. In terms of reaching out to women, the LGA program could consider benefiting from the lessons learned in the SELP TDEA strategy in its successful outreach to women. The Project’s communication products could be disseminated through LGA-led community-level awareness sessions where the TVCs could be played on projectors, especially in areas with low TV and internet access. Characters created in the TVCs could also be promoted through flyers and pamphlets to leverage the recognition of these characters, such as the “Gubru”[1] character. 

 


 

10

10:- The Project’s Pashto communication products are in the most popular dialect of Pashto, i.e., the Peshawar valley dialect. While this is likely to be understandable in most of the tribal districts, but for the districts of Kurram, North Waziristan, and South Waziristan there might be some difficulty in understanding this dialect, especially for women as they tend to have lower exposure to other dialects than most men from these areas. Therefore, it is recommended that the local dialects for these tribal districts should also be considered for translation

1. Recommendation:

1.It is therefore recommended that the Project’s design is updated to reflect these new securities and political realities of the merged areas and Khyber Pakhtunkhwa.

Management Response: [Added: 2021/12/08]

Agreed with the Recomendation

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 MAGP is aware of the changing situation in the Merged Areas and will be updating the Project Document to reflect its strategic re-positioning, modified goals and updated management structure.
[Added: 2021/12/08]
PMU-UNDP 2022/06 Initiated
2. Recommendation:

2. Decentralized implementation of the Project’s interventions in the coming months, with units moved to divisional headquarters could be explored as a solid alternative to the “province-heavy” delivery mechanism, adopted so far; this would likely reflect the grass root impact and ownership of local administrative structures in a more meaningful and impactful manner.

Management Response: [Added: 2021/12/08]

Agreed with the Recommendation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1. District focus and strengthening is a top priority for MAGP in all its planning moving forward, and has been supported through deploying a team in each district of the MA. The team of 21 development experts is
[Added: 2021/12/08] [Last Updated: 2022/05/08]
MAGP Result Area 4: District support teams (led by the Field Augmentation Support team) 2022/06 Initiated History
3. Recommendation:

3. The Project’s work on the production of legal drafts, instruments and bills for the KP Government seems to be limited in terms of the required linkages with relevant Government departments. It is therefore recommended that the work under this stream be passed over to the UNDP’s Amn o Insaaf Project, as that project is better suited to undertake this work

Management Response: [Added: 2021/12/08]

agreed with the Recommendation nd its already completed.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1  Justice reform work has been carried out under the Rule of Law project; future RoL work will be undertaken by MAGP/Amn-o-Insaf subject to future funding availability and will be closely linked to AIP II.
[Added: 2021/12/08]
PMU 2021/12 Completed The project has already been taken care of it so its completed.
4. Recommendation:

4. The VC/NC data could be used to calculate a composite index that makes use of the various variables in the dataset to arrive at a single numeric value that could be used to rank VC/NCs on their overall level of development and availability of facilities. This will be useful in identifying priority VC/NCs as well as priority regions within the merged areas

Management Response: [Added: 2021/12/08] [Last Updated: 2021/12/08]

Not Agreed with the recommendation as any further work on VC/NC district profiles is not possible as it is the Local Government department’s property and is to be used at the discretion of the new LGs in the Merged Areas

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Not Applicable
[Added: 2021/12/08]
N/A 2020/12 No Longer Applicable [Justification: Any further work on VC/NC district profiles is not possible as it is the Local Government department’s property and is to be used at the discretion of the new LGs in the Merged Areas]
5. Recommendation:

5. The datasets produced by the Project fill a vacuum in data available on the merged areas. However, dissemination of these datasets could be improved. For starters, advocacy should be carried out with the KP Government to encourage them to allow public access for all the datasets. Furthermore, dissemination events could be carried out in major universities across the country to encourage students as well as academia to use the data for their research projects. Similarly, the datasets could be used to create more infographics to disseminate through social media accounts of the Project

Management Response: [Added: 2021/12/08]

Agreed with the Recomendation

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1- Currently MAGP holds dissemination of survey results with a closed group of donors and KP govt. since it is the government’s discretion to make the results of the survey's public. Results of the AIP baseline survey are publicly available on the KP government’s P&D website. For future results, MAGP will advocate for increased transparency and publicly available data.
[Added: 2021/12/08] [Last Updated: 2022/05/08]
UNDP- MAGP, PMU 2022/06 Initiated History
6. Recommendation:

6. Given the issues with the recruitment of survey firms. It is recommended that a Long-Term Agreement (LTA) is signed with a survey firm to avoid the procurement delays in the onboarding process. Similarly, efforts should be made to give the selected firm a special status with regards to NOC requirements for conducting the survey. This is expected to result in reduced processing times for these two crucial bottlenecks

Management Response: [Added: 2021/12/08]

agreed 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1 In the same vein, MAGP has just signed a long-term agreement with BOS for baseline data collection for selected SEPs, to build on the existing capacity and reach of the BOS and strengthen project partnerships.
[Added: 2021/12/08]
Project Management/Operations 2021/12 Completed In the same vein, MAGP has just signed a long-term agreement with BOS for baseline data collection for selected SEPs, to build on the existing capacity and reach of the BOS and strengthen project partnerships.
7. Recommendation:

7. The dearth of data in merged areas is such that the robocall/SMS surveys provide a convenient option for data collection. However, to truly assess the reliability of these surveys, there is a need to map the catchment areas of the telecom providers, to highlight areas that are not accessible via this technology. Furthermore, an assessment could be carried out to gauge the technical abilities, such as the ability to read and respond to text messages and robocalls, of the population of the merged areas. It is essential to explore the limitations of this method, before more rounds of such surveys are carried out.

Management Response: [Added: 2021/12/08]

agreed 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Not applicable
[Added: 2021/12/08]
M&E, MAGP Data analytics team 2021/12 No Longer Applicable [Justification: Limitations of this method were already discussed, and the design was amended for Round 3 of the mobile-based Citizen Poll of 2021, where the SMS option was dropped. Further analysis will be done when refining the design for future mobile-based surveys.]
Limitations of this method were already discussed, and the design was amended for Round 3 of the mobile-based Citizen Poll of 2021, where the SMS option was dropped. Further analysis will be done when refining the design for future mobile-based surveys.
8. Recommendation:

8. It is recommended that the LGA Program be extended with an expanded scope of work. Besides awareness raising the LGAs could also perform enumeration duties for surveys. Given the constraints in conducting the regular surveys as well as SMS/Robocall surveys, the LGA presence and mobility in the merged areas could be leveraged to conduct surveys as well.

Management Response: [Added: 2021/12/08]

Partially agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
N/A
[Added: 2021/12/08]
N/A 2021/12 No Longer Applicable [Justification: GA program has been discontinued moving forward due to lack of project funds, a factor beyond MAGP’s control]
9. Recommendation:

9:-  In terms of outreach, social media tend to have a higher outreach in some tribal districts than TV. The Project should conduct an analysis of the overall engagement using social media analytics to understand the demographics of those groups that are being reached through the social media campaign. This will help with finetuning the campaign according to the response from the audiences. The Local Government Ambassador (LGA) Program can play a crucial role in increasing the Project’s outreach. It is therefore recommended that the program be extended with an extended scope of the program. On awareness raising the primary focus should be on decentralized service delivery and grass root empowerment. Furthermore, other awareness-raising needs such as that on COVID-19 as well as on the landmines awareness issue could also be considered and developed in consultation with the Government of Khyber Pakhtunkhwa. In terms of reaching out to women, the LGA program could consider benefiting from the lessons learned in the SELP TDEA strategy in its successful outreach to women. The Project’s communication products could be disseminated through LGA-led community-level awareness sessions where the TVCs could be played on projectors, especially in areas with low TV and internet access. Characters created in the TVCs could also be promoted through flyers and pamphlets to leverage the recognition of these characters, such as the “Gubru”[1] character. 

 


 

Management Response: [Added: 2021/12/08]

agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
9.1- MAGP’s communication team has already started doing some analysis for the social media demographics, which is dependent on the resources and software available. The future work plan includes efforts to reach more online audiences of the MA by video boosting and strategies to increase online traffic. LGA cannot be used for outreach as that component is now closed.
[Added: 2021/12/08] [Last Updated: 2022/04/03]
SCU 2022/05 Initiated steps have been taken to meet the key action and the current AWP 2022 also covers this. They action will be completed by May 2022 History
10. Recommendation:

10:- The Project’s Pashto communication products are in the most popular dialect of Pashto, i.e., the Peshawar valley dialect. While this is likely to be understandable in most of the tribal districts, but for the districts of Kurram, North Waziristan, and South Waziristan there might be some difficulty in understanding this dialect, especially for women as they tend to have lower exposure to other dialects than most men from these areas. Therefore, it is recommended that the local dialects for these tribal districts should also be considered for translation

Management Response: [Added: 2021/12/08]

agreed 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
10.1 MAGP is aware of the changing situation in the Merged Areas and will be updating the Project Document to reflect its strategic re-positioning, modified goals and updated management structure.
[Added: 2021/12/08]
PMU-UNDP 2022/06 Initiated

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org