Evaluation of National Area-Based Development Project Phase-III

Report Cover Image
Evaluation Plan:
2010-2014, Afghanistan
Evaluation Type:
Mid Term Project
Planned End Date:
04/2014
Completion Date:
07/2014
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
50,000

Share

Document Type Language Size Status Downloads
Download document Report - NABDP Evaluation.pdf report English 3062.06 KB Posted 800
Download document ToR - NABDP Evaluation.pdf tor English 525.41 KB Posted 332
Title Evaluation of National Area-Based Development Project Phase-III
Atlas Project Number: 00081443
Evaluation Plan: 2010-2014, Afghanistan
Evaluation Type: Mid Term Project
Status: Completed
Completion Date: 07/2014
Planned End Date: 04/2014
Management Response: Yes
Focus Area:
  • 1. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.1. National and sub-national systems and institutions enabled to achieve structural transformation of productive capacities that are sustainable and employment - and livelihoods- intensive
Evaluation Budget(US $): 50,000
Source of Funding: Project own funding
Joint Programme: No
Mandatory Evaluation: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Sue Emmott Evaluation Team Leader
Abdul Moien Jawhary Evaluator AFGHANISTAN
GEF Evaluation: No
Key Stakeholders: Ministry of Rural Rehabilitation and Development
Countries: AFGHANISTAN
Lessons
Findings
Recommendations
1 Building on what exists: NABDP has an eleven year history of intention to implement an area-based approach but actually doing something different. What MRRD and the staff of NABDP are comfortable with is delivering rural infrastructure, predominantly through CDCs and using a community development methodology. Change does not come easily and concepts such as economic growth and large scale livelihoods are outside the experience of most of those involved. Using an institutional development lens, we know that it is important to start from where an organisation actually is. The risk of the new design is that it will start in a different place which is unfamiliar, not well understood, and potentially resisted. Things which would remain the same are the focused approach and an emphasis on livelihoods. The evidence of NABDP suggests that neither of these will happen because they are politically unfeasible or conceptually misunderstood. To introduce an even more complicated livelihoods concept will risk the same misunderstanding and the same default to infrastructure delivery. The big change would be requiring several ministries to collaborate together. This will almost certainly fail. Afghan ministries are still in a state of competition and, like many other countries, have not yet arrived at the point where they can collaborate. Experience in NABDP shows that even collaboration with MAIL was too difficult because the focus of the ministries was very different and the mechanism for collaboration did not exist. A less risky design would start with infrastructure and focus all efforts on trying to make it more productive. Combined with developing sustainable operation and maintenance arrangements, this would be a huge improvement on NABDP. The model might follow a similar path to micro hydro power, starting as a component but, at the point that it is proving successful, transforming into a separate project in order to take it to scale.
2 Agreeing a realistic, affordable and sustainable implementation mechanism: With the loss of DDAs, this is the single biggest challenge for a new design. Although the what? is not straightforward, the how? is very complicated. NABDP has almost been acting like an NGO within government. It has tried to do community development, which is not usually a government function, and, where there was no implementation mechanism, it created one for its own purpose. These have not been wrong, and they have certainly given some positive results. But it has only been possible because donors have been prepared to support the approach at a time when many have plenty of resources. This situation is already changing. If MRRD tries to implement through cluster CDCs it is basically trying to do the same thing all over again, but on a scale that is unmanageable. It would also pin NABDP to NSP at a time when separation is more appropriate. NSP will face its own sustainability challenges. Two things can potentially change in a new design. One is to give PRRDs a bigger role in planning. With the experience gained over the last decade, many of them are in a position to understand the province well and to plan for rural development without needing to get the ideas from the communities. They would not exclude communities but would develop their own plans and use the DCCs as a consultative mechanism at district level to ensure the plans were broadly supported. This would be a normal government function. It is not, however, straightforward. In the absence of a decentralisation policy, the PRRDs are currently responsible for implementing programs from central level and do not have their own decision making powers. It would require a changing relationship between MRRD and the PRRDs and a form of pilot of how a decentralised relationship might work. A success of NABDP has been its experimentation so evidence suggests that a different kind of experiment would be possible if there was buy-in to the concept. The second change would be to contract more work to the private sector. Experience in some provinces has been positive and allows larger scale construction which may bring greater benefit. If community mobilisation is required, the contractors could be required to build this component into their bid. Alternatively, contracts could be given to qualified NGOs. A design in which contracting out was a methodology would require less staff and different skills sets and would therefore be more efficient. New mechanisms would not need to be built in for quality control as the PRRDs already have an M&E unit which can be strengthened.
3 Breaking with governance: Perhaps the biggest challenge of a new design is a decision about whether or not to continue with a governance function. The remainder of Phase 3 will be a collaboration between MRRD and IDLG in the capacity assessment and transition of capable DDAs to DCCs. Whilst MRRD and NABDP staff are keen to continue with their baby a number of complications would likely arise. IDLG has less capacity than MRRD and is only piloting in four provinces. If MRRD retains capacity in a follow on project, the temptation will be to continue business as usual. The timeframe for change would likely keep expanding and NABDP Phase 4 would happen by default. This is not desirable. Therefore it is not recommended that governance support be included in the follow on to NABDP because governance is not the mandate of MRRD and focusing on core mandate is important at this time of transition. It might appropriately fit, for UNDP, with support to IDLG.
4 Increasing impact for rural women: Creating separate projects for women has confined them to traditional roles in a small corner of NABDP and isolated them from the far larger program in infrastructure. Their income generating projects are weakly linked to markets and the value chain and so have little impact. Continuing with this kind of approach is therefore not recommended. The number of women's projects is very small and there are not enough staff to ensure quality. Increasing the staff would not be appropriate because it is not the role of government to engage in small scale projects at local level. More impact is likely to be realised by bringing women into the main stream. Productive rural infrastructure is intended to improve livelihoods for all people, increasing their incomes and sustaining productivity in the long term. In a new design, more attention can be given to analysis of womens' role in the value chain and to the establishment of a process by which each infrastructure project can be analysed in terms of the differential impact on men and women. This will enable choices that maximise the contribution of both and increase overall value for families and communities. Genuine economic empowerment often requires both men and women to take on roles that may be outside the traditional gender stereotypes. This is long term work but it is appropriate and feasible, at least in some parts of Afghanistan if not yet in the most conservative areas.
5 Timing: The timing of design is unfortunate, running in parallel with Presidential elections. A change of government is certain and the new government, probably with new ministers, will take time to settle. Designing a program at this time risks not having the buy-in of the new government, either because it predates it or because it happens before it has had chance to be clear on priorities. An option would be to extend NABDP, at least until the end of 2015. On balance the findings of the evaluation suggest that change of any kind is a long, difficult and slow process. If NABDP closes, a great deal would be lost and a new start up would not be effective for a long time. Buying extra time would allow the design to be developed over a longer period with more time to work through the difficult issues and a greater chance of genuine ownership.
6 The role of UNDP: UNDP has had limited oversight of NABDP and has lost opportunities to influence outcomes. This is always a challenge under NIM but, in a system which is politicised and not yet mature, a stronger role for UNDP is recommended. It has provided a good standard of financial oversight but oversight of program in terms of focus and quality has been weak. In part this role needs to be built into the design carefully so that UNDP's role is clear and accepted by MRRD. But NABDP went off track from the design and major problems such as the lack of alignment between planning and budget were beyond the influence of UNDP international staff. Some of the solutions may be quite simple, such as NABDP's own suggestion that the annual plans be approved and then not allowed to be altered. This builds in protection for those who come under political pressure and enables them to resist.
1. Recommendation: Building on what exists: NABDP has an eleven year history of intention to implement an area-based approach but actually doing something different. What MRRD and the staff of NABDP are comfortable with is delivering rural infrastructure, predominantly through CDCs and using a community development methodology. Change does not come easily and concepts such as economic growth and large scale livelihoods are outside the experience of most of those involved. Using an institutional development lens, we know that it is important to start from where an organisation actually is. The risk of the new design is that it will start in a different place which is unfamiliar, not well understood, and potentially resisted. Things which would remain the same are the focused approach and an emphasis on livelihoods. The evidence of NABDP suggests that neither of these will happen because they are politically unfeasible or conceptually misunderstood. To introduce an even more complicated livelihoods concept will risk the same misunderstanding and the same default to infrastructure delivery. The big change would be requiring several ministries to collaborate together. This will almost certainly fail. Afghan ministries are still in a state of competition and, like many other countries, have not yet arrived at the point where they can collaborate. Experience in NABDP shows that even collaboration with MAIL was too difficult because the focus of the ministries was very different and the mechanism for collaboration did not exist. A less risky design would start with infrastructure and focus all efforts on trying to make it more productive. Combined with developing sustainable operation and maintenance arrangements, this would be a huge improvement on NABDP. The model might follow a similar path to micro hydro power, starting as a component but, at the point that it is proving successful, transforming into a separate project in order to take it to scale.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

The recommendation is noted; It has been fully communicated with programme's formulation mission and to the extent possible will be taken into consideration during the design of new programme. However, considering below factors, the current notion is to diverge from solely rural infrastructure to more livelihood and income generating activities in both rural and semi-urban areas, which is expected to create sustainable jobs and improve agriculture products as opposed to the targets of current NABDP. The concept note of the new programme responds to changing thinking about rural development, not only in low- and middle-income countries, but also in high-income countries. There are limits on rural enterprise growth and rural job creation resulting from thin local consumer and labour markets and unreliable or unavailable utilities. These limits can however be lifted for a subset of rural enterprises through their becoming sub-contractors for larger, probably urban-based firms. Enterprise development in urban settings is much easier than in rural areas, given more reliable utilities, access to other services, better inter-connectivity with markets, and thicker labour markets. The strength of rural-urban economic inter-linkages means that separating rural and urban development is not meaningful and that the general stronger growth of demand and firms in urban areas can provide a stimulus to agriculture and to rural enterprise. The programme therefore is expected to target the below sub outputs: 1. skills in the target areas match labour market demand and the rural population (particularly young people) is able to access rural and urban jobs, and female labour force participation and employment rates increase; 2. existing firms in urban areas and the connected rural areas increase turnover and employment; and 3. agriculture in the urban periphery and more distant surrounding rural grows and responds to increasing and changing market demand by producing more, different and better quality food. The new project is expected to work with four different ministries. These are ministry of rehabilitation and rural develop, ministry of agriculture, ministry of commerce and ministry of labor and social affairs. While there has been competition between government institutions before, we believe this is not the case anymore. The experience of IDLG and MRRD working together on development and implementation of the District Coordination Councils is only one example how two previously adverse institutions are ready to work together. Shrinking in the funds, which was witnessed in2014 and is expected to continue, will also force the government institutions to work together. While coordination might be a challenge between four ministries, we believe having one programme board, regular interactions with all ministries and strong management staff can ensure cooperation between these ministries. It is well noted that we cannot initiate the new project in a big scale. The new project will be piloted in limited areas, before it is expanded to more areas based on the lessons learned from the pilot phase.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Recruitment of consultants for the design of the new project.
[Added: 2014/10/26] [Last Updated: 2016/03/14]
SNGDU 2015/12 Completed Consultant hired and Project document has been developed.
Pilot the new project in four provinces before it is expanded to more provinces
[Added: 2014/10/26] [Last Updated: 2015/10/31]
SNGDU/new project management 2015/12 Completed The project will be piloted in few targeted provinces. The project is scalable conditional to availability of financial resources and security.
2. Recommendation: Agreeing a realistic, affordable and sustainable implementation mechanism: With the loss of DDAs, this is the single biggest challenge for a new design. Although the what? is not straightforward, the how? is very complicated. NABDP has almost been acting like an NGO within government. It has tried to do community development, which is not usually a government function, and, where there was no implementation mechanism, it created one for its own purpose. These have not been wrong, and they have certainly given some positive results. But it has only been possible because donors have been prepared to support the approach at a time when many have plenty of resources. This situation is already changing. If MRRD tries to implement through cluster CDCs it is basically trying to do the same thing all over again, but on a scale that is unmanageable. It would also pin NABDP to NSP at a time when separation is more appropriate. NSP will face its own sustainability challenges. Two things can potentially change in a new design. One is to give PRRDs a bigger role in planning. With the experience gained over the last decade, many of them are in a position to understand the province well and to plan for rural development without needing to get the ideas from the communities. They would not exclude communities but would develop their own plans and use the DCCs as a consultative mechanism at district level to ensure the plans were broadly supported. This would be a normal government function. It is not, however, straightforward. In the absence of a decentralisation policy, the PRRDs are currently responsible for implementing programs from central level and do not have their own decision making powers. It would require a changing relationship between MRRD and the PRRDs and a form of pilot of how a decentralised relationship might work. A success of NABDP has been its experimentation so evidence suggests that a different kind of experiment would be possible if there was buy-in to the concept. The second change would be to contract more work to the private sector. Experience in some provinces has been positive and allows larger scale construction which may bring greater benefit. If community mobilisation is required, the contractors could be required to build this component into their bid. Alternatively, contracts could be given to qualified NGOs. A design in which contracting out was a methodology would require less staff and different skills sets and would therefore be more efficient. New mechanisms would not need to be built in for quality control as the PRRDs already have an M&E unit which can be strengthened.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

Noted. The recommendation has been taken into consideration during the design of the new programme. At the concept level, the new programme has captured this recommendation very well. PRRD are expected to act as one of the main implementing partners in the provinces for the same reasons mentioned in the recommendation. In addition, the municipalities (provincial or district) would be also key partners in implementation of this new project. The DDAs will not cease to exist immediately. It will be a gradual process to transfer the current representative bodies at the district level to District Coordination Councils (DCCs). While DCCs will not implement projects, they can plan and also monitor development work at the district level. Instead, the CDCs or cluster CDCs will be used to implement basic infrastructure projects which are needed for livelihood purposes in the selected districts. The CDCs are mostly used for implementation of infrastructure projects under NABDP even. Therefore, there would not be a big shift in terms of implementation modality of infrastructure projects. For creating livelihood opportunities, private sector should be more involved and DDAs or DCCs could not make a big difference even if they had a role. While the municipalities and PRRDs are official partners from the government, the private sector plays a key role in implementation of the project. The private sector is not only trained for market research and productive manufacturing, they will also be contracted to do the actual job. For this purpose, the ministry of commerce and industries, chamber of commerce and ministry of labor and social affairs are engaged in the project.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure full involvement of ministry of commerce, ministry of labor and social affairs and private sector in designing and implementation of the new project
[Added: 2014/10/26]
SNGDU 2014/05 Completed
3. Recommendation: Breaking with governance: Perhaps the biggest challenge of a new design is a decision about whether or not to continue with a governance function. The remainder of Phase 3 will be a collaboration between MRRD and IDLG in the capacity assessment and transition of capable DDAs to DCCs. Whilst MRRD and NABDP staff are keen to continue with their baby a number of complications would likely arise. IDLG has less capacity than MRRD and is only piloting in four provinces. If MRRD retains capacity in a follow on project, the temptation will be to continue business as usual. The timeframe for change would likely keep expanding and NABDP Phase 4 would happen by default. This is not desirable. Therefore it is not recommended that governance support be included in the follow on to NABDP because governance is not the mandate of MRRD and focusing on core mandate is important at this time of transition. It might appropriately fit, for UNDP, with support to IDLG.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

Noted. Agreed. Communicated during the design of the new programme. For the same reasons mentioned in the recommendation, it is expected that ILDG will have a bigger role to carry out the governance functions in the rural level, and MRRD will support them in development activities The implementation of the DCC policy is subject to the new government/administration endorsement. Considering the fact that the new administration (the National Unity Government) has agreed to conduct District Elections for the district councils, the DCC policy may not get implemented. However, if the new government agrees with establishment of the DCCs, the local governance project will take the lead on establishing and supporting the DCCs. Furthermore, all efforts are used to create more harmony between the new local livelihood project and the new local governance project. The thinking is to create one umbrella programme led by a senior person and appoint project managers for each project. In addition, joint programme plan and joint board meetings are the other tools that will be used. The two projects are designed in a way so they complement each other. In fact, support to subnational governance through the local governance project aim is to improve service delivery.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure that the role and responsibilities are clear for each implementing partner i.e. IDLG and MRRD to carry out the governance and service delivery functions under the local governance project
[Added: 2014/10/26] [Last Updated: 2015/10/31]
SNGDU 2015/03 Completed The recommendation is taking care of while developing the new project document.
Make sure there are joint board meetings, joint plans, joint field missions, etc. for the two new projects.
[Added: 2014/10/26] [Last Updated: 2018/10/23]
SNGDU 2015/10 Completed
4. Recommendation: Increasing impact for rural women: Creating separate projects for women has confined them to traditional roles in a small corner of NABDP and isolated them from the far larger program in infrastructure. Their income generating projects are weakly linked to markets and the value chain and so have little impact. Continuing with this kind of approach is therefore not recommended. The number of women's projects is very small and there are not enough staff to ensure quality. Increasing the staff would not be appropriate because it is not the role of government to engage in small scale projects at local level. More impact is likely to be realised by bringing women into the main stream. Productive rural infrastructure is intended to improve livelihoods for all people, increasing their incomes and sustaining productivity in the long term. In a new design, more attention can be given to analysis of womens' role in the value chain and to the establishment of a process by which each infrastructure project can be analysed in terms of the differential impact on men and women. This will enable choices that maximise the contribution of both and increase overall value for families and communities. Genuine economic empowerment often requires both men and women to take on roles that may be outside the traditional gender stereotypes. This is long term work but it is appropriate and feasible, at least in some parts of Afghanistan if not yet in the most conservative areas.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

Noted. Agreed. Communicated during the design of the new programme and is expected to get reflected in the pro-doc. The new project will work with the private sector at the district level. There would be a strong emphasis on supporting businesses that can benefit the women and women are active in. We will also try to find business belonging to women and support them under the project. In addition, UNDP held a number of discussions with UNWomen on implementation of safe market for women concept. Based on this concept, markets would be made safer so women can work there and also access to markets without any fear.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Ensure the role of women in the project is streamlined and recognized in all aspects.
[Added: 2014/10/26] [Last Updated: 2015/10/31]
SNGDU 2015/03 Completed The recommendation is taking care of while developing the draft project document
Work closely with UNWomen and Gender Project of UNDP on application of safe market concept
[Added: 2014/10/26]
SNGDU 2014/05 Completed
5. Recommendation: Timing: The timing of design is unfortunate, running in parallel with Presidential elections. A change of government is certain and the new government, probably with new ministers, will take time to settle. Designing a program at this time risks not having the buy-in of the new government, either because it predates it or because it happens before it has had chance to be clear on priorities. An option would be to extend NABDP, at least until the end of 2015. On balance the findings of the evaluation suggest that change of any kind is a long, difficult and slow process. If NABDP closes, a great deal would be lost and a new start up would not be effective for a long time. Buying extra time would allow the design to be developed over a longer period with more time to work through the difficult issues and a greater chance of genuine ownership.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

NABDP ends by mid-2015, by then the new administration will be fully in place. There is no further extension foreseen for the current NABPD, unless the project board decides otherwise. The design of the new progarmme also progresses in parallel with the new administration. The final pro-doc will be pending till it gets discussed with the new administration and receives their endorsement.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Start consultation with the new administration and particularly the ministers of the institutions involved on the new project.
[Added: 2014/10/26] [Last Updated: 2015/10/31]
UNDP Senior DCD/SNGDU 2015/03 Completed Thorough Consultations have been conducted while developing new project document.
6. Recommendation: The role of UNDP: UNDP has had limited oversight of NABDP and has lost opportunities to influence outcomes. This is always a challenge under NIM but, in a system which is politicised and not yet mature, a stronger role for UNDP is recommended. It has provided a good standard of financial oversight but oversight of program in terms of focus and quality has been weak. In part this role needs to be built into the design carefully so that UNDP's role is clear and accepted by MRRD. But NABDP went off track from the design and major problems such as the lack of alignment between planning and budget were beyond the influence of UNDP international staff. Some of the solutions may be quite simple, such as NABDP's own suggestion that the annual plans be approved and then not allowed to be altered. This builds in protection for those who come under political pressure and enables them to resist.
Management Response: [Added: 2014/10/26] [Last Updated: 2014/10/26]

Noted. Will be communicated and taken into consideration during the design of the implementation/oversight mechanisms of the new programme. The current NABDP is mostly about infrastructure projects and that is the main reason for political influence over selection of projects. On the contrary, the new project is about livelihood with infrastructure that improves livelihood in pre-selected areas. This will substantially reduce the level of political influence, since the districts are already selected based on assessments and only those infrastructure projects will be built which are connected to livelihood opportunities. In addition, we plan to examine the implementation with the plans on quarterly basis in the board meetings of the project. This way, the government partners are made to implement the already approved plans or receive no objection from UNDP if the plan needs alteration.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
Project manager to report to board on compliance to the approved plans.
[Added: 2014/10/26] [Last Updated: 2015/10/31]
Project Manager 2015/07 Completed

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org