YouthStart Mid Term Evaluation

Report Cover Image
Evaluation Plan:
2012-2013, UNCDF
Evaluation Type:
Others
Planned End Date:
02/2013
Completion Date:
07/2013
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
150,000

Share

Document Type Language Size Status Downloads
Download document Terms of Reference_YouthStart_MidTermEvaluation.pdf tor English 948.90 KB Posted 497
Download document YOUTHSTART_MIDTERM_0713_ENG.pdf report English 1306.56 KB Posted 676
Download document ANNEXES_YOUTHSTART_MTE_ENG.pdf report English 1628.04 KB Posted 1470
Title YouthStart Mid Term Evaluation
Atlas Project Number:
Evaluation Plan: 2012-2013, UNCDF
Evaluation Type: Others
Status: Completed
Completion Date: 07/2013
Planned End Date: 02/2013
Management Response: Yes
Focus Area:
  • 1. Poverty and MDG
Corporate Outcomes (UNDP Strategic Plan 2008-20013)
  • 1. Inclusive growth and social equity promoted through pro-poor macroeconomic and fiscal policies that support income, employment and social protection of youth, women and vulnerable groups in a sustainable way
  • 2. Policies, strategies and partnerships established to enhance public-private sector collaboration and private sector and market development that benefit the poor and ensure that low-income households and small enterprises have access to a broad range of financial and legal services
  • 3. Strengthened national capacities to integrate into the global economic system and to negotiate and manage traditional & emerging development finance for inclusive development
Evaluation Budget(US $): 150,000
Source of Funding: Programme funds
Joint Programme: No
Mandatory Evaluation: Yes
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders: UNCDF staff, national partners, The MasterCard Foundation, programme beneficiaries
Location of Evaluation: Global
Countries: SENEGALBURKINA FASOETHIOPIAUGANDACONGO DRCMALAWITOGORWANDA
Comments: The mid-term evaluation of the YouthStart programme is being conducted as agreed in the project document (partnership protocol with The MasterCard Foundation) and in accordance with UNCDF's Evaluation Policy. The evaluation counts as a programme evaluation because programme activities are taking place in multiple countries simultaneously.
Lessons
Findings
Recommendations
1 Evaluation Recommendation 1: Give more precision, depth, and meaning to program reporting
2 Evaluation Recommendation 2: Improve the capacity for FSPs to conduct better monitoring and reporting
3 Evaluation Recommendation 3: Deepen engagement with key regional stakeholders
4 Evaluation Recommendation 4: Improve coordination with and leverage of the regional UNCDF / UNDP network
5 Evaluation Recommendation 5: Improve the efficiency and effectiveness of program management
6 Evaluation Recommendation 6: Considering a broader set of strategic drivers of successful YFS
7 Evaluation Recommendation 7: Focus TA for FSPs on consolidating business models
1. Recommendation: Evaluation Recommendation 1: Give more precision, depth, and meaning to program reporting
Management Response:

Giving more precision to the data collected, confirms one of the challenges the programme manager was facing and was already addressing. For example, reports? templates were refined during Q4 of 2012 (see MCF quarterly report) and UNCDF received for the first time the reports with the new template in April 2013 (reports of Q1 of 2013). Therefore, management agrees with this recommendation. However, management does not agree with the recommendation that the main indicator should be net accounts open as opposed to net active youth clients. This is because a youth could have a loan with the FSP and not necessarily have a savings account. Since the programme is about access to financial services and not only about accessing savings, it makes more sense to use ?net youth active clients? as the main indicator , as the definition os net youth active clients is ?The number of active borrowers, depositors, and other clients between 12 and 24 years old who are currently accessing any of the FSP?s financial services designed under the YouthStart programme. Youth who access multiple services with an FSP should be counted as a single client. Youth who are not currently receiving services are not included. Neither borrowers whose loans have been written off nor depositors who have not had a deposit, withdrawal, or interest earned in the past 12 months are considered to be active. (current period)? Although reasonable proxies were used at the inception of the programme (average savings balance lower than 20% of GDP per capita) to ensure the FSPs were reaching low income youth and that other indicators (eg. average balance savings of youth) are collected and analyzed on a quarterly basis, YS will encourage the FSPs to start collecting more meaning full data regarding level of poverty of clients. The indicator that will be used to determine the poverty level of youth clients will be ?number of meals during a day and will be collected using mini-surveys on twice per year using the sampling methodology known as Lot quality Assurance Sampling. Regarding the recommendation about developing a set of intermediary indicators that provides better barometers of programme efficiency and effectiveness, it is important to note that YS already tracks changes in key performance ratios among FSPs. We will, however, revise the result framework to make sure that other intermediary indicators are included (eg. Level of satisfaction of TA activities). Please note that even though, this is not added in the result framework, all our group based TA activities measure participant?s level of satisfaction. Finally, providing resources become available, YouthStart will increase technical assistance interventions like the one provided to Finca-DRC to help the FSPs gather qualitative data in an effort to better understand issues affecting young women and girls and provide a strong evidence of change. These stories will be collected and shared during webinars to increase lateral learning with partners and during conferences and publications to disseminate learning within the industry of youth financial inclusion.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1. Develop new template adding precision to indicators YouthStart Programme Manager (YPM) 2013/09 Completed
1.2. Ensure FSPs understood the new template and are entering accurate data (during September webinar) YouthStart Programme Manager (YPM) 2013/10 Completed Done for all 10 FSPs in December 2013.YS consultants did it on-site for every single FSP.
1.3. Test the new templates and address any doubts/inconsistencies of understanding YouthStart Programme Manager (YPM) 2013/12 Completed Done in December 2013. 7 out of the 10 FSPs were asked to send the Q42013 reports using the new template to test it and refinements were done to the template.
1.4. Develop a set of intermediary indicators that provide better barometers of programme efficiency (eg. Level of satisfaction of capacity building activities), include them in the results framework and start tracking them. YouthStart Programme Manager (YPM)with support of evaluation unit 2013/09 Completed FSPs started reporting through Survey Monkey in October 2013 and scoring system of monitoring visits completed.
1.5. Manage team of consultants to ensure qualitative data is collected by FSPs in an effort to better understand issues affecting young women and girls and provide a strong evidence of change YouthStart Programme Manager (YPM) 2015/06 Completed Permission for budget reallocation granted by MCF. Recruitment process started in Q12014.Data collection on the field started with financial diaries. The study will be ready by next year in June
2. Recommendation: Evaluation Recommendation 2: Improve the capacity for FSPs to conduct better monitoring and reporting
Management Response:

Improving the capacity for FSPs to conduct better monitoring and reporting was also another challenge already identified by the YPM and therefore this recommendation is also welcomed by management. To that end, the programme manager, in an effort to improve monitoring and reporting of outcomes at the youth level, planned and held the YouthStart Annual Training ( May 2013) on monitoring outcomes of the FSP?s youth programmes. During this training participants acquired the necessary knowledge and skills to start actively monitoring more depth and meaningful outcomes such as account usage, cross-selling (complementary products sold to youth clients and/or their family members) and levels of poverty of youth serve, using low-cost mini-surveys with the Lot Quality Assurance Sampling methodology. LQAS is a low-cost management tool that relies on small, randomly selected samples to assess performance of key indicators, as well as high and low performance. Developed for quality control in the manufacturing sector more than 75 years ago, LQAS is now widely used in the public health sector1 as a valuable management tool, and has been employed in unique ways more recently in microfinance across program units. The strength of LQAS is that it provides decision-makers with quick and cost-effective information to make programmatic decisions. It is not used to measure long-term impact, based on a hypothesis of direct causality, but rather, used to provide managers with the data they need to determine whether their programs or activities are reaching a certain, expected performance level. In June 2013, the FSPs submitted their monitoring plans. As a follow up to these activities, consultants will be providing them technical assistance on the design and implementation of the LQAS studies.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1. Commissioning a review of FSP MIS to identify major weaknesses or quick fixes that could help improve reporting YouthStart Programme Manager 2013/10 Completed Achieved in December 2013. Done by YS consultants.
2.2. Manage team of consultants to ensure FSPs are conducting the mini-surveys using LQAS to gather baseline data on more meaningful indicators regarding usage of accounts, cross-sells, improvement in savings behaviors, client satisfaction and levels of poverty of clients, using Lot Quality Assurance sampling methodology YouthStart Programme Manager 2013/09 Completed Mini-surveys questionnaires were refined and Action plans for the LQAS study submitted by all the FSPs.
2.3. Ensure FSPs are conducting the mini-surveys using LQAS to monitor progress on more meaningful indicators regarding usage of accounts, cross-sells, improvement in savings behaviors, client satisfaction and levels of poverty of clients, using Lot Quality Assurance sampling methodology YouthStart Programme Manager 2014/03 Completed Achieved in February 2014. All the 10 FSPs submitted the LQAS reports to UNCDF.
2.4. Ensure FSPs are conducting the mini-surveys using LQAS to monitor progress on more meaningful indicators regarding usage of accounts, cross-sells, improvement in savings behaviors, client satisfaction and levels of poverty of clients, using Lot Quality Assurance sampling methodology YouthStart Programme Manager 2014/10 Completed The methodology of LQAS was in some cases replaced with the financial diaries and the final evaluators also helped to provide insights on the bechavior changes
3. Recommendation: Evaluation Recommendation 3: Deepen engagement with key regional stakeholders
Management Response:

While YS was not designed to be a programme focused on policy influence and change, management acknowledges the importance of having regulatory environments that are both youth-friendly (eg. no age barriers for opening savings accounts ) and protective of youth rights (eg. Ensure maximum control of youth of their savings account). To that end, YS has published 1 policy brief that was disseminated at 7 multi-stakeholder meetings at the country level in 2013, and adapted the smart client protection indicators to youth (partners received training on youth client protection in 2012). The 7 multisatkeholder meetings were conducted in Togo, Burkina Faso, Malawi, Uganda, Ethiopia, Rwanda, and DRC. The objective of the meeting was to disseminate at the country level key the YS publication Policy Opportunities and Constraints to Access Youth Financial Services. The policy brief summarized (i) the barriers that limit access and use of youth financial services, (ii) the opportunities to increase access of youth to financial services, and (iii) the recommendations that policy makers and regulators should consider in order to advance financial inclusion for youth. Participants at each meeting included: representatives from the ministry of finance or the central bank, ministry of youth, financial service providers, the microfinance association and UNDP. As a follow up to these activities, YS suggests to continue the dialogue with national stake holders during 2013 in a more targeted manner and with the support of the CTAs and RTAs.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 With the support of RTAs and CTAs put together a steering committee at the country level and at the regional level (for the countries of the West Africa Economic and Monetary Union) composed of representatives from the Ministry of Finance or the Central Bank, the Ministry of education, the microfinance association, the FSPs supported by YouthStart and UNDP to ensure national parties are informed about the progress of YS at the country level in particular about young women and girls, disseminate messages to the context familiar to the regional audience, and develop and implement a more targeted agenda on policy change. YPM, RTAs of 8 YS countries, CTAs or National Experts of Togo, Malawi and DRC 2014/12 Completed A new timeline needs to be developed: Done for Rwanda and Uganda Done for WAC region and Congo Done for Ethiopia and Malawi
3.2 Facilitate session at FIPA retreat or a webinar around the specificities of serving youth, tailoring messages to the context familiar to the regional audience, so that the UNCDF network of RTAs and CTAs can advocate for youth financial services in the countries and regions they work. YPM 2013/12 Completed Done at the East and Southern Africa Regional office in Ethiopia and done for the WAC office at the BCEAO workshop in Dakar
3.3. Conduct 2 steering committee meetings in each of the YS countries and 2 for the countries of the West Africa Economic and Monetary Union. YPM RTAs of 8 YS countries CTAs or National Experts of Togo, Malawi, and DRC. 2014/07 Completed Conducted workshop in Dakar with BCEAO countries. Conducted workshop with other countries in Dakar in June
3.4 Disseminate messages (about youth financial inclusion familiar to the regional audience and the role of financial and non-financial services in meeting the socio-economic needs of girls and young women) at least 2 regional for a per year. YPM, CTAs and RTAs 2014/12 Completed Participated at MicroLead annual workshop in Kigali and disseminate lessons learned of YS to FSPs in the region Participated at BCEAO financial inclusion workshop Conducted workshop on youth financial inclusion in DRC Conducted workshop on youth financial inclusion for AFR partners in Rwanda
4. Recommendation: Evaluation Recommendation 4: Improve coordination with and leverage of the regional UNCDF / UNDP network
Management Response:

Although roles and responsibilities for RTAs and CTAs vis-à-vis their support to YS are well defined in the Programme Document, management acknowledges that UNCDF needs to make an effort in terms of accountability. Coordination between thematic initiatives and country programmes is an important issue that was discussed during the FIPA retreat in 2012. Since then, some corrective measures have been put into place (eg re-focus of UNCDF CTAs? TORs regarding representation of and support to thematic initiatives in their countries, testing of the celoxis as a web-enabled platform for better coordination of work planning with regional / country staff). Wherever UNCDF develop new FI programs using the MAP framework, support to YouthStart and other thematic initiatives will also be embedded in that programming framework. etc). In short, UNCDF will make this improved coordination with and leverage of its regional and country network a top priority and will monitor it carefully. Please note that the recommendation provided regarding UNDP?s involvement is addressed with the measures taken to implement recommendation of issue 3.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1 Enforce accountability for RTAs and CTAs vis-à-vis thematic initiatives FIPA management team 2013/12 Completed Achieved by FIPA management.
4.2 Clarify the roles and responsibilities of CTAs, RTAs and GTI Programme Managers to all FIPA staff FIPA management team 2013/12 Completed Achieved by FIPA management.
4.3 Position UNCDF as the leader agency in the InterAgency Network for youth in tracking progress towards youth financial inclusion and working with UNCDF? RTAs and CTAs and UNDP?s country and regional offices to gain the attention of key macro and meso level actors in their countries , and in turn, use their power to convene and lead, thereby giving YouthStart an entry point to meaningful dialogue with key players. YPM 2014/06 Completed Participated in conferences in October to make sure financial inclusion for youth is included in the Post 2015 debate Co-organizing MSHM with CYFI to bring youth financial inclusion to the Post-2015 agenda Participated in annual meeting of the network Initiated process for UNCDF to take the lead of the youth employment working group in the West Africa region
4.4 Share knowledge with UNCDF? RTAs and CTAs and UNDP in the region and coordinate implementation of YS with other youth related projects YPM 2014/06 Completed Presented YouthStart at UNDP?s regional workshop of the Youth Employment Network (joint programme of the with ILO, UNDP and the World bank)in Dakar. In partnership with other UN Agencies, contributed to the development of a Youth employment strategy Senegal Presented YS at SEA office Presented YouthStart to partners of UNCDF country programme in DRC
5. Recommendation: Evaluation Recommendation 5: Improve the efficiency and effectiveness of program management
Management Response:

Management agrees with the recommendation of upgrading the YS consultant position to a P3 and provided funded is granted it will be taken into consideration. Regarding the knowledge manager, management also agrees with that recommendation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 Explore with the MCF the possibility of getting additional funds or agreement to reallocate current funding to hire another full time staff YPM 2013/10 Completed Achieved in September 2013. Full time staff to start in March 2014.
5.2 Provided funding is granted, YS will hire the P3 FTA YPM 2013/11 Completed Full time staff to started in March 2014
5.3. FIPA to explore different options for KM of the thematic initiatives that ensure an economy of scale with the other thematic programmes. YPM-FIPA 2013/11 Completed Achieved in September. New knowledge manager started working in October 2013.
6. Recommendation: Evaluation Recommendation 6: Considering a broader set of strategic drivers of successful YFS
Management Response:

Management agrees with this recommendation. In addition to continue integrating within its monitoring visits indicators (eg quality of pilot test implementation, client protection assessment etc) that go beyond making sure FSPs are reaching PBA targets, YouthStart will seek to improve its risk management frameworks. Please note that the Concept note for YouthStart Global already takes into consideration all the recommendations provided in terms of rebalancing for depth, as opposed to breadth., in particular in those countries where YS was active during the pilot in sub-Saharan Africa.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.1. Identify and select key strategic drivers of successful YFS that go beyond making sure FSPs are reaching PBA targets YPM 2013/11 Completed Achieved for monitoring visits of 5th and 6th payments of the grant (September and October 2013. Set of indicators were included into monitoring visit reports
6.2 Monitor strategic drivers of success of the second phase of YFS roll out rather than only PBA targets YPM 2013/11 Completed Achieved for monitoring visits of 5th and 6th payments of the grant (September and October 2013)
6.3 Develop UNCDF due diligence templates and processes to use before granting funding to FSPs. YPM 2013/09 Completed Achieved in September 2013. The template was tested in Q2 2013 and refined in Q3
6.4 Adapt UNCDF due diligence templates to select FSPs to provide services to youth YouthStart-FIPA 2013/11 Completed Achieved in September 2013
6.5 Monitor strategic drivers of success of the third phase of YFS roll out rather than only PBA targets. YPM 2014/01 Completed
6.6 Monitor strategic drivers of success of the fourth and final phase of YFS roll out rather than only PBA targets YPM 2014/05 Completed Done at monitoring visits. See monitoring visit reports in the intranet for reference
7. Recommendation: Evaluation Recommendation 7: Focus TA for FSPs on consolidating business models
Management Response:

Management agrees with this recommendation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
7.1 Explore with the MCF the possibility of getting additional funds and/or agreement to reallocate current funding to boost increase the amount of technical assistance provided capacity of to FSPs by Reach Global. YPM 2013/09 Completed Achieved in September 2013
7.2 Provided resources are available, YS will extend the contract to Reach Global to 200 days per year YPM 2013/09 Completed Achieved in September 2013. The contract of Reach Global was extended until July 2014

Latest Evaluations

Contact us

220 East 42nd Street
20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org