Success message
error message
warn message
Mid-term Evaluation of the Shaping Inclusive Finance Transformations Programme SAARC Region
Commissioning Unit: UNCDF
Evaluation Plan: 2018-2021
Evaluation Type: Others
Completion Date: 08/2019
Unit Responsible for providing Management Response: UNCDF
Documents Related to overall Management Response:
 
1. Recommendation:
Management Response: [Added: 2020/01/30]

Overall comments: Management would like to thank the evaluation team for the evidence, analysis, and insights presented in the Mid Term Evaluation Report. SHIFT in SAARC found the recommendations provided useful to further build on the achievements of the programme to date and to strengthen programme implementation moving forward.

One of the goals of the SHIFT SAARC program is to stimulate investment, business innovations, and regulatory reform to bring about lasting changes for low-income people. Specifically, it aims to expand economic participation and opportunities for women and small and growing businesses to be active agents in the formal economy. The evaluation acknowledges that the SHIFT SAARC has delivered several activities and outputs over the past two years including impactful research, six innovation grants, capacity development of policy makers’ and regulators,’ partners and entrepreneurs and built a coalition of public and private stakeholders in the DFS ecosystem. The evaluation has suggested that the program had some shortcomings in communication with donors and stakeholders. SHIFT SAARC management has promptly taken action and reinforced the communication at all levels and improved overall communication.

SHIFT in SAARC notes the assessment of the evaluation that the program design is relevant to the context of the pilot country in SAARC namely Bangladesh and that SHIFT SAARC’s policy efforts have been able to influence the policymakers and regulators to some extent.

On the program’s effectiveness, the evaluation has mentioned that the evidence is inconclusive in areas like the extent to which SHIFT in SAARC’s efforts contribute to promoting enabling policy and regulations with Bangladesh Bank,  the extent to which the programme has built awareness of the opportunities and constraints for gender equality or awareness of the women’s market. SHIFT SAARC program team is aware of the disagreement among stakeholders as in Bangladesh policymakers and regulators seldom acknowledge the influence of outsiders in the policy and regulatory space. The program team noticed that the evaluation acknowledges that both public and private sector stakeholders have acknowledged the role of SHIFT SAARC in bringing the DFS issue at the forefront of discussion in Bangladesh. SHIFT SAARC team recognizes the remaining challenge and will continue to work during the rest of the program period to minimize the gap.

Overall, UNCDF management accepts the recommendations of the evaluation, with some caveats as indicated in the detailed response, and will take necessary measures to strengthen the SHIFT SAARC programme.  The evaluation’s recommendations will also be applied as UNCDF designs the next phase of its programming in the SAARC under its new integrated “Leaving no one behind in the digital era” strategy. Other evaluation exercises scheduled for SHIFT ASEAN and the PFIP program in the Pacific will be used similarly to design successor programming in those regions under the new global “Leaving no one behind in the digital era” strategy. This more integrated programmatic approach will enable UNCDF to develop a more integrated and strategic approach to financial inclusion and digital finance across Asia, with intense cross-learning and shared experiences between the three sub-regions.

Key Actions:

2. Recommendation:

Recommendation from the Evaluation or Issue-1: Innovate Faster

Start innovation actions faster. It is not possible to have an impact or achieve sustainability without the adoption and scale-up of successful innovation pilots. The pilots need to be done in a timely way to give the market evidence that will encourage them to invest. Along these lines, the expected demonstration effects of the pilots should be explicit in the Theory of Change (TOC) and the measurement and results management (MRM) system. Related to this, the Digital Finance Consultative Group (DFCG) needs to become more action-oriented. The DFCG is currently effective at achieving its main objective “to open up dialogue on the strengths, opportunities, challenges, gaps in the use of DFS for a more inclusive economy and sustainable economic growth in Bangladesh.” As a neutral organization, SHIFT in SAARC is well positioned to introduce a more purpose-oriented, participatory, action-oriented working group that can take on specific topics to promote real change in the sector. The programme could co-facilitate this group as the secretariat with a senior level leader from the Central Bank (i.e., a committed champion who will devote time), regular meetings, designated members, and specific topical sub-groups that engage on priority issues selected by the group. UNCDF has experience coordinating DFS working groups in other regions (e.g., West Africa) that have action-oriented agendas from which they could borrow.

Management Response: [Added: 2020/01/30] [Last Updated: 2020/01/30]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1 Fast-track implementation of the innovation grants: SHIFT SAARC has already operationalized 6 innovation pilots, and they are being implemented. The donor has agreed to the proposal of SHIFT SAARC to extend the project for 8 more months recently till Sep 2020. The extension is mainly aimed at capturing the learning from implementation and business innovation pilots and sharing the evidence with relevant stakeholders.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/12 Completed
1.2. Periodically update the Theory of Change and MRM to include demonstration effects and learnings from pilots.
[Added: 2020/01/30] [Last Updated: 2020/03/31]
SHIFT SAARC Programme Manager 2020/07 Initiated 1.2. Periodically update the Theory of Change and MRM to include demonstration effects and learnings from pilots. Every Quarter SHIFT SAARC Programme Manager 30 June 2020 In Progress We have made some significant progress in this regard during Q1 2020. MRM dashboards for each of the 6 Business Innovation Pilots are in place. These dashboards are tools to reflect the demonstration effects and learnings from the pilots so that the project managers can make course corrections during implementation. The MRM for each pilot now includes data sources ranging from Periodic Reports and Milestone Reports to MIS, mobile app, and website data. However, the core ToC and MRM framework of SHIFT SAARC has been deliberately kept unchanged. Revision of a project’s core ToC and MRM framework is a critically sensitive task, and the exercise of developing 6 separate MRM dashboards for innovation pilots is an important step towards it. Depending on outcomes of the use of separate MRM dashboards for innovation pilots in Q1 2020, the core ToC and MRM framework of SHIFT SAARC will be updated in the next quarter. Accordingly, the project team will keep referring to the dashboards, ToC, and the MRM framework for evidence-based programming and update the same in the following quarters, if necessary. History
1.3 SHIFT SAARC will attempt to transform DFCG into an action-oriented platform together with Gov through the UNDP a2i initiative, re-activate members participation and engagement with policymakers and regulators.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/09 Initiated
3. Recommendation:

Recommendation from the Evaluation or Issue-2: Create a more robust capacity building system 

SHIFT in SAARC is well positioned to play a lead capacity development role at the sector level. Though several training workshops have been conducted with various stakeholder groups, the evaluation team received feedback that the training was not sufficient for those trained and was not extended to other relevant segments. SHIFT in SAARC documentation also did not demonstrate a systematic and robust approach to measuring the results of the capacity building and training output. There remain many additional opportunities to increase DFS usage, but capability remains a critical barrier. Reinforced capacity development for key stakeholders, especially on gender issues, is relevant and could support long-term behaviour change while creating institutional change management with a range of FSPs

Management Response: [Added: 2020/01/30]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1 Capacity Development is a long term effort, and it requires prior planning and budget. SHIFT SAARC arranged different kinds of interventions, including the development of a DFS module that can be tailored to the needs of the regulators and handed it over to the training academy. Further efforts will be made to convince the Central Bank to organize regular courses based on that module. Key action: Support the Central Bank to organize regular courses based on the DFS module developed by SHIFT.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/09 Initiated
2.2 Key Action: The programme will also work towards raising resources or collaborate with institutions having resources on stakeholder capacity building to draw up a comprehensive plan for capacity building of regulators and critical providers as well, especially addressing how decision-makers consult data and evidence and how gender issues and needs of the left-behind segments are viewed on a policy level. Capacity Development through data insights on micro-merchants, its customers, and policy aspects will be undertaken once the innovation pilots are completed.
[Added: 2020/01/30]
SHIFT SAARC Programme Manager 2020/09 Initiated
4. Recommendation:

Recommendation from the Evaluation or Issue-3: Adapt programme management to time and resources 

Given SHIFT in SAARC’s focus on changes in market actor behaviour leading to economic opportunity for women and small businesses, realistic expectations for market systems change should be better aligned with project and funding length. The market system development approach requires time to develop the system and achieve changes. To achieve impact by the end of SHIFT, the evaluation team recommends a deep dive on key sectors where energies and investments are already focused rather than spreading resources across new topics and activities. While adapting to the changing context is essential to remain relevant, reacting without considering how a new activity fits within the strategy diminishes focus on the impact of key activities. Considering the time remaining, the evaluation team recommends not expanding to new segments, sectors or initiatives; instead, it would be useful to consolidate resources to go “deeper” on fewer activities. This approach would include focusing on key sectors already researched and creating better tailored knowledge management resources, distribution channels, and learnings highlighted from the activities already carried out. The programme should focus on quality over quantity. For the innovation grants, focusing on quality would mean focusing on a couple of well-designed projects with robust PBAs that could deliver some quick wins for evidence. It is inadvisable to start market research on new topics or organize large stakeholder roundtables/conferences to present findings.

Management Response: [Added: 2020/01/30]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
In support of the recommendation to provide ‘deep dive support’ on crucial sectors rather than spreading across new topics and activities, the following key actions are proposed: 3.1 i) Use insights from the micro-merchant assessment for more tailored innovation pilots and for fine-tuning innovation models. ii) Commission additional research on digital transformation of supply chain operations and payments to better understand value chain actors and inform potential investors and market actors.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/06 Initiated History
3.2 Realign focus on a few well-designed innovation pilots that could deliver quick wins for evidence. In support of this, SHIFT SAARC will closely monitor the pilots to ensure adherence to the principles of lean innovation and customer-centricity and for overall improvements.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/06 Initiated History
3.3 The programme will periodically document the knowledge management lessons of the innovations being piloted, thereby capturing learnings for wider market and stakeholder benefits
[Added: 2020/01/30]
SHIFT SAARC Programme Manager 2020/09 Initiated
3.4 Through training, immersive workshops and Partner Review Meetings with the innovation partners, SHIFT SAARC has been making sure that they have a proper understanding of gender issues, challenges and needs of these overlooked segments and the techniques of capturing and building on implementation lessons. This will be further intensified during the remaining period of the project period.
[Added: 2020/01/30]
SHIFT SAARC Program Manager 2020/09 Initiated
5. Recommendation:

Recommendation from the Evaluation or Issue-4: Review and Revise the MRM System   

The MRM system should be better aligned with the CGAP framework for measuring market development in inclusive finance to improve accountability. This involves identifying and measuring systemic change, as well as changes in the inclusive financial and real economy market systems, and distinguishing these changes from intermediate outcomes over which the programme has more control. A careful review of existing results framework indicators and targets is needed to ensure that indicators are in the right category. These changes would produce evidence that staff can learn from to better manage the programme.

The programme could also improve change monitoring in market systems to better support necessary course corrections by introducing new indicators that address the specific constraints SHIFT in SAARC is trying to alleviate. For example, most micro-merchants have unregistered businesses which prevents them from opening merchant DFS accounts with higher daily transaction amount limits that would allow them to do more business with DFS. As the programme is encouraging merchants to register their businesses, the number of merchants that register their businesses and take up merchant accounts would be a strong indicator of programme effectiveness and changing practices of a market actor.

The timing of data collection could be improved. The co-applicant’s data is reported to SHIFT in SAARC when they reach a milestone and need to apply for the next tranche of funding. It would benefit the monitoring function if this data were provided by co-applicants on a regular basis, say every three to six months, so that the SHIFT in SAARC team could obtain a snapshot of the whole programme at regular intervals and use the data to make management decisions.

The programme does not appear to have a plan to conduct follow-up monitoring visits three to six months after the MDDRM business trainings are finished, such as with micro-merchant or women entrepreneur trainees. One co-applicant recommended that SHIFT in SAARC provide resources to allow for ongoing monitoring of changes at the micro-merchant level, “so we can know the impact of the activities we have implemented and take corrective measures.”

Finally, with respect to the TOC and the structure of outputs, communications activities should be pulled out and treated as a cross-cutting theme like gender equality. This would ensure they receive the required attention and would increase TOC logic.

Management Response: [Added: 2020/01/30]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1. The MRM system will be further developed and aligned with CGAP framework on measuring market system development for the poor.
[Added: 2020/01/30] [Last Updated: 2020/03/31]
PM 2020/07 Initiated 4.1. The MRM system will be further developed and aligned with CGAP framework on measuring market system development for the poor. March 2020 SHIFT SAARC Programme Manager 30 June 2020 In Progress The MRM system of SHIFT SAARC has been thoroughly reviewed, and some areas were identified where there could have been better alignment with the CGAP framework for measuring market development in inclusive finance to improve accountability. Notably, the current MRM framework tends to focus more on aspects within the programme’s control as outcome indicators. The team has identified indicators that can more objectively measure systemic changes and changes in the inclusive financial and real economy market systems as a result of the programme’s activities. Based on further review and approval of this identification and in alignment with the outcomes of the dashboard exercise of innovation pilots, the MRM system will be updated in the next quarter. History
4.2 SHIFT SAARC will further strengthen its existing MIS which presently tracks both programme and partner level activities. Data from digital media platforms will be collected in near real-time basis.
[Added: 2020/01/30]
PM 2020/09 Initiated
4.3: SHIFT SAARC will step up follow-up visits to understand how and why training did or did not make any difference to the target segments.
[Added: 2020/01/30]
PM 2020/09 Initiated
4.4 SHIFT SAARC will further enhance its ongoing communications where findings from research initiatives and anecdotal impact stories is transformed into targeted messaging for specific stakeholder groups and broadcasted via newsletters, info sheets and social media channels. Further strengthening of cross-sharing of updates and insights among partner networks will be undertaken
[Added: 2020/01/30]
PM 2020/09 Initiated
6. Recommendation:

Recommendation from the Evaluation or Issue-5: Incorporate gender issues as a key strategic focus

UNCDF FIPA should require gender analysis for all proposed programmes. This analysis should involve secondary and primary research with the target market segments during the design phase to ensure that the data is current. In the words of one key informant “first, we need to know the problem" regarding women as entrepreneurs.

Within the country or regional office, SHIFT in SAARC or UNCDF should recruit a technically qualified gender champion at the senior management level in order to advance the women’s economic empowerment agenda. The champion should build the capacity of the programme team and focus on the innovation grants and DFCG to raise the awareness of women as a critical market segment. Specifically, the champion should:

  • Build the national team’s awareness of gender equality and women’s empowerment issues so they can ask the right questions and better engage with donors and partners on this issue. Headquarters should take the lead on this to ensure a common understanding of gender equality throughout the agency;
  • Ensure that the team understands the POWER findings, a study to which they contributed, and determine how they may implement the programme based on the study’s findings;
  • Work with innovation grant recipients to ensure female customers/entrepreneurs are included;
  • Create a DFCG working group on gender issues; and
  • Ensure that gender is incorporated into all programmes, activities and institutional structures.
Management Response: [Added: 2020/01/30]

Agreed

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1 The programme will specifically identify issues related to female customers/entrepreneurs onboarding and benefits accrued under its innovation grant recipients project. Based on the issues identified, technical assistance will be provided to maximise on gender outcomes under the innovation pilots.
[Added: 2020/01/30]
PM 2020/09 Initiated
5.2 SHIFT SAARC programme will support a workshop with its partners and national stakeholders on incorporating gender issues as a key strategic focus while working on digital financial services.
[Added: 2020/01/30]
PM 2020/09 Not Initiated

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org