Mid-Term Evaluation of Mobile Money for the Poor (MM4P)

Report Cover Image
Evaluation Plan:
2016-2017, UNCDF
Evaluation Type:
Others
Planned End Date:
09/2017
Completion Date:
12/2018
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
168,000

Share

Document Type Language Size Status Downloads
Download document MM4P ToR .docx tor English 473.75 KB Posted 445
Download document Genesis_UNCDF_MTE of MM4P_Final report - 29.01.2019.pdf report English 2441.77 KB Posted 1330
Title Mid-Term Evaluation of Mobile Money for the Poor (MM4P)
Atlas Project Number:
Evaluation Plan: 2016-2017, UNCDF
Evaluation Type: Others
Status: Completed
Completion Date: 12/2018
Planned End Date: 09/2017
Management Response: Yes
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
Evaluation Budget(US $): 168,000
Source of Funding:
Evaluation Expenditure(US $): 178,146
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
Richard Ketley Team Leader
Mauro Mela Digital Financial Services specialist
Caitlin Smit Senior Evaluator
Comfort Phelane Digital Financial Services specialist
Mark Schoeman Digital Financial Services specialist
Ashleigh Flynn Evaluator
Thandiwe Ng’ombe Evaluator
Dikshya Dawadi Local Consultant (Nepal)
Suman Gelal DFS Expert: Nepal
Agnes Fall DFS Expert: Senegal SENEGAL
Rasmi Pillar DFS Expert: Uganda INDIA
GEF Evaluation: No
Key Stakeholders:
Countries: GLOBAL
Lessons
Findings
1.

5. EVALUATION FINDINGS

5.1. RELEVANCE

MM4P plays a unique role in the markets where it operates by bringing strong technical knowledge and DFS experience locally, accentuated with international experience; in its responsiveness and level of engagement with partners; and in its role as convenor and honest broker within the ecosystem. While DFS has been demonstrated to have a development impact in LDCs, there are a number of constraints – regulatory, digital infrastructure bottlenecks, and on the demand and supply side – that may limit a market’s potential to realise these gains. MM4P’s ecosystem approach is designed to address these binding constraints. However, the ability to adequately address all of these binding constraints in each country is constrained by funding limitations and in some instances the mandates of the funding donor. These limitations aside, the programme’s design, partner selection and phasing of engagement have been appropriate for the LDC market contexts where the programme has operated. The TOC has played out as expected with some potential for changes going forward. The use of TA in combination with small grants has been relevant to the needs of both partners and the programme. While external consultants have been effective, in-country technical teams have been critical to the success of TA by managing external relationships or delivering the TA directly. The programme has collaborated well with other UNCDF programmes, with the broader UN system, and with other development stakeholders. Despite a few challenges with some partnerships and room for improvement in cross-programme collaboration, these partnerships have been built on synergies between development stakeholders and have removed duplication of effort when supporting the same partner organisations. In addition, MM4P has provided critical learning and a number of institutional assets which will assist UNCDF in the execution of its new strategic vision of digital innovation to support the sustainable development goals.


2.

5.2. EFFICIENCY

MM4P’s TA support (both from its staff and consultants) is considered to be one of the most valuable activities by the programme. Partners particularly valued the MM4P team’s level of expertise, professionalism, and availability on-the-ground to help guide the partners throughout the implementation of a project and also keep implementation on track. The DFS Working Group has been an important contributor to DFS development and established MM4P as a go-to DFS expert in most of its markets. Implementation delays at inception, and short project durations could negatively affect MM4P’s results in the future. MM4P’s internal processes were cited as being cumbersome for partners, but MM4P also faced a challenge with having to spend more time than anticipated to negotiate with partners and thirdparty partners before implementation. Projects were also a year-long on average and were considered too short to achieve the anticipated results. In Uganda, this was also exacerbated by the mandate from BMGF to identify and conclude up to ten value chain projects within 16 months—which was ultimately extended through negotiations for another year, six months before the original agreement expiration date. A few donors were unaware of the total amount of programme overhead, which amounts to 23% of overall MM4P funding, composed of the standard 8% UNCDF overhead, plus the cost of the Brussels hub structure. This indicates that UNCDF needs to better communicate its overhead costs to funders at the outset. A more detailed analysis of the direct versus indirect charges related to Brussels and the countries may also help clarify for funders the relative cost effectiveness of the programme and help identify if any efficiencies can be gained. The results measurement function of MM4P has achieved a number of successes thus far. MM4P has done well at measuring changes within partner institutions and tracking market development shifts, however, not all outcomes of MM4P’s ecosystem activities are systematically captured in the RM framework. Staff find it easier to report on set indicators and outcomes of TA and grant work in comparison to the informal DFS advice offered to different ecosystem players (which is where many important market system contributions are being made).


3.

5.3. EFFECTIVENESS

The TA support provided by MM4P to its private-sector partners, including MNOs, financial institutions and FinTechs has successfully contributed to building all of their capacity to develop and/or improve DFS. Partners attribute changes in organisational processes, strategy and increased managerial buy-in “It would be natural that we would come across MM4P reports when looking at a particular issue. But I think that as an industry we are producing too much. I have every interest to look at these reports, but just don’t have the time”. - External “Challenges are part of the programme’s success because you cannot understand how the market operates without experiencing these challenges”. - MM4P “When I looked at it and shared it with colleagues I felt that it was too much selfpromotion… it’s not constructive for knowledge exchange”. - External 26 and investment in DFS to the TA provided by the MM4P team and/or the external consultants. Although the grants were also beneficial in conducting research and piloting digital solutions, partners reported that the TA was more valuable in effecting how organisations approach DFS. In a few instances, the MM4P programme has also contributed to positive changes in regulatory environments through its relationships with regulators. The provision of formal and informal TA, benchmark visits and supporting improvements in internal practices has resulted in an increase in regulators’ commitment to building a supportive environment for providers to offer digital financial solutions. The key success drivers of the programme include the local presence of the MM4P programme; the MM4P team’s DFS expertise and experience working with private sector organizations; and the flexibility in programming approach to suit the respective markets of intervention.


4.

Success drivers at the partner level

Building good relationships is a key success driver of MM4P’s partner engagements. Across all four countries, three themes were noted as contributing factors to the success of the program: local presence, in-house DFS expertise, professionalism and commitment of the MM4P team and flexible approach to programming.

  • Local presence of MM4P team: This has enabled the team to develop close relationships with partners and establish its presence in the market. Partners, consultants and external stakeholders value MM4P’s in-country presence as it eases the provision of strategic advice and discussions on project ideas, both formally and informally. MM4P’s successful establishment in-country as a trusted broker and go-to market player for DFS has improved partners’ willingness to share organisational data “It’s great that the MM4P team can easily pop into the office and give insights on a particular idea.” – Partner 28 and made it easier for partners to request assistance. From a programme perspective, having a local presence has enabled the MM4P teams to assess potential partnerships and has been valuable in allowing MM4P to assess the impact and sustainability of the partners. Additionally, it provides the team with opportunities for MM4P to monitor implementation as well as find more partner opportunities.
  • MM4P team’s DFS expertise, professionalism and reliability: The MM4P team is highly valued for their technical skills and experience in the DFS sector. In Zambia, partners reported that the MM4P team plays a strategic role in their everyday thinking through the open channels of communication that allow for feedback and discussions. Partners also valued the MM4P team’s professionalism, openness, willingness to help and reliability. Partners also shared that the MM4P team has a rigorous approach to project management that is beneficial in ensuring that both parties are aware of project deliverables, developments and any potential challenges as implementation is carried out. As a result, partners consider MM4P as their strategic, thinking partner and value both the formal and informal support provided. Partners also reported that the TA support offered through external consultants was found to be highly valuable. Prior to UNCDF’s intervention, most partners had not engaged with consultancy services due to financial constraints and the availability of local consultants in the market. Partners were impressed with the engagements with the consultants and quality of outputs produced. The MM4P team played a key role in ensuring the success of the TA provided by consultants. Both partners and the consultants interviewed shared that the projects were framed as a tripartite agreement between the partner, consultant and the MM4P team which allowed for buyin and a shared understanding of the objectives of the project across all stakeholders. The MM4P team enhanced the success of the projects by establishing and managing the relationship between the consultant and the partner which allowed partners to see the consultants as a part of their team and not externals. Additionally, the MM4P team’s provision of project oversight allowed any projects challenges that arose to be dealt with in a timely manner.
  • Flexibility: Partners and external stakeholders believe that MM4P is good at assessing the market and adapting their implementation strategy to accommodate the different market needs and dynamics, for example the partner selection process. MM4P Senegal implemented the request for applications (RFA) approach to uncover innovations and market actors they could partner with (which they did not feel were emerging from the short-term TA and concept note process employed by the other country teams). In Nepal, the programme shifted its partner focus to working with different financial institutions other than banks where there was high donor dependency and, as a result, risks of non-unsustainable solutions.

5.

Challenges

The implementation of projects experienced delays due to a number of reasons, such as:

  • Contextual challenges that interfere with the time allocated for project implementation: For example, partners in Uganda stated that given the contextual challenges that inhibit the DFS market, it was difficult to see any impact of the projects or report on set key performance indicators due to delays in implementation or challenges in rolling out their solutions. For example, the Heifer project on digitising farmer payments in the livestock value chain was a year-long project that required more time as the mobilisation of booster teams and sensitisation of farmers took 6 months. While the project was able to show farmers the value of DFS, customer uptake was low. The Mukwano project also experienced delays in implementation due to a longer than anticipated time to finalise agreements and the challenging nature of the agricultural sector. This resulted in partners feeling rushed to complete the project within the set timelines. Other examples of projects in Uganda that experienced this challenge include Laboremus and DanChurchAid. This challenge is particularly difficult where technical assistance is contracted through consulting firms, who are not always as flexible in accommodating delays given their structure and workflows.
  • High staff turnover in large organisations, such as financial institutions and MNOs: This challenge was mostly prevalent in Senegal and Zambia. According to KIIs, key staff in these institutions rotate around the market, which affects continuity in projects. To overcome this challenge in Senegal, MM4P established a steering committee with various stakeholders to build and maintain relationships across the organisation, and preventing the loss of institutional capacity when staff leave.
  • Time taken for partners to establish third-party agreements: Under the Mukwano project in Uganda, there was a lag in the time it took for Yo! Uganda to sign agreements with MNOs, which led to a delay in project implementation. In Senegal, the Orange youth kiosk project was delayed due to prolonged negotiations between BASIF and PAMECAS.
  • Lack of capacity to implement projects. Where partners did not have the appropriate resources, the implementation of projects was challenging. For example, an MNO in Zambia had a small team who were required to provide oversight and implementation of the TA project implemented by a partner. Both the partner and consultant reported that staff struggled to balance their set daily tasks with what was considered to be an “extra” work under the MM4P project. A number of projects in Uganda also required more time than anticipated, as partners did not have the necessary staff and systems in place to implement the projects.
  • Varying commitment of partner senior management to implement project recommendations. While partners found the support provided by MM4P valuable and well-tailored to meet organisational needs, there were instances of a lack of commitment and buy-in from senior management to implement recommendations and make additional investments (which is exacerbated in instances of high staff turnover). This in turn poses a threat to the success and ultimate sustainability of the projects. For example, MM4P had to stop providing TA to Living Goods in Uganda due to the lack of commitment from management. As mentioned above, MM4P Nepal’s programming was also affected by the lack of commitment of banks to invest in the scaling up of their agency banking solutions due to a heavy reliance on donor funding.
  • Competing organisational priorities and changes in strategy. For example, the TA provided to NCell in Nepal was negatively affected by the sale of the company. This led to delays in the piloting of the digital solution and in turn the cancellation of the partnership between MM4P and NCell.
  • Partner readiness to commit to projects. For example, the partnership between BASIF and PAMECAS experienced delays in the pilot of the Orange youth kiosk project due to BASIF not having the necessary DFS expertise required to manage an agent network.
  • Low customer awareness and adoption of DFS. The main challenge that partners face in offering DFS is low consumer awareness and trust of DFS, especially in rural areas. Partners, MM4P and consultants reported the need to invest resources to sensitise customers on the use and value of DFS. Through the HCD research conducted in Zambia, and Prabhu’s use of word-of mouth marketing and training of agents, there has been an improvement in the uptake of digital services. Overall, ecosystem actors reported that there has not been sufficient investment in customer awareness, financial and digital literacy by partners and regulators, which continues to threaten the uptake of DFS. MM4P’s role in this regard should draw on its existing strengths – convening stakeholders and providing data and information on the importance of building the financial and digital capacity of consumers – as well as supporting distinct initiatives within partners through TA.

6.

5.4. LIKELY IMPACT

Partners: MM4P’s direct financial and technical support to partners, facilitated by strong in-country technical teams and partnerships with good consultants, has contributed to greater interest among partners in DFS, improvements in partners’ capacity, the formation and improved implementation of partnerships, and specific instances of increased investment in DFS. However, commercial sustainability remains a challenge for many MM4P-supported projects. “When people ask me questions about DFS, I call MM4P.” – Regulator 32 DFS systems: The DFS systems in all MM4P countries have developed and availability and usage of DFS products has increased. However, it is insufficient to use national statistics for active DFS accounts as a headline measure of success of the programme (and the programme’s contribution to market development). Rather, these indicators are important as diagnostic tools for understanding the market context and how the DFS market is shifting in each country. This evaluation did not attempt to quantify MM4P’s contribution to national DFS usage numbers, but instead focused on the programme’s theory of change and building a qualitative narrative of contribution. Evidence from this evaluation shows that some positive linkages can be made between MM4P’s activities and some sector and customer outcomes, as well as the programme’s ultimate goal. MM4P’s activities have increased the level of awareness, capacity and commitment to DFS within the sector. At the DFS stakeholder outcome level, DFS offerings have been made more accessible and available (rural outreach is still proving to be a challenge) with MM4P support, but continue to struggle with affordability (largely due to high MNO charges), reliability (due to poor MNO coverage), and customer understanding (prevalence of low customer trust, awareness, literacy and financial capability). These challenges have had a negative impact on the commercial viability of a number of MM4P’s partnerships. As a result, there is little evidence that increasing access to DFS is occurring within non-MM4P partners as a result of demonstration effects (sector outcome level). MM4P’s contribution to the DFS markets in each country is a result of both the programme’s direct support to partners and MM4P’s ecosystem activities, specifically industry data and information, convening and facilitation, and formal and informal support and capacity building provided to regulators. Clients: Considering MM4P’s theory of change, and achievements at the DFS stakeholder outcome level, it is likely that MM4P will contribute (and has already contributed in specific instances) to increasing access to DFS for underserved clients. However, at this stage, sustained uptake and usage of DFS remains a challenge in many MM4P-supported projects. UNCDF: MM4P has had a significant impact on how UNCDF is approaching future programming and is considered critical to the agency’s new strategy, which is centered on leveraging digital solutions to support the achievement of the Sustainable Development Goals.


7.

5.5. SUSTAINABILITY

The sustainability of the projects MM4P has supported is mixed across the portfolio. Provider projects tend to have commercial viability built in as a consideration due to the operating nature of for-profit providers, but there is not yet much evidence that provider projects are commercially sustainable. In comparison, regulator and policymaker projects tend to have a slower pace of change, requiring longerterm support and engagement. At the client level, the programme is still in the early stages to assess client outcomes, but a significant challenge thus far has been the client preferences for agent-assisted transactions rather than conducting transactions using a mobile phone, although this is beginning to change. Overall, the type of support and nature of engagement provided by MM4P is better for the continuation of partner outcomes compared to other types of donor support, such as large grant-making that subsidises providers’ costs of operation. At the ecosystem level, there is an ongoing need for a market facilitator to catalyse DFS development to the tipping point at which market development takes off organically. This suggests a strong potential role for MM4P to play going forward, particularly within the regulatory and policymaker space and playing a strong convening role within the ecosystem. Where funding and programming will end, this means MM4P requires a plan to continue the programme’s convening role in its absence. However, the programme’s early ecosystem work has already generated outcomes which are contributing to the momentum of DFS market development


Recommendations
1
2

Evaluation Recommendation or Issue 1:

 

The programme TOC and country results chains should be updated to reflect what has been learnt in each market. MM4P has been operating in each market for some time and has generated important learning about the market context, the key barriers to DF market development, and programming opportunities that have arisen in each country. Since the programme’s TOC and country results chains were developed at inception when programming activities had not been rolled out, there is now an opportunity for these to be updated. These should also explicitly state assumptions, risks and possible external factors for each link in the ToC, to assist with future assessments of MM4P’s contribution to market development shifts. The learning generated from this mid-term review should provide some input to this process, such as the role of innovation as a potential workstream and the need to incorporate KM and ecosystem work as causal links in the chain, as well as better articulating gender outcomes at the customer outcome level. However, each country team should consider how the state of the market has evolved since inception, and where results chains need to be updated and adapted as a result.

3

Evaluation Recommendation or Issue 2:

 

MM4P will have to incorporate innovation into its work stream as DFS markets develop and new types of providers emerge. Innovation here relates to the development of new business models for financial services providers, such as fintechs, and other service providers where digital payments form part of the new business models. Examples of innovation already emerging from the programme include fintechs (agile tech-focused financial service providers that use technology to change or improve the way financial services are produced and consumed) and digital platforms and marketplaces in the sharing economy that make use of digital payment options (such as Tootle in Nepal). Once digital payment rails are in place and the focus shifts to developing use cases around these rails, supporting innovation becomes an important activity for DFS market development. MM4P is already thinking about how to incorporate this into its programme design from the outset, including how to encourage and engage with non-traditional financial service providers, such as fintechs, which should be continued and formally built into the programme design.

4

Evaluation Recommendation or Issue 3:

 

MM4P should articulate the impact of its programming beyond financial inclusion to account for shifting funding priorities. The funding of financial inclusion activities within the donor community has shifted from considerations of financial inclusion as the end objective, to a broader focus on financial inclusion as a means to other objectives. These include improvements in the livelihoods of specific customer segments – such as women, refugees, and the youth – and the support of particular sectors – such as energy, food security, and agriculture. MM4P should monitor these shifting funding priorities among key donors and articulate the impact its work has on these priority segments/sectors in order to attract funding going forward

5

Evaluation Recommendation or Issue 4:

Given funding restrictions and donor priorities that have resulted in limitations on regulatory/policy and infrastructure work in some countries, MM4P needs to have plans in place to work around these limitations. The table below (see final report for the table as it was too long to insert here) provides a few examples from the programme of where policy/regulatory impact was created despite the limitations, in countries where these limitations existed. These examples should be replicated across the programme where possible. Importantly, these plans need to be in place at the outset of programming and built in to the programme design. MM4P can also draw on its existing strengths at ecosystem-level, namely information provision and convening to focus stakeholders’ attention on these limitations, even where MM4P does not have funding or resources to directly address these limitations.

6

Evaluation Recommendation or Issue 5:

MM4P needs to develop careful sustainability plans for its ecosystem facilitation role and start implementing this plan now to provide enough time before programming winds down. MM4P is playing a critical convening function within the DFS ecosystem, for example with the formation of DFS working groups. However, MM4P is often the only stakeholder playing this convening role and so needs to develop mechanisms for stakeholders to continue collaborating once the programme ends. Taking a lesson from MM4P in Malawi, the formation of an industry association with funding contributed by members appears to be a sustainable approach to having this convening function continued. MM4P should look to set up similar industry associations in countries with DFS working groups, and in Nepal where a DFS working group has yet to be formed.

7

Evaluation Recommendation or Issue 6:

 

MM4P can better incorporate the mainstreaming of gender into its programming activities over the second half of the programme. To date, requiring providers to collect sex-disaggregated data has been a tick-box exercise rather than encouraging providers to make use of the data. Programming should therefore focus on building the business case for using sex-disaggregated data among providers as part of MM4P support. This could include capacity building/TA or research such as market sizing and opportunity analysis for female customers, encouraging providers to make use of their data to tap in to latent opportunities, or using the data to show providers where women are desirable clients to attract. In addition, as programming matures to specific DFS use cases, there is scope for the programme to focus on use cases and sectors that are known to have a livelihood impact on women in particular, such as digital education and healthcare payments. The work that has been completed already on Zambian mothers and female farmers in Nepal, as well as the DFS4Women partner even that led to Orange Liberia rolling out a pilot on onboarding female agents, is promising in this regard.

8

Evaluation Recommendation or Issue 7:

 

Knowledge management activities need to capture and disseminate lessons on project failures as well as project successes. Thus far, MM4P’s KM activities have largely focused on successful projects with limited coverage of the challenges and failures experienced by providers. This information is equally useful for stakeholders in the ecosystem to learn from the experience of other providers, and more focus should be placed on covering these lessons on failure. KM activities can also disseminate learnings on failures throughout a project’s lifecycle, rather than waiting for the end of a project to publish lessons[1].

 


[1] See http://www.cgap.org/blog/series/embracing-failure-ultimate-success-branchless-banking and https://www.raflearning.org/post/smallholder-finance-big-learnings-start-sharing-real-failures

9

Evaluation Recommendation or Issue 8:

Country teams will have to carefully consider which stakeholders to partner with as programming matures. Since MM4P has established its credibility in the markets where it operates, country teams are receiving more requests for support. This evaluation has found that MM4P support is most impactful when it is longer-term and where in-country DFS experts are embedded in the partner organisation.  While it was appropriate for the programme to experiment with different partner types at the beginning of the programme, this suggests that MM4P should now focus on fewer projects with larger budgets and longer ongoing technical support. This does not necessarily mean that MM4P should only partner with a particular partner type, or those that it has partnered with before. For example, working with fintechs may become a priority in the future as DFS market development continues. Given that fewer partners equate to higher risk, these partnerships must have a high focus on sustainability and demonstration of a viable business case as selection criteria.

10

Evaluation Recommendation or Issue 9:

 

UNCDF’s procurement and selection process for partners should be reviewed given the changing dynamics in the market. With the emergence of new types of financial service providers, such as fintechs which tend to be more innovative and fast and flexible, UNCDF will have to adapt its procurement process for partners to reflect these changing market dynamics. This should be achieved through a review of the current process against its ability to respond to the current and future needs of the DFS market.

11

Evaluation Recommendation or Issue 10:

 

MM4P country teams will have to assess the capacity of their technical staff given the importance of longer-term TA provided directly by DFS experts in-country. Most of the country teams are currently coping with the pipeline of projects but would require additional in-country staff should this pipeline increase further. In cases where current technical specialists’ contracts are ending, it is important for the programme to replace these with in-country specialists that can develop long term relationships with partners. In countries where a greater focus on policy/regulatory work is required, the programme may need to consider widening its skill set to include staff with experience in policy and regulatory issues. 

12

Evaluation Recommendation or Issue 11:

The programme should consider how to strengthen its pre-project feasibility assessment to improve the likelihood that investments will lead to sustainable customer outcomes. In several cases, projects were unable to be implemented as they were planned in the design phase. The table below provides a few examples from the programme where this was the case. MM4P undertakes a thorough assessment of prospective partners before providing support, including a peer review of the project document by a technical specialist in another country and review by the investment committee. However, these examples suggest that the programme could benefit from other tools or frameworks for assessing viability, particularly methods for assessing whether the economics of a project makes sense (sufficient customer numbers and appetite, adequate infrastructure coverage, right regulatory approval processes in place, etc.). Learning from the programme thus far suggests that the first project with a partner should be short, with a relatively short feasibility assessment, since the team learns most about a partner and their feasibility by working directly with them. These small projects can act as the feasibility assessment for larger projects going forward.

13

Evaluation Recommendation or Issue 12:

 

MM4P should not focus solely on its market development indicators, and should simplify its country and project RM frameworks by significantly reducing the number of indicators it tracks. Priority indicators should be decided on by identifying a) priority outcomes for donors, b) indicators that the MM4P team frequently uses to make strategic and operational decisions. The programme’s market development stages provide a helpful framework for understanding market dynamics to inform programming decisions, but do not adequately capture MM4P’s contribution to shifts along the spectrum.

1. Recommendation:
Management Response: [Added: 2019/02/05]

Overall comments:

In 2013, based on the initial results of the new UNCDF MM4P programme, UNCDF chose to implement a new approach, moving away from large scale grant making in favor of placing strong DF (digital finance) expertise at country level to properly support all stakeholders, grow the expertise locally and position UNCDF as neutral technical broker in each country.  It introduced a market development approach for the DF market and is now recognized as a unique player in supporting digital finance.  To keep the team focused on technical work, MM4P decided to consolidate important operational, communications and results measurement functions for a growing number of countries in a Project Management Office in Brussels to better build cross-country skills, learning & sharing and to ensure the best quality support were available to all countries to allow country teams to focus on technical support rather than programme management.  The team of more than 40 persons mixed international and local staff with highly qualified technical experts recruited from industry leaders.  MM4P considers this mid-term review as a very positive evaluation which highlights the value of the technical team on the ground, their approach to engaging partners through continuous interaction and relationship building as an effective approach to building markets.  The recommendations provided are in line with future orientation of the programme and UNCDF has already taken steps to implement many of them.

Key Actions:

2. Recommendation:

Evaluation Recommendation or Issue 1:

 

The programme TOC and country results chains should be updated to reflect what has been learnt in each market. MM4P has been operating in each market for some time and has generated important learning about the market context, the key barriers to DF market development, and programming opportunities that have arisen in each country. Since the programme’s TOC and country results chains were developed at inception when programming activities had not been rolled out, there is now an opportunity for these to be updated. These should also explicitly state assumptions, risks and possible external factors for each link in the ToC, to assist with future assessments of MM4P’s contribution to market development shifts. The learning generated from this mid-term review should provide some input to this process, such as the role of innovation as a potential workstream and the need to incorporate KM and ecosystem work as causal links in the chain, as well as better articulating gender outcomes at the customer outcome level. However, each country team should consider how the state of the market has evolved since inception, and where results chains need to be updated and adapted as a result.

Management Response: [Added: 2019/02/05]

Given the dynamic market and the lessons learned implementing a market development programme, the TOC has been an iterative process since its development in 2013.  Adapting the TOC has been a primary focus since late 2017 when MM4P started the process of developing a new strategy and framework to adapt its activities to quickly changing market dynamics around digital finance and digital economies. The strategy includes two new workstreams:

  • Inclusive Innovation: this workstream reflects the expanded scope of the program to contribute to the development of inclusive innovation ecosystems and accelerate the development and usage of digital services in key sectors such as energy, health, education and agriculture.
  • Empowered Customers: this workstream is included as a separate work stream to reinforce within our program the focus on the empowerment of key segments (youth, women, refugees, migrants, farmers, MSME’s).

 

As part of the new strategy, the programme Theory of Change and country results chains are updated to reflect the programme’s lessons learnt and include the inclusive innovation and customer empowerment workstreams and to show the specific outcomes related to focus segments including women. Also, MM4P is currently reviewing its iterative processes to more frequently update the Theory of Change and results chains based on results and learnings from programme activities.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1. Include workstream Inclusive Innovation in strategy
[Added: 2019/02/05]
MM4P 2018/12 Completed
1.2. Include workstream Empowered Customers in strategy
[Added: 2019/02/05]
MM4P 2018/12 Completed
1.3. Update programme Theory of Change
[Added: 2019/02/05] [Last Updated: 2019/03/29]
MM4P/ RM 2019/06 Completed Updated History
1.4. Update results chain
[Added: 2019/02/05] [Last Updated: 2019/10/01]
MM4P/ RM 2019/09 Completed The RMF has been updated globally with key indicators for the new strategy. The ToC and RMF has been applied to 2 FIPA countries during the Kuala Lumpur retreat in June. To measure our impact at sector level, the Inclusive Digital Economy Scorecard is now used in 8 countries and will be applied globally this year and next year. For new projects with a specific scope, the global ToC is adapted to reflect the scope of the activities History
3. Recommendation:

Evaluation Recommendation or Issue 2:

 

MM4P will have to incorporate innovation into its work stream as DFS markets develop and new types of providers emerge. Innovation here relates to the development of new business models for financial services providers, such as fintechs, and other service providers where digital payments form part of the new business models. Examples of innovation already emerging from the programme include fintechs (agile tech-focused financial service providers that use technology to change or improve the way financial services are produced and consumed) and digital platforms and marketplaces in the sharing economy that make use of digital payment options (such as Tootle in Nepal). Once digital payment rails are in place and the focus shifts to developing use cases around these rails, supporting innovation becomes an important activity for DFS market development. MM4P is already thinking about how to incorporate this into its programme design from the outset, including how to encourage and engage with non-traditional financial service providers, such as fintechs, which should be continued and formally built into the programme design.

Management Response: [Added: 2019/02/05]

Management Response:

 

See response to Recommendation 1 - MM4P is in the process of developing a new strategy which includes a workstream Inclusive Innovation. As part of its annual planning exercise, MM4P has started to structure its activities according to the new programme design. The 2019 country workplans all include activities to deliver under workstream Inclusive Innovation.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1. Include workstream Inclusive Innovation in strategy
[Added: 2019/02/05]
MM4P 2018/12 Completed History
2.2. planning and budgeting under new program design for 2019 to include inclusive innovation workstream at country level
[Added: 2019/02/05]
MM4P 2018/12 Completed
4. Recommendation:

Evaluation Recommendation or Issue 3:

 

MM4P should articulate the impact of its programming beyond financial inclusion to account for shifting funding priorities. The funding of financial inclusion activities within the donor community has shifted from considerations of financial inclusion as the end objective, to a broader focus on financial inclusion as a means to other objectives. These include improvements in the livelihoods of specific customer segments – such as women, refugees, and the youth – and the support of particular sectors – such as energy, food security, and agriculture. MM4P should monitor these shifting funding priorities among key donors and articulate the impact its work has on these priority segments/sectors in order to attract funding going forward

Management Response: [Added: 2019/02/05]

Management Response

 

Linking digital finance to the real economy and greater development issues is important for several reasons, namely:  (i) one of the greatest hurdles to increasing customer use is the lack of good uses cases for low income families; (ii) actors in the real economy are more likely to drive these uses cases and invest in them than the financial sector itself; (iii) such a shift is consistent with the United Nation’s commitment that its resources be used to achieve the Sustainable Development Goals; and (iv) funding opportunities for UNCDF necessitate this shift as its traditional funders also align their support to the 2030 Agenda. 

MM4P is in the process of developing a new strategy that reflects the changes in market dynamics which increase the potential of leveraging technology and innovation in improving people’s livelihoods, beyond financial inclusion. The strategy accommodates different impact pathways, depending on the focus segment and sector. As part of the new strategy, a Theory of Change will be developed including each target segment (women, youth, refugees, farmers) and sector (finance, agriculture, energy, health, education), visualizing the potential benefits and customer impact of services leveraging technology for these segments/sectors.

Since its inception, MM4P has targeted specific population segments in some of its projects (e.g. youth agents in Senegal, needs and barriers of mothers in Zambia, women agents in Malawi, smallholder farmers in Uganda and Nepal). More recently MM4P has started implementing projects across various sectors, including agriculture, education, health, energy and transport).  These projects have allowed UNCDF to increase skills and knowledge of specific sectors and segments that are leveraged in the development of the new strategy. 

To ensure alignment to donor priorities, the new strategy is developed in consultation with key donors. MM4P has designed a donor mapping at country and global level, showing donor priorities across the different sectors and segments. The programme will continue to implement across its countries.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1 Document UNCDF projects around key sectors and segments to use with donors and proposals
[Added: 2019/02/05] [Last Updated: 2019/07/08]
MM4P/ KM/FIPA 2019/06 Completed Document “Topic 3 1 KM” is the list of our global publications. For each publication, it is mentioned the workstreams, the sectors and/or segments. This document is our database for knowledge management. It is used to select key references used with donors, proposals, etc. History
3.2. Implement donors mapping, including donor priorities - global and country level
[Added: 2019/02/05] [Last Updated: 2019/07/09]
MM4P 2019/06 Completed The donor mapping has been done for Benin, Senegal and Zambia (MCF countries). But on top of that we used same approach for an additional 10 countries. The approach has also been presented to the FIPA team meeting and will be used widely within FIPA this year (22 countries). History
3.3. Develop segment- and sector- Theory of Change
[Added: 2019/02/05] [Last Updated: 2019/07/08]
MM4P/RM 2019/06 Completed It has been agreed that a specific Theory of Change will not be done for each segment and sector. The global ToC has been currently used for other segments without issues. We will keep it simple and use one ToC as of today. But we will make sure to adapt the RMF with the right indicators for segments and sectors. The last version of the Theory of Change is now widely used in all FIPA countries (after the Team Meeting).0 Again, the impact is well beyond the 3 MCF countries. It is FIPA wide. History
5. Recommendation:

Evaluation Recommendation or Issue 4:

Given funding restrictions and donor priorities that have resulted in limitations on regulatory/policy and infrastructure work in some countries, MM4P needs to have plans in place to work around these limitations. The table below (see final report for the table as it was too long to insert here) provides a few examples from the programme of where policy/regulatory impact was created despite the limitations, in countries where these limitations existed. These examples should be replicated across the programme where possible. Importantly, these plans need to be in place at the outset of programming and built in to the programme design. MM4P can also draw on its existing strengths at ecosystem-level, namely information provision and convening to focus stakeholders’ attention on these limitations, even where MM4P does not have funding or resources to directly address these limitations.

Management Response: [Added: 2019/02/05]

MM4P always recognized and stressed the importance of the policy and regulatory environment and infrastructure as key building blocks of the development of a digital finance ecosystem. MM4P will continue to engage with regulators as explained in the examples mentioned in the mid-term review. Infrastructure is a key component of the new strategy, as part of the workstream that aims to establish Open Digital Payment Ecosystem. Focus areas related to infrastructure include enhancing network and data coverage, phone ownership and appropriate financial infrastructure accessible to all. 

 

Bill and Melinda Gates Foundation recently committed US$ 5.9 million for activities specifically focused on Policy & Regulation in Africa which MM4P will implement in the coming 4 years. MM4P will continue to fundraise for activities specifically aimed on improving the regulatory environment and addressing infrastructural gaps.

 

Country strategies will be drafted during Q1 & Q2 2019 based on the new strategy, including country-specific objectives and activities in policy, regulation and infrastructure.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
4.1. Develop country strategies, including objectives and activities related to policy, regulation and infrastructure
[Added: 2019/02/05] [Last Updated: 2019/07/09]
MM4P and all FIPA countries adopting new strategy as of 2019 2019/06 Completed Since the beginning of the year, MM4P adopted the new strategic framework for their strategies and country plans. It includes the workstreams “Enabling Policy and Regulation” and “Open Digital Payment Ecosystem” highlighting key activities on policy, regulation and infrastructure. The country strategies template is now being used for the 22 priority countries of FIPA. The lessons learned from MM4P are now used for all FIPA. This template has been used during the FIPA Team Building and Sharing meeting in June 19. History History
6. Recommendation:

Evaluation Recommendation or Issue 5:

MM4P needs to develop careful sustainability plans for its ecosystem facilitation role and start implementing this plan now to provide enough time before programming winds down. MM4P is playing a critical convening function within the DFS ecosystem, for example with the formation of DFS working groups. However, MM4P is often the only stakeholder playing this convening role and so needs to develop mechanisms for stakeholders to continue collaborating once the programme ends. Taking a lesson from MM4P in Malawi, the formation of an industry association with funding contributed by members appears to be a sustainable approach to having this convening function continued. MM4P should look to set up similar industry associations in countries with DFS working groups, and in Nepal where a DFS working group has yet to be formed.

Management Response: [Added: 2019/02/05]

MM4P aims at sustainability of its engagement in the market, including the ecosystem activities. In 2019, MM4P will engage in specific activities to ensure the sustainability of its facilitation role in the countries where the programme is closing. Note that the facilitation role is mostly aimed at improved public-private dialogue and ensuring greater consultation around policies and regulation.  As noted by the evaluators, UNCDF plays this unique role in many countries and this should be a “selling point” to funders for UNCDF’s continued presence in countries where sustaining such engagement is unlikely without outside funding and expertise.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
5.1. Include activities to ensure sustainability of MM4P facilitation role after closing of a country programme in 2019 country workplans
[Added: 2019/02/05] [Last Updated: 2019/07/08]
MM4P 2019/06 Completed As of today, the only country that will be closed end of this year is Benin. For Senegal and Benin, we are continuing the implementation with new funders. Other countries (Zambia, Malawi, Uganda, Nepal, Malaysia, China, Senegal) are not closing. The country plan for Benin includes specific activities regarding sustainability. History
5.2. Include long-term plan for sustainability of facilitation role in country strategies
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P and all FIPA countries adopting new strategy as of 2019 2019/12 Completed In all countries, UNCDF aimed at turning DFS working groups into sustainable models after the end of UNCDF presence. It is part of the activities for countries where UNCDF has presence and is ending its programs. In Malawi, the DFS working group has been turned into an association. In Benin, the industry has taken over the responsibility of this model. History
7. Recommendation:

Evaluation Recommendation or Issue 6:

 

MM4P can better incorporate the mainstreaming of gender into its programming activities over the second half of the programme. To date, requiring providers to collect sex-disaggregated data has been a tick-box exercise rather than encouraging providers to make use of the data. Programming should therefore focus on building the business case for using sex-disaggregated data among providers as part of MM4P support. This could include capacity building/TA or research such as market sizing and opportunity analysis for female customers, encouraging providers to make use of their data to tap in to latent opportunities, or using the data to show providers where women are desirable clients to attract. In addition, as programming matures to specific DFS use cases, there is scope for the programme to focus on use cases and sectors that are known to have a livelihood impact on women in particular, such as digital education and healthcare payments. The work that has been completed already on Zambian mothers and female farmers in Nepal, as well as the DFS4Women partner even that led to Orange Liberia rolling out a pilot on onboarding female agents, is promising in this regard.

Management Response: [Added: 2019/02/05]

Recognizing that women represent a substantial development and business opportunity for providers, supporting activities that address the unique barriers and needs of women has been one of the priority areas of MM4P since inception. The programme has implemented several projects specifically targeting women, conducted focused research on the barriers and needs of women, included sex-disaggregated indicators in reporting templates developed for and adopted by regulators for market monitoring, and organized a global partner event with the theme DFS4Women. Market research is disaggregated by sex, and MM4P requests sex-disaggregated data from all partners as part of their reporting requirements.

 

Generally, stakeholders’ interest in better reaching women with appropriate DFS remains low. This is also reflected in the limited number of stakeholders that monitor sex-disaggregated KPIs, and the challenges that persist in partner reporting on sex-disaggregated data. The programme recognizes the need to improve its efforts in better reaching women. As part of the new strategy MM4P will identify how to better mainstream the gender perspective across programme activities, incorporating the programme lessons learnt so far. This also includes an approach on improving partners’ data collection, analysis and reporting of sex-disaggregated results. The strategy also specifically focuses on how programme activities can contribute to women empowerment, which will be incorporated and tailored to country strategies and activities.

It should be noted that MM4P experience shows that at earlier stages of DFS market development, when the commercial potential of DFS is not yet recognized by most providers, it is less likely stakeholders show interest in increasing their reach to women. Providers operating in more advanced DFS markets are more likely to prioritize better reaching women.

Also, the programme expects an increased interest in women from Fintech and start-up companies that are entering markets with specific products and services. These types of companies generally adopt a more customer centric approach. In its new strategy, MM4P will leverage and support these players in addressing the barriers and needs of women. 

Finally policy and regulatory pressure make a difference. Where private sector “buy-in” is lagging, UNCDF may need to focus on external pressure to get the basic prerequisite of data collection and reporting on gender on the industry’s agenda.   

Key Actions:

Key Action Responsible DueDate Status Comments Documents
6.2. Mainstream gender and design focus on women empowerment in global and country strategies
[Added: 2019/02/05] [Last Updated: 2019/07/09]
MM4P and all FIPA countries adopting new strategy as of 2019 2019/06 Completed MM4P and FIPA have updated the gender strategy as part of the new strategic framework. In the strategies for Benin, Senegal, Zambia, Nepal, there are specific activities regarding gender mainstreaming and women empowerment. The same framework is used within MM4P and also FIPA. History
8. Recommendation:

Evaluation Recommendation or Issue 7:

 

Knowledge management activities need to capture and disseminate lessons on project failures as well as project successes. Thus far, MM4P’s KM activities have largely focused on successful projects with limited coverage of the challenges and failures experienced by providers. This information is equally useful for stakeholders in the ecosystem to learn from the experience of other providers, and more focus should be placed on covering these lessons on failure. KM activities can also disseminate learnings on failures throughout a project’s lifecycle, rather than waiting for the end of a project to publish lessons[1].

 


[1] See http://www.cgap.org/blog/series/embracing-failure-ultimate-success-branchless-banking and https://www.raflearning.org/post/smallholder-finance-big-learnings-start-sharing-real-failures

Management Response: [Added: 2019/02/05]

 

It is important to differentiate projects that have failed (and were discontinued) from failures within projects that MM4P used as learnings to adapt the projects for better impact. For the latter, MM4P widely shared successes but also challenges faced in various projects.  This has been for example the case for the various agriculture value chains and all projects. But this can be improved in future publications.

 

For the former, we have some projects that have been discontinued for various reasons (partnership not found between partners, disengagement of the government, etc.).  Partners often do not want to publicly speak about these failures and this is a challenge for MM4P.  For these projects, MM4P will analyze best ways to disseminate lessons from these projects with the concerned partners.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
7.1. Review KM/RM processes and templates to better capture lessons learnt on failures and challenges along the project life cycle.
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/KM RM 2019/12 Completed The KM/RM processes have been reviewed and include a thorough and iterative cycles at project (quarterly), country (annual) and global (annual) levels. It allows a continuous review of UNCDF work and a continuous questioning and review of the theory of change to increase the impact of our work. Processes and templates will be continuously improved during the coming years based on the feedback from the team. In the attached powerpoint presentation, you can find the processes for RM, presented last week in Brussels to the team. History
7.2. Include publications on challenges and failures in country KM workplans
[Added: 2019/02/05] [Last Updated: 2019/03/29]
MM4P/KM 2019/03 Completed Included History
9. Recommendation:

Evaluation Recommendation or Issue 8:

Country teams will have to carefully consider which stakeholders to partner with as programming matures. Since MM4P has established its credibility in the markets where it operates, country teams are receiving more requests for support. This evaluation has found that MM4P support is most impactful when it is longer-term and where in-country DFS experts are embedded in the partner organisation.  While it was appropriate for the programme to experiment with different partner types at the beginning of the programme, this suggests that MM4P should now focus on fewer projects with larger budgets and longer ongoing technical support. This does not necessarily mean that MM4P should only partner with a particular partner type, or those that it has partnered with before. For example, working with fintechs may become a priority in the future as DFS market development continues. Given that fewer partners equate to higher risk, these partnerships must have a high focus on sustainability and demonstration of a viable business case as selection criteria.

Management Response: [Added: 2019/02/05]

The maturity of the market and the depth of the relationship with a partner are critical factors when choosing partners.  MM4P engages with and supports stakeholders in the market through its full-time in-country presence of digital experts. In its engagement with market players, MM4P plays the role of a neutral broker. These are key components of the programme that allow MM4P to continue to identify trends and opportunities in the market rather than building exclusive relationships.

 

The selection of projects and partners are determined by the existing market constraints and the scope of the engagement and design of projects reflect the need of the partners and the market as a whole. MM4P’s experience shows that partnerships with private sector players, particularly in new markets, require a buy-in phase that is done through one or two short term engagements, to establish the relationship, trust and show how the programme can support the partner. MM4P recognizes that longer term involvement with partners is beneficial and is assessing the appropriate approach as part of the new strategy—but not all partners wish longer term engagements and not all remain good (or necessary) partners over time.  The number and type of partners MM4P supports depends on the market constraints that MM4P aims to unlock, and the funding available in each country.

As part of its new strategy to better leverage Fintech and start-up companies to unlock market development, MM4P plans to provide shorter-term support to a larger number of such players. Through these engagements, players with the most viable business plan and largest potential for contributing to market development will be identified for longer-term support. The programme recognizes that the commercial viability of the business models developed with MM4P support is key in the sustainability of programme activities. As part of the new strategy, due diligence processes are reviewed as well as the monitoring systems related to project sustainability.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
8.1. Detailed analysis of key market constraints in each country strategy (2019-2024) and potential activities and support to partner
[Added: 2019/02/05] [Last Updated: 2019/07/08]
MM4P and all FIPA countries adopting new strategy as of 2019 2019/06 Completed A detailed analysis of key market constraints is a key pillar of the new strategic framework as we focus on market development for inclusive digital economies. Ecosystem activities and partners activities are planned based on the identified constraints. Our objective is to unlock these constraints to develop sustainable development of the market in the countries where we operate. It has been done for Benin, Senegal and Zambia. Again the framework is used for MM4P and beyond. All FIPA countries used the same framework and this is part of all country strategies. History
8.2. Review due diligence processes
[Added: 2019/02/05] [Last Updated: 2019/10/01]
MM4P 2019/09 Completed The review is already reflected in UNCDF’s operation manual. Further policies will be issued after the approval of Digital Strategy Prodoc. History
8.3. Improve monitoring processes of project sustainability
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/RM 2020/04 Completed As mentioned in topic 7.1, the sustainability of the project will be assessed during the quarterly review and evaluation of the project. During the quarterly reporting (quantitative and qualitative) and the steering committee, sustainability will be part of the project monitoring. History
10. Recommendation:

Evaluation Recommendation or Issue 9:

 

UNCDF’s procurement and selection process for partners should be reviewed given the changing dynamics in the market. With the emergence of new types of financial service providers, such as fintechs which tend to be more innovative and fast and flexible, UNCDF will have to adapt its procurement process for partners to reflect these changing market dynamics. This should be achieved through a review of the current process against its ability to respond to the current and future needs of the DFS market.

Management Response: [Added: 2019/02/05]

We understand that the recommendation refers to how MM4P adapts to the market dynamics.  Please note that (1) procurement is the process of acquiring goods and services for the organization whilst partner selection is a part of the (2) programming process, which enables UNCDF to provide technical and financial assistance to partners. Within this context both the procurement processes and programming guidelines (known as UNCDF’s operations manual) are decided upon by UNCDF senior management, whereas partner selection is largely the pure view of the programme management.

 

Since its inception, the programme has continuously looked at innovative and flexible ways to engage with partners, to be as reactive as possible in fast and dynamic markets.  With an increased focus on innovation and the willingness to work with start-ups and entrepreneurs, this is becoming a key priority for MM4P.

 

Partner selection and assistance:

MM4P’s method of partner selection is somewhat different than other programmes. Given the early failures in using request for applications and proposals, MM4P uses and a series of engagements that include training, direct technical support, contracted technical support and grants to help identify the best partners and project going forward.  It is a process of winnowing.  This works particularly well in nascent markets or within nascent stakeholder groups (e.g. Fintechs). 

 

MM4P is currently testing a model where technical assistance support (mentoring) and financial support (small grants) for innovation journeys with start-up companies are managed by an external company. This approach proposed by MM4P also led to the modification of the operations manual. Based on the learnings of implementing the approach in Nepal, Malaysia and Uganda in 2019, the model may also be adopted in other countries.

Procurement processes:

If MM4P is keen to working together with its programme management unit to continuously improve these processes to respond to the market needs. For example, in order to facilitate processes the programme proposed changes to UNCF individual consultant policy that have led to better working conditions to retain DFS experts. MM4P has also always been at the forefront of UNCDF when using agile procurement instruments such as Log Term Agreements and rosters. It agrees that working with FinTechs requires a rapid response approach and will prioritize this issue in its work and with the UNCDF MSU. 

Key Actions:

Key Action Responsible DueDate Status Comments Documents
9.1 Test new section approach to support start-ups and entrepreneurs
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P 2019/12 Completed Our work with start-ups in various countries (Zambia, Uganda, Burkina Faso, Malaysia, Nepal) enabled to test the approach with starts-ups. From now on, the approach will be improve on a continuous basis with UNCDF’s Management Support Unit (MSU) based on the feedback of the team. History
9.2 Enhance procurement plan and resource tracking system for the programne,
[Added: 2019/02/05] [Last Updated: 2019/12/16]
PMU/MSU with MM4P support 2019/12 Completed All countries use standardized annual workplan’ (AWP) templates. The Program Support team will compile all projects from the AWPs in a centralized project’ pipeline. This will result in improving the planning and procurement for the program and FIPA. As for every processes and templates, there will be continuous improvements based on the feedback of the team. History
11. Recommendation:

Evaluation Recommendation or Issue 10:

 

MM4P country teams will have to assess the capacity of their technical staff given the importance of longer-term TA provided directly by DFS experts in-country. Most of the country teams are currently coping with the pipeline of projects but would require additional in-country staff should this pipeline increase further. In cases where current technical specialists’ contracts are ending, it is important for the programme to replace these with in-country specialists that can develop long term relationships with partners. In countries where a greater focus on policy/regulatory work is required, the programme may need to consider widening its skill set to include staff with experience in policy and regulatory issues. 

Management Response: [Added: 2019/02/05]

Since its inception, MM4P has focused on developing a team with strong technical expertise in DFS, which has proven to be key in the achievements of the program.  The team is composed of UNCDF staff and long-term consultants.  Long-term engagement of our team on the ground with stakeholders is a unique asset for UNCDF.  A key priority of MM4P management has been to grow the team and to offer them opportunities in various countries.

 

MM4P recognizes the importance of continuing its efforts in building a team with strong DFS expertise, but also to increase the mobility of resources across teams, particularly in the case of funding gaps. 

 

Several areas should be investigated by UNCDF:

  • Contracts allowing to keep consultants for longer periods
  • Investigate career growth paths within UNCDF that include long-term consultants
  • Putting operational procedures in place to allow staff and experts to do short-term assignments across UNCDF
  • Plan DFS demand centrally with UNCDF and make budget available to fill-in funding gaps for key resources within UNCDF

Key Actions:

Key Action Responsible DueDate Status Comments Documents
10.1 Include contracts’ renewals and DF experts procurement in countries workplans
[Added: 2019/02/05]
MM4P 2018/11 Completed
10.2 Investigate career growth paths within UNCDF
[Added: 2019/02/05] [Last Updated: 2019/07/08]
HR with MM4P support 2019/06 Completed We started investigation and it will be a ongoing process to see best ways to keep our team to deliver the strategy. History
10.3 Plan DFS demand centrally with UNDF and make budget available to fill-in funding gaps for key resources within UNCDF
[Added: 2019/02/05] [Last Updated: 2020/03/31]
FIPA Management 2020/03 Completed For the planning exercise of 2020, standardized templates have been used for workplans and budgets in all countries. This allows FIPA to have a standardized way of working for all countries. These standardized workplans and budgets allow a streamlined assessment of resources, as well as identifying potential gaps in resources and addressing funding gaps as needed. History
12. Recommendation:

Evaluation Recommendation or Issue 11:

The programme should consider how to strengthen its pre-project feasibility assessment to improve the likelihood that investments will lead to sustainable customer outcomes. In several cases, projects were unable to be implemented as they were planned in the design phase. The table below provides a few examples from the programme where this was the case. MM4P undertakes a thorough assessment of prospective partners before providing support, including a peer review of the project document by a technical specialist in another country and review by the investment committee. However, these examples suggest that the programme could benefit from other tools or frameworks for assessing viability, particularly methods for assessing whether the economics of a project makes sense (sufficient customer numbers and appetite, adequate infrastructure coverage, right regulatory approval processes in place, etc.). Learning from the programme thus far suggests that the first project with a partner should be short, with a relatively short feasibility assessment, since the team learns most about a partner and their feasibility by working directly with them. These small projects can act as the feasibility assessment for larger projects going forward.

Management Response: [Added: 2019/02/05]

It must be considered that the programme operates within the framework of the Least Developed Countries (LDCs). Consequently, the risks are higher than in other markets and there is often a “chicken and egg paradigm” (e.g. customer demand is weak because of the lack of infrastructure which is weak because of the lack of perceived customer demand).  Lacking the resources to fully address all sides of supply and demand, the programme has pushed some projects that may not have the perfect market conditions to build awareness, interest and hopefully a more attractive market opportunity or pressure to change regulations.

 

In the context of the new strategy, processes will be reviewed, simplified and strengthened. The risks are frequently discussed and are in the Project Appraisal Documents, however they could be better acknowledged as part of the pre-project feasibility. 

Kindly note that MM4P is also working in new tools (Toolkit for managing partner steering committees (SCs) as well as streamlining its’ monitoring tools (project works books).

Key Actions:

Key Action Responsible DueDate Status Comments Documents
11.1 Review and strengthen project design phase including due diligence
[Added: 2019/02/05] [Last Updated: 2019/10/01]
MM4P/PM 2019/09 Completed The review is already reflected in UNCDF’s operation manual. Further policies will be issued after the approval of Digital Strategy Prodoc History
11.2 Review and strengthen project workbooks
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/ PM 2020/04 Completed Projects workbooks have been updated during the review of the Strategic Support unit processes. As mentioned under topic 7.1, this will improve monitoring of our projects and our theory of change for better impact. As for every processes and templates, there will be continuous improvements based on the feedback of the team. History
11.3. Launch and implement toolkit for managing SCs
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/ PM 2019/09 Completed Key country staff were trained in the management of Service Contracts during the MSU workshop in October 2019. A new tool kit including standardized templates has been released and shared with staff. History
11.3. Launch and implement toolkit for managing SCs
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/PM 2019/12 Canceled A tool kit will be released after the inception phase of SIDA Uganda project. History
13. Recommendation:

Evaluation Recommendation or Issue 12:

 

MM4P should not focus solely on its market development indicators, and should simplify its country and project RM frameworks by significantly reducing the number of indicators it tracks. Priority indicators should be decided on by identifying a) priority outcomes for donors, b) indicators that the MM4P team frequently uses to make strategic and operational decisions. The programme’s market development stages provide a helpful framework for understanding market dynamics to inform programming decisions, but do not adequately capture MM4P’s contribution to shifts along the spectrum.

Management Response: [Added: 2019/02/05]

The programme Results Measurement Framework (RMF) was developed by a DCED expert, in close consultation with the Mastercard Foundation, one of MM4P key donors. The market development measurement reflects the priority outcomes of this donor. In addition to the market development measurement (top-down measurement), the programme also measures the results of its projects (bottom-up measurement).

 

It must also be noted that after the first year of implementation of the RMF, the programme has focused on the eight market development indicators to measure the progress of the markets. This reduced the number of indicators tracked on country level. The project monitoring tools have also been simplified to ensure the ability of the partner to report on the requested indicators.

The programme acknowledges the need to better include the aggregation of direct programme results in its monitoring and reporting, in combination with market development measurement. MM4P also recognizes the need to more systematically assess and document the contribution of its activities on market development, particularly of its ecosystem activities, and better use these insights to inform programming.

In the context of the new strategy, MM4P is currently updating the Theory of Change, and will develop the RMF for the new strategy, including indicators and measurement plan. The RM strategy will also be revised, including all RM-related processes, tools and templates.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
12.1. Develop programme and country RMFs for new strategy including ToC, indicators and measurement plans
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/RM 2020/04 Completed The global results measurement framework has been finalized and is currently adopted by the countries. Each country reported global indicators for 2020 in an harmonized way. The adoption of these tools will be progressive as we finalized previous programs and apply the new strategic framework. History
12.2 Revise RM strategy (processes, tools, templates)
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/RM 2019/12 Completed The RM strategy is reviewed on an ongoing basis. As for every processes and templates of the Strategic Support Unit, there will be continuous improvements based on the feedback of the team. In Q1 2020, we will finalize the work with our MSD expert in Uganda who will also provide input on RM strategy. History
12.3. Develop tracking and measurement plan for MM4P ecosystem contribution
[Added: 2019/02/05] [Last Updated: 2019/12/16]
MM4P/RM 2019/12 Completed The RM strategy is reviewed on an ongoing basis. As for every processes and templates of the Strategic Support Unit, there will be continuous improvements based on the feedback of the team. In Q1 2020, we will finalize the work with our MSD expert in Uganda who will also provide input on RM strategy. History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org