Evaluation Detail

Terminal evaluation - Sustainable Urban Transport

Evaluation Plan:: 2013-2017, India
Evaluation Type:: Final Project
Planned End Date:: 12/2017
Completion Date:: 06/2017
Status:: Completed
Management Response:: Yes
Evaluation Budget(US $):: 25,000

Document	Type	Language	Size	Status	Downloads
ToR International Consultant for SUTP.pdf	tor	English	764.91 KB	Posted	353
ToR National Consultant for SUTP.pdf	tor	English	1050.59 KB	Posted	331
TE- Sustainable Urban Transport Project-Final.pdf	report	English	1740.46 KB	Posted	436

Title

Terminal evaluation - Sustainable Urban Transport

Atlas Project Number:

00048794

Evaluation Plan:

2013-2017, India

Evaluation Type:

Final Project

Status:

Completed

Completion Date:

06/2017

Planned End Date:

12/2017

Management Response:

Yes

Focus Area:

1. Others

Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)

1. Output 5.2. Effective institutional, legislative and policy frameworks in place to enhance the implementation of disaster and climate risk management measures at national and sub-national levels

Evaluation Budget(US $):

25,000

Source of Funding:

Evaluation Expenditure(US $):

21,000

Joint Programme:

Joint Evaluation:

Evaluation Team members:

Name	Title	Email	Nationality
Mr. Roland Wong	International Consultant	rolandwong@shaw.ca
Dr. Sudhakar Yedla	National Consultant	s_yedla@yahoo.com	INDIA

GEF Evaluation:

Yes

GEF Project Title:	Sustainable Urban Transport Program
Evaluation Type:	Terminal Evaluation
Focal Area:	Climate Change
Project Type:	FSP
GEF Phase:	GEF-4
GEF Project ID:	3241
PIMS Number:	3214

Key Stakeholders:

Countries:

INDIA

Lessons
1.	Implementation of Component 1A contained activities related to national research such as municipal level data collection for sustainable urban transport purposes, an activity which expanded from 5 cities to more than 40 cities, diffusing efforts to effectively manage data for the Knowledge Managment Centre.
2.	With a mandate to build the capacity of IUT as the main entity to advise MoUD on all issues related to SUT development in India, the PMO could have accelerated this process through networking with relevant national institutes on urban transport in addition to international institutes.
3.	To ensure constant improvement of training programs, aggressive follow-up on feedback surveys of training programs is necessary.
4.	An energetic and efficient project management unit is required to manage a large capacity building project where there are numerous consultations and approvals required to select attendees of various training programs, especially within the Indian Government system
Findings
1.	3.1 Project Design and Formulation Design of the overall SUTP Project was conducted during the period of 2008-2009. The overall objective of the SUTP Project was to “reduce the growth trajectory of GHG emissions from the transport sector through the promotion of environmentally sustainable urban transport, strengthening government capacity to plan, finance, implement, operate and manage climate friendly and sustainable urban transport interventions, and increasing the moral share of environmentally friendly transport modes in selected cities”. The SUTP Project was to be implemented within 2 components: a component on national capacity development initiatives jointly managed by UNDP (Component 1A) and the World Bank (Component 1B), and the World Bank-managed Component 2 on demonstration SUT projects in selected cities. Tag: Emission Reduction Environment Policy Challenges Urban Local Governance Country Government Institutional Strengthening
2.	3.1.1Analysis of Project Planning Matrix As mentioned in Para 24, a PPM for Component 1A was prepared in July 2013. As shown in Appendix E, the PPM for Component 1A of SUTP provides 7 outcome level indicators and targets (3 goal/objective level and 4 outcome level) and 19 output level indicators and targets to guide implementation of Component 1A towards its objective of “Government capacity strengthened to plan, finance, implement, operate and manage climate-friendly and sustainable urban transport interventions at national, state and city levels”. While the intent of the PPM and its description of indicators is reasonably clear in terms of its targets, wording of most of the indicators and targets do not meet SMART criteria 6 and best practices for preparing PPMs. While a large number of comments can be made on the language of the PPM, some specific comments includes: The description of the Project outcomes are more oriented to Component 1A outputs rather than intended Component 1A outcomes. For example, Outcome 2 could have been described as “government officials, urban planners, practitioners are enabled to prepare investment-level sustainable urban transport plans”. Similarly, Outcome 3 could have been described as “sustainable urban transport plans being prepared on the basis of available and updated manuals, toolkits and reference guides that were based on best international practices”. The “description of indicators” are actually “sub-outcomes” and not “indicators” that describe a quantified output. For example, on the overall project goal (impact), the indicator could have been “number of cities with MoUD approved Comprehensive Mobility Plan (CMP)”. This would be clearer in terms of what is being monitored for progress; The evaluators believe there is no necessity to provide outcome level indicators and targets, since the outcome itself should describe the target. Moreover, the outputs under a particular outcome should contribute towards the achievement of the outcome; All output descriptions should be described as outputs, not outcomes. For example, Output 1.2 can be described as “IUT accreditation for Sustainable Urban Transport”, or Output 1.3 described as “a functional knowledge management data centre at IUT”. The targets for each of these indicators would remain as “1”; Tag: Programme/Project Design Project and Programme management
3.	3.1.2 Risks and Assumptions The July 2013 PPM for Component 1A provides assumptions under which the proposed outcomes and outputs could be achieved. These assumptions also infer risks to the achievement of outcomes and outputs. Many of these assumptions are related to SUTP’s implementing partner, MoUD, providing strong support and direction towards the use of the Institute of Urban Transport (IUT) as a national resource for developing SUT projects in large urban centres throughout India. Examples of the assumptions in the July 2013 PPM includes: Tag: Human and Financial resources Urbanization Technical Support
4.	The ProDoc, however, does contain an Off-line Risk Log for Component 1A in Annex 2 for which Component 1A management staff can manage risks against the goals of Component 1A and its intended outcomes. Examples of some of these risks includes: Institutional and capacity development under the project will not be sustained; Delays in project implementation and poor quality due to involvement of multiple cities and states whose capacity varies and whose commitments to the projects may change; Late or delayed provision of the one-time financial corpus to IUT from MoUD; and Demonstration projects will not be replicated in other cities and states without GEF grant support. These risks should also be listed with the PPM against the various Component 1A outcomes as a link to the Off-line Risk Log. Tag: Human and Financial resources Risk Management Institutional Strengthening
5.	3.1.4 Planned Stakeholder Participation The ProDoc does not contain any stakeholder participation plan, nor has the evaluation team been provided with any evidence of any strategic plan for stakeholder participation for Component 1A. Since the primary purpose of Component 1A was to increase the knowledge and build the capacity of selected stakeholders at the federal, state and municipal levels to plan, design, implement and manage sustainable urban transport initiatives, the ProDoc should have identified the stakeholders, especially considering the numerous federal agencies, state governments in India, and municipalities who would have a stake in improving urban mobility within India’s busiest cities. This would have involved identification of priority federal agencies, and targeted selected cities in India for pilot SUTP initiatives. Moreover, there appears to be an absence of any analysis of stakeholders, which would assist in the prioritization of stakeholders for engagement. Tag: Capacity Building Institutional Strengthening National Institutions Regional Institutions
6.	3.1.5 Replication Approach Component 1A did not have a specific replication approach to building capacity of all relevant stakeholders in India on sustainable urban transport. However, the need to transfer knowledge on sustainable urban transport in India to a wide spectrum of stakeholders implies the replication (or vertical scale-up) of knowledge transfer and training sessions to a large number of stakeholders. The UNDP ProDoc for Component 1A describes a focus on building the capacity of IUT to become a primary SUT resource centre in India that would serve the needs of urban transport professionals. This would include building the capacity of IUT to initiate capacity development programs. These IUT programs would provide training of trainer (ToT) programs for over 1,000 planners, policymakers, and urban transport professionals throughout all levels of government. The design of Component 1A could have spread the focus onto additional institutions to assist IUT, many of whom could play a key role in replication (or a cascading effect) of the capacity building efforts. Excessive focus on IUT alone places higher risks of not achieving replication targets. Furthermore, another replication aspect of Component 1A was the need to decentralize IUT functions through the development of regional IUT centres. Given the importance of local participation on any SUT initiative, activities of Component 1A were to include support from IUT to build the capacity of local institutions in selected urban centres throughout India (under Subcomponent 1.4). This aspect does not appear on the July 2013 PPM as a target. Tag: Knowledge management Sustainability Capacity Building
7.	3.1.6 UNDP Comparative Advantage UNDPs comparative advantage to other donor agencies is its focus on policy-based and cross-sectoral approaches as well as building local capacities through effective collaboration with a wide range of local stakeholders (in this case, federal, state and municipal levels of government and the general public). This would include public and private sectors as well as technical experts, civil society and grassroots level organizations. These approaches are strongly applicable on national capacity building initiatives for SUT projects under Component 1A of SUTP. Given UNDP’s long track record on a wide variety of projects within the transport and energy sectors, UNDP has earned a reputation as a credible and trusted partner to the GoI in capacity building initiatives and is suited as an implementing agency for Component 1A. Particularly in issues that require high quality technical inputs, UNDP has a strategic advantage as a partner and the method of deploying technical experts with GoI’s line ministries has increased the effectiveness of GoI-UNDP initiatives. Tag: Policies & Procedures Donor Capacity Building Technical Support Civil Societies and NGOs National Institutions Private Sector Regional Institutions
8.	3.1.7 Linkages between Component 1A of SUTP and Other Interventions within the Sector 34.Component 1A is a part of a larger SUTP project, the other components of which are being implemented through World Bank management. However, the UNDP ProDoc for Component 1A does not identify any linkages of capacity building initiatives with other transport-related interventions in India including the World Bank-managed Component 1B. This is unfortunate since the capacity building taking place on Component 1B is the dissemination of technical transport planning skills to cities and states and financing urban transport projects that contribute towards reducing emissions. As such, this linkage to Component 1B would provide an end point of an exit strategy to those implementing Component 1A. Tag: Urban Human and Financial resources Project and Programme management International Financial Institutions Capacity Building
9.	3.1.8 Management Arrangements The implementing partner of Component 1A is the Ministry of Urban Development in accordance with UNDPs National Implementation Modality (now referred to as National Execution or NEX modality. The NEX modality tasks MoUD with responsibility for certifying work plans and approved budgets, reporting on procurement, coordinating and tracking co-financing, terms of reference for contractors and tender documentation, and chairing the Project Steering Committee (PSC). The Chair of the PSC was to be the National Project Director (NPD) from MoUD. The ProDoc also acknowledges that UNDP is a partner GEF agency for SUTP with the World Bank being the leading GEF agency, responsible for overall technical quality assurance for the entire SUTP Project. As such, UNDPs responsibilities to GoI were to ensure quality assurances in the implementation of national capacity building initiatives of Component 1A. These initiatives would then feed into the implementation of selected pilot states and cities for sustainable urban transport managed by the World Bank, as well as tailored capacity development at the local level to enable local staff to implement demonstration SUT projects. PMO staff also managed implementation of the World Bank-implemented components of SUTP that provided seamless interaction between UNDP and World Bank supported activities.
10.	3.2 Project Implementation The following is a compilation of key events and issues of Component 1A of SUTP implementation in chronological order: Signing of the SUTP ProDoc by the Ministry of Urban Development and UNDP India was on April 15, 2010, marking the official start of Component 1A implementation; The IUT business plan was completed with Component 1A support in late 2011; Early activities of UNDP on Component 1A involved IUT organizing and drafting manuals and toolkits for training as well as dissemination activities at international and national conferences and the SUTP website; The Component 1A midterm review (MTR) was conducted in early 2013 with a report issued in June 2013 that observed the need for a Component 1A logical framework, and the need for an exit strategy prior to the withdrawal of UNDP from its implementation of Component 1A; Consultant for the setup of the KMC commenced work in November 2014. Due to difficulties in obtaining municipal-level data and integrating available data into user-friendly database structures, this contract was extended into late 2016; A formal request was made in July 2015 to extend the project for 27 months to March 2018. This request was granted in December 2015; On October 31, 2016, UNDP substantially completed its activities on Component 1A with the completion of the training of trainers and professionals, the launching of the Knowledge management centre by MoUD and the development of all training modules as outlined in the project document. Tag: Communication Knowledge management Programme/Project Design
11.	3.2.1Adaptive Management Adaptive management is discussed in GEF terminal evaluations to gauge the project performance in its ability to adapt to changing regulatory and environmental conditions, common occurrences that afflict the majority of GEF projects. Without adaptive management, GEF investments would not be effective in achieving their intended outcomes, outputs and targets. Unfortunately, the early stages of the implementation of Component 1A were marked by perceptions of slow progress when in fact, the PMO for Component 1A were operating without a Project Planning matrix (PPM) without specific outcomes and targets on which to plan annual activities and allocate GEF budgets. Tag: Global Environment Facility fund Knowledge management Project and Programme management Jobs and Livelihoods
12.	3.2.2 Partnership Arrangements The PMO for Component 1A facilitated numerous strong and effective partnerships between GoI, IUT and local governments at the state and municipal level for SUT development. Without strong partnerships, the PMO would not have been able to effectively assist cities and state governments in developing SUT service level benchmarks and SUT project investment plans. The strong partnership arrangements came from training sessions organized and provided by IUT personnel, giving officials from these city and state governments more confidence in IUT as a partner in developing SUT projects locally. This has resulted in IUT signing numerous long-term MoUs to provide SUT technical assistance and handholding to over 60 city and state governments throughout India. Tag: Local Governance Partnership Country Government Technical Support
13.	3.2.3 Feedback from M&E Activities Used for Adaptive Management Feedback for M&E activities was provided primarily through PIRs from 2013 to 2016 and a few PSC meeting minutes (only 2010 and 2014 PSC meeting minutes were available to the evaluation team) that provided details of activities for adaptively managing the Project. In evaluating the quality of feedback provided by these reports, the evaluators note the following: PIRs were not available for 2011 and 2012, likely due to the lack of an available PPM that would have provided useful indicators on which to report progress; Progress reporting from the 2013 to 2016 PIRs against indicators from the July 2013 PPM was satisfactory. However, as mentioned in Paras 24 to 26, the PPM should have contained indicators that would have reflected the desired outcome of the strengthening of IUT at the end of the Component 1A. An example of such an indicator could have included "number of financing commitments for the operation of IUT" or "% of staffing levels (as defined in business plan) reached at EOP". Tag: Business Model Human and Financial resources Monitoring and Evaluation Quality Assurance Capacity Building
14.	3.2.4 Project Finance Component 1A of SUTP had a GEF budget of USD 4.05 million that was disbursed over a 6-year duration and managed by UNDP India. Table 1 reveals: lower rates of disbursements during the entire 6-year duration of the Project from its original design period of 4 years. This lower rate of fund disbursement is an indication of the absorption ceiling for capacity building activities which were limited by the pace on which training workshops and seminars for over 1,000 persons (ranging from senior and mid-level government officials to transport professions and police officers) could be organized. Some of the training organizers mentioned that considerable efforts were made to identify and approve suitable candidates for SUT training, and that difficulties were experienced by the PMO to identify suitable candidates to serve as Master SUT trainers; deviations of original ProDoc Outcome expenditures including:oonly 50% and 60% of Component 1 (IUT strengthening) and Component 2 (provision of training) respectively were expended;oexpenditure of Component 3 (preparation of training materials and reference documents) were more than double the USD 250,000 originally planned;othese deviations were likely a result of project designers allocating Component 1A resources without the benefit of a PPM which would have helped in defining the extent of expenditures for a particular component; roughly 6.5% of the USD 4.05 million GEF grant remains unspent with the intention of the SUTP PMO to re-allocate these funds to implementation of either Components 1B or 2 or the operation of the SUTP PMO in 2018. Tag: Human and Financial resources Country Government Jobs and Livelihoods Capacity Building
15.	3.2.5 M&E Design at Entry and Implementation The M&E design is covered on pages 17-20 in the ProDoc for Component 1A of SUTP. The design thoroughly covers all M&E activities including: the Project inception phase; measurement of means for verification of project results, outputs and implementation; annual project reviews and project implementation reports (APRs/PIRs); independent evaluations that includes the Midterm Review as well as the Final Evaluation; and •audits and site visits. However, the PMO for Component 1A did not manage UNDP activities with a PPM between 2010 and 2013, making it difficult for the PMO to envision the outcomes to be achieved by the conclusion of Component 1A activities, and to properly plan project activities and budgetary allocations. The absence of a PPM was rectified in July 2013, though there were shortcomings of this PPM as described in Paras 24-28. As such, the M&E design is rated as moderatelyunsatisfactory. Tag: Monitoring and Evaluation
16.	Despite the shortcomings of the PPM that affected the M&E design, the M&E design was implemented as many of the indicators of the July 2013 PPM were output-based. There were some issues in the lack of monitoring of certain indicators as can be seen in Section 3.3 on Tables 3 to 7. Unless there were changes made on the PPM that were not reported to the evaluation team, there were a few inconsistencies in the PIR reporting including: The overall Project goal of “65 cities with MoUD approved Comprehensive Mobility Plan (CMP), which is appraised by IUT, for making investment in sustainable urban transport services”. In the PIRs, the actual target being reported on was “all project cities have an identifiable urban transport planning process in place” which is confusing in terms of what is being monitored: the number of cities or which project cities have CMPs; Output 1.4 whose target is either the “number of partnerships formed with other professional transport organizations and academia to carry out research activities” or “the number of policy research conducted by IUT for MoUD”. In the end, there is an appearance that the PIRs are reporting on the number of policy research projects is being completed; and Output 3.3 that has the indicators of “Topics/title for new standards approved by MoUD” and “Standards prepared and released by IUT after peer review”, both indicators with a target of 4. The PIRs do not report any progress on work of this output but appear to have replaced this output with this indicator: ”Number of validation workshops conducted by IUT to test the developed training manuals and toolkits” for which there is a target of “15”. If this indicator was changed, there should have been some documentation on why this indicator and output have been changed. Tag: Monitoring and Evaluation Quality Assurance
17.	3.2.6 Performance of Implementing and Executing Entities The performance of the implementing partner (formerly known as an Executing Agency) of Component 1A of the SUTP Project, the Ministry of Urban Development (MoUD), can be characterized as follows: Early stages of Component 1A implementation were marked by a period of familiarization for both MoUD and IUT in terms of strategic approaches to strengthening of IUT and in the structure of the training approaches for a wide range of SUT stakeholders in both government (central and local), the private sector and academia; MoUD were helpful in the identification and initiation of contacts with various local and state government personnel for training as Master trainers and working level trainees, as well as for policy formulation; and Senior MoUD officers developed close and collaborative working relationships with the PMO and key SUTP project staff as resources to resolve any particular issues that would serve as bottlenecks to progress of any activities related to training or institutional strengthening of IUT and the KMC. Overall performance of MoUD has been assessed as satisfactory. Tag: Partnership Institutional Strengthening National Institutions Private Sector Regional Institutions
18.	The performance of UNDP (the Implementing Agency) can be characterized as follows: Early stages of Component 1A implementation were marked mainly by organization of tenders for the preparation of the IUT business plan and the KMC, and the PMO working on the management of Component 1A activities without a PPM; Early stages of Component 1A implementation that placed considerable efforts into the organization of numerous training workshops that involved lengthy discussions and adaptive management with several state and municipal-level governments on the selection of personnel to participate (see Para 40); Prompt responses to implementing the MTR recommendation to improve the management of Component 1A with the formulation of a PPM in July 2013 which was quickly adopted by the PMO; Struggles with management of data collection activities of the KMC which did not appear to have involved a strategic approach. This is evidenced by a move to increase collection of transport-related data from 5 cities to over 40 cities, collecting data without knowing how it may be used in future SUT project designs, and not collecting or not setting up data collection of more relevant information such as transport energy-related data, GHG emissions and meteorological data; UNDP maintaining a good collaborative working relationship with MoUD, especially at the senior level, and identifying adaptive management measures in concert with MoUD personnel to ensure good and effective progress to works under Component 1A. Overall performance of UNDP on Component 1A of SUTP can be assessed as being satisfactory. Tag: Knowledge management Project and Programme management Data and Statistics
19.	3.3 Project Results 3.3.1Overall Results A summary of the achievements of Component 1A at the Project Objective level with evaluation ratings are provided on Table 3. Intended Outcome Overall Project Goal (Impact): Government capacity strengthened to plan, finance, implement, operate and manage climate-friendly and sustainable urban transport interventions at national, state and city levels Project Goal (Outcome): IUT is recognized by states and cities as a national urban transport knowledge centre The overall project-level goals were linked to strengthening IUT as a Center of Excellence in Sustainable Urban Transport in India that would serve as a resource to local governments and transport professionals on information vital to successful SUT projects, and the increasing number of local governments who are enabled to plan, design, finance, implement, operate and manage SUT projects and systems. Through these indicators, Component 1A would be able to achieve goals of strengthened local capacity for SUT development in a number of urban centers throughout India. With this strengthened capacity, personnel from these local governments would be better prepared for the World Bank-managed Component 2 on technical transport planning skills for cities and states and financing urban transport projects that contribute towards reducing emissions, and be enabled to plan, design and implement pilot SUT interventions under the World Bank-managed Component 2 of the overall SUTP Project. Tag: Emission Reduction Urban Local Governance Knowledge management Institutional Strengthening National Institutions Regional Institutions
20.	3.3.2 Component 1: Institutional Capacity Development focusing on Strengthening the Institute of Urban Transport To achieve Outcome 1 which is the “Institute of Urban Transport strengthened so as to provide substantial support to local governments in implementing the NUTP”, GEF resources under Component 1A were intended to support the growth of IUT from a business plan and its implementation towards an institute that supports several municipalities in their formulation of SUT plans using best international practices. Specific activities to achieve Outcome 1 included: preparation of a business plan for IUT; •assisting IUT in receiving certification to serve as an accreditation body on SUT; assisting IUT in the setup of an operational and commercially sustainable Knowledge Management Data Centre (KMC); facilitating IUT policy research for MoUD; facilitating collaboration of IUT with international institutions to build knowledge and expertise; building IUT capacity to appraise Comprehensive Mobility Plans (CMPs) that will serve as a basis for cities to seek support under JnNURM; providing IUT technical assistance and advisory support to states; preparing Service Level Benchmarks (SLBs) for cities; and appraising technical aspects of CMPs from cities on behalf of MoUD. A summary of the actual achievements of the Outcome 1 with evaluation ratings are provided on Table 4. Tag: Local Governance Human and Financial resources Knowledge management Urbanization Technical Support
21.	Despite the preparation of these policy research topics by IUT as a side venture, IUT neither had plans to become a leading research entity for sustainable transport in India nor did they have the necessary human resource to be an independent policy research unit. As such, there is a need for IUT to establish a network with other well-known environmental and transport institutions within India such as NIUA, CEPT, Indian Institute of Technology (IIT) Delhi, The Energy and Resources Institute (TERI), Indira Gandhi Institute of Development Research (IGIDR) and Central Institute of Road Transport (CIRT) as well as international institutions who would accelerate the pace of urban transport related research that can be applied in an Indian context for its numerous cities. To date, however, Component 1A did not have a focus with IUT on expanding its network of national transport institutions. For international institutions, 6 MOUs have been signed with ITDP, GIZ, EMBARQ, LTA, UITP and JTPA for closer collaboration for sustainable urban transport. Tag: Environment Policy Human and Financial resources Policies & Procedures National Institutions
22.	Component 1A has supported the growth of a knowledge management data centre (KMC) under IUT that currently houses urban transport related databases. The purpose of the KMC was to serve as a repository for traffic and transport data collected from various SUT studies including CMPs of selected cities throughout India. Additional work, however, is still required to improve the user-friendliness of the information and data. In addition, the database will need financial support to be sustained, albeit not on a commercial basis. The tendering process to set up the KMC commenced in 2013 and was not completed until 2014. Delays were due to lack of agreement between MoUD, IUT and UNDP on the process to be followed for appointing the agency. By late-2014, work had commenced on the development of the KMC by a consulting consortium of companies, UMTC and VBSoft. While the original plan was to collect transport-related data from approximately 5 CMPs on a pilot basis, the scope of work was expanded to include the collection of data from 49 cities. The consultants experienced considerable difficulties in obtaining usable data from the 49 cities; the data was not properly formatted and had gaps making entry into KMC’s databases difficult. Despite Component 1A achieving the delivery of this output, UMTC as well as VBSoft have been contracted through 2015 and 2016 to improve the quality of the databases and to enhance interaction of the data to develop different analytical scenarios for decision making. Tag: Urban Knowledge management Data and Statistics
23.	Mobilization of resources to upgrade the current database will require a more strategic approach, especially considering the difficulties in collecting urban transport-related data from 49 cities, which will only diffuse efforts of the consulting consortium to provide a user-friendly database. Continuation of the building of the urban transport related database after the completion of Component 1A of SUTP should focus on a few cities first followed by a scale-up. It is also important that such continued and long term efforts be coordinated by IUT under the oversight and ownership of MoUD. For its long term usefulness, the KMC business plan should look closely at having a holistic framework of data collection (that would include identification of specific data sets of information to collect) and focusing on the quality of the database that should undergo consultations amongst a consultative group of urban transport practitioners in India. While suggestions have been made by various stakeholders to collect revenue from the sale of KMC data and information, MoUD has declared that this information should be in the public domain implying that it would be free of charge, and forcing KMC to seek other means of self-sustaining financing sources. A service provision model of database where the capacity building component is integrated in KMC has also been contemplated as a potential finance model for long term sustainability of KMC; however, KMC financing will come primarily from MoUD. Tag: Resource mobilization Urban Oversight Ownership Capacity Building Technical Support Data and Statistics
24.	3.3.3 Component 2: Individual Capacity Development through Training of Trainers and Professionals involved in Urban Transport Activities under Component 2 were intended to have “Government officials, urban planners, practitioners receive training on various aspects of sustainable urban transport”. Project resources would be utilized to: provide training for a minimum of 100 trainers on various topics of sustainable urban transport; evaluate absorption of capacity building activities by participants; and conduct feedback surveys from participants on the usefulness of capacity building activities. A summary of the actual achievements of the Component 2 with evaluation ratings are provided on Table 5.71. Training activities are covered under the SUTP website32 including the names of officers who have participated in the numerous SUTP training workshops since 2011: A total of 10 training workshops on various SUT topics have been held since 2011. This includes topics such as metro rail systems (2 workshops), intelligent transport systems or ITS (3 workshops), comprehensive mobility plans or CMPs (1 workshop), urban transport planning (1 workshop) and city bus services (3 workshops); Four thematic training programmes for training of master trainers (ToT) was conducted for 79 master trainers, short of the target of 100 master trainers by the EOP. According to the SUTP PMO, they experienced problems finding suitable candidates to be master trainers; 23 training workshops were conducted by master trainers at the sub-national level, 17 short of the target of 40. Details of the reasons for shortfall are provided in Para 75; The number of transport professionals trained at sub-national levels was 1,021 up to March 2016. Another 500 will be trained upon approval from the Steering Committee and funds from MoUD. Tag: Urban Jobs and Livelihoods Capacity Building
25.	In the preparation of training material, IUT has reportedly used the assistance of international agencies with national institutions involved only for the process of review. While this approach is understandable at the nascent stages of a national sustainable urban transport programme, this process should reverse itself in future; with the completion of more Indian-based urban transport projects to learn from, national and local institutions will be able to prepare training material with the process of review being undertaken by international experts and institutions. A review of training materials also revealed that there was a lack of material on technological aspects of construction and management. This is especially important since implementation of sustainable transport projects will likely challenge local officers and project managers in how to construct infrastructure (unique to their jurisdictions) in compliance with local codes, with the likelihood that its development may need unique construction protocols and building materials (such as barriers to separate pedestrians from cycle pathways, signaling along a BRT corridors or elevated BRT platforms). Tag: Rural Jobs and Livelihoods Capacity Building National Institutions Regional Institutions
26.	3.3.4 Component 3: Preparation of Manual, Toolkits and Standards on various aspects of Urban Transport 79 Activities to achieve Outcome 3 were intended for “manuals, toolkits and standards to be prepared to serve as reference documents, guides to develop and implement of sustainable urban transport”. Project resources would be utilized to: Assist IUT In the preparation of 10 Manuals on Sustainable Urban Transport; Assist IUT In the preparation and validation of 11 toolkits on Sustainable Urban Transport; Assist IUT In the preparation of 4 peer-reviewed standards for MoUD approval on Sustainable Urban Transport. A summary of the actual achievements of the Component 3 with evaluation ratings are provided on Table 6. Tag: Communication Knowledge management
27.	3.3.5 Component 4: Promotion, awareness raising and dissemination of information to expand and enhance the impacts of GEF SUTP Activities to achieve Outcome 4 were intended to “increase awareness of Sustainable Urban Transport interventions among city government officials and transport sector professionals”. Project resources would be utilized to: Publish 20 newsletters on Sustainable Urban Transport; Develop and operate an SUTP interactive web portal; Organize 4 annual events to promote sustainable urban transport for the purposes of sharing of experience and knowledge. A summary of the actual achievements of the Component 4 with evaluation ratings are provided on Table 7. Tag: Communication Knowledge management Awareness raising
28.	3.3.6 Relevance Component 1A of SUTP is relevant to the National Urban Transport Policy that has been adopted by the Government of India. More importantly, the NUTP also mentions the necessity of a strong Institute of Urban Transport implement this policy. Tag: Relevance
29.	3.3.7Effectiveness and Efficiency The effectiveness of Component 1A of SUTP in improving the capacity of IUT to initiate and develop SUT projects in project cities in India has been satisfactory. However, the effectiveness of Component 1A could have been improved had it not only focused on IUT to be the agent for change in building local SUT capacities, but focusing on other transport-related institutions to increase the impact and replication of training activities. Furthermore, IUT has not yet been developed into an independent body, but only as a smaller organization under MoUD which is supposed to provide SUT guidance on behalf of MoUD to all of India’s cities. Efficiency of Component 1A in delivering its goals for under USD 4.05 million has been satisfactory. Cost effectiveness is discussed in further detail in Paras 46 to 48. Tag: Effectiveness Efficiency Sustainability
30.	3.3.8 Country Ownership and Drivenness Strong indicators of the drivenness of the Government of India to undertaking programs to develop sustainable urban transport initiatives within its cities are provided in 12th Five Year Plan that identifies the “integration” of transport governance, policy development and implementation. The Plan advocates that transport systems at the macro level are developed around sustainable freight and passenger transport networks. At a micro level, the Plan requires higher levels of accessibility between public transport modes to ensure that a new network is user friendly and socially inclusive. With the expected costs of this “integration” in the order of USD 1 trillion, the GoI through a National Transport Policy Development Committee (NTPDC) advised the Planning Commission that a “quantum jump” is required to develop the requisite expertise to ensure that such a high level of investment is spent wisely. This would include new institutions with authority over all aspects of transport policy and management, staffed with a new generation of planners, trained to modernize urban and inter-city transport with skills that require concepts beyond traditional development of only infrastructure. This new generation of planners will be supported by stronger research and statistical institutions that are multidisciplinary in nature. The GoI under its National Urban Transport Policy (NUTP) of 2006 is undertaking programs to develop sustainable urban transport initiatives within its cities. The NUTP envisions urban transport systems that focus on personal mobility, urban livability, and urban growth that is oriented towards local geography and not transport infrastructure per se. The impact of the NUTP, however, would only be limited if the capacities and knowledge base of its implementers are not strengthened. This endorses the vision outlined in NUTP for IUT to serve as a premier national research and advisory institution to strengthen India’s knowledge base for urban transport sector signalling the GoI’s strong drivenness on modernizing India’s urban transport sector. Tag: Integration Knowledge management Policies & Procedures Infrastructure
31.	As such, the vision of the Government of India to seek international assistance to build its capacities for implementing the NUTP is expressed in the design of the SUTP Project. For the UNDP-implemented Component 1A of the SUTP Project, GoI through its Ministry of Urban Development, has sought to strengthen the Institute of Urban Transport to undertake independent planning, development, operation, education, research and development activities as well as information dissemination activities (i.e. lectures, seminars, workshops and conferences) that would broaden the knowledge of sustainable urban transport over a wide spectrum of stakeholders. To this end, the strengthening of IUT is within the objectives enshrined within the NUTP to establish the necessary knowledge management systems that would provide the data and information necessary for planning sustainable urban transport initiatives throughout India. Tag: Communication Knowledge management Institutional Strengthening
32.	3.3.9 Mainstreaming Component 1A of SUTP has successfully mainstreamed with the UNDAF for India (2013 to 2017)39. This includes Component 1A activities that work towards the UNDAF’s Outcome 6: Sustainable Development, specifically India’s aim to reduce GHG emissions, and support its mission on enhanced energy efficiency, in this case on the urban transport sector. As such, the contribution of Component 1A of SUTP includes the enabling and subsequent strengthening of one of India’s largest ministries, the Ministry of Urban Development to introduce and implement measures to decrease the energy intensity of urban mobility in India’s cities at an accelerated rate in comparison with the business as usual scenario. Moreover, Component 1A has contributed towards the strengthening of the Institute of Urban Transport of MoUD as a Centre of Excellence for urban transport in India, as a means to ensure the dissemination of best practices in sustainable urban transport to numerous municipal and state personnel in India’s cities. Tag: Emission Reduction Energy Environment Policy
33.	3.3.10 Sustainability of Project Outcomes In assessing sustainability of Component 1A of SUTP, the evaluators asked “how likely will Component 1A outcomes be sustained beyond SUTP termination?” Sustainability of these objectives was evaluated in the dimensions of financial resources, socio-political risks, institutional framework and governance, and environmental factors, using a simple ranking scheme: •4 = Likely (L): negligible risks to sustainability; •3 = Moderately Likely (ML): moderate risks to sustainability; •2 = Moderately Unlikely (MU): significant risks to sustainability; and •1 = Unlikely (U): severe risks to sustainability; and •U/A = unable to assess. Overall rating is equivalent to the lowest sustainability ranking score of the 4 dimensions. Tag: Sustainability Local Governance Human and Financial resources Sustainability
34.	3.3.11Impacts Component 1A of the SUTP Project has made a significant impact on the strengthening of IUT as a focal and quasi-independent organization that represents the GoI on SUT issues including the appraisal of SUT investments on behalf of the GoI. This in turn has led to the scale up and development of SUT projects in pilot cities under the SUTP Project in India. In addition, a strengthened IUT has provided the GoI with an entity that has provided handholding of city and state governments in the design of SUT systems in a number of cities participating under SUTP throughout India to an advanced design or investment stage. Tag: Impact Sustainability Capacity Building

Recommendations
1	For projects that involve two implementing agencies, assurances should be made that each implementing agency has its own project planning matrix for the monitoring and evaluation of its own activities.
2	Training materials for government personnel as well as to urban transport professionals needs to become more comprehensive and broad-based. This would include topics for training such as energy savings and GHG emission reductions from Sustainable Urban Transport (SUT) projects, construction techniques, construction management, and the maintenance of SUT-related infrastructure
3	Institute of Urban Transport (IUT) needs to prepare a strong and updated business plan that proposes collaborative mechanisms with and complements the work of other existing institutions involved with urban transport in India.

1. Recommendation:

For projects that involve two implementing agencies, assurances should be made that each implementing agency has its own project planning matrix for the monitoring and evaluation of its own activities.

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

During the preparation of the projedct document, a seperate project planning matrix was developed for both UNDP and World Bank components. However, the two matrices did not complement each for the achievement of the overall project objective.

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
The recommendation to be shared with the Ministry of Urban Development (MoUD) for future project / interventions [Added: 2017/11/14]	UNDP	2017/10	Completed

2. Recommendation:

Training materials for government personnel as well as to urban transport professionals needs to become more comprehensive and broad-based. This would include topics for training such as energy savings and GHG emission reductions from Sustainable Urban Transport (SUT) projects, construction techniques, construction management, and the maintenance of SUT-related infrastructure

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

Agreed and accepted

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
New topics for training related to energy savings, GHG-emissions etc., to be identified by Institute of Urban Transport (IUT) and modules to be developed for planned trainings in year 2018 [Added: 2017/11/14] [Last Updated: 2018/04/09]	UNDP	2018/02	Completed

3. Recommendation:

Institute of Urban Transport (IUT) needs to prepare a strong and updated business plan that proposes collaborative mechanisms with and complements the work of other existing institutions involved with urban transport in India.

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

Agreed and accepted

Key Actions:

Key Action	Responsible	DueDate	Status	Comments	Documents
A comprehensive business plan to be developed by IUT involving existing institutions like Centre for Environment Planning & Technology (CEPT), Indian Institute of Management, IIM-Ahmedabad and National Institute of Urban Affairs (NIUA) [Added: 2017/11/14] [Last Updated: 2018/04/09]	Institute of Urban Transport (IUT)	2018/03	No Longer Applicable	[Justification: The Ministry has refused preparing such a plan at this point in time.]

Terminal evaluation - Sustainable Urban Transport

Share

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

Key Actions:

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

Key Actions:

Management Response: [Added: 2017/11/14] [Last Updated: 2021/01/15]

Key Actions:

Evaluations by Unit

Latest Evaluations

Evaluation Snap Shot

Contact us