Support to Full scale Agricultural Census in Armenia Final Evaluation

Report Cover Image
Evaluation Plan:
2016-2020, Armenia
Evaluation Type:
Final Project
Planned End Date:
12/2017
Completion Date:
04/2017
Status:
Completed
Management Response:
Yes
Evaluation Budget(US $):
29,000

Share

Document Type Language Size Status Downloads
Download document TOR_Project Final Evaluation_Agricultural Census.docx tor English 50.61 KB Posted 812
Download document Agicultural Census Final Evaluation 2017.pdf report English 1943.64 KB Posted 801
Download document Signed Management Response Census.pdf related-document English 2059.95 KB Posted 666
Download document EvalQuestions for UNDPAgCensus.docx related-document English 28.89 KB Posted 768
Title Support to Full scale Agricultural Census in Armenia Final Evaluation
Atlas Project Number: 00081908
Evaluation Plan: 2016-2020, Armenia
Evaluation Type: Final Project
Status: Completed
Completion Date: 04/2017
Planned End Date: 12/2017
Management Response: Yes
Focus Area:
  • 1. Poverty and MDG
  • 2. Others
Corporate Outcome and Output (UNDP Strategic Plan 2014-2017)
  • 1. Output 1.1. National and sub-national systems and institutions enabled to achieve structural transformation of productive capacities that are sustainable and employment - and livelihoods- intensive
Evaluation Budget(US $): 29,000
Source of Funding: UNDP
Evaluation Expenditure(US $): 25,000
Joint Programme: No
Joint Evaluation: No
Evaluation Team members:
Name Title Email Nationality
GEF Evaluation: No
Key Stakeholders:
Countries: ARMENIA
Lessons
1.

·       The project’s design in two phases contributed to smooth execution and the achievement of the intended Outputs and Targets. The first phase was a test phase and allowed the NSS, project consultants, and other development partners to identify weak points and improve the design and implementation of the full census. The PES of the second phase allowed to verify and validate data and help to format the data in a user friendly way.

·       Both phases contributed to practical on-the-job experience while also adding more formal training through workshops, conferences and country visits.

·       Promotion of a first time agricultural census (or any census) to the public, especially interviewees, is essential especially in a culture that is not used to such information gathering and sharing. Both phases of the project contributed to the public campaign with a first phase finding indicating more emphasis was needed on the public awareness campaign and a second phase finding being the more public awareness was needed for future efforts.

·       A project benefits from a set of discrete activities with related indicative activities and clear measurement indicators. The project has such discrete activities and results were measured on a periodic basis that has led to appropriate activity and indicative activity tracking.

·       Professional expertise was provided by FAO on a timely basis. The consultants were qualified, worked well with counterparts, and provided real world regional experience that counterparts could understand and relate to.

·       A skeletal part-time management/staff structure did not take away form project implementation or results. In fact, the time sharing with related projects and situating the project with a cluster of similar projects probably contributed to efficient and effective use of resources.


2.

·       Activities planned with the participation of stakeholders and participating development partners can achieve a large, complex endeavor if coordination and collaboration is present with a committed and strong main counterpart (NSS in this case).

·       A clear time plan with timed inputs can help in the execution of such a step-by-step intervention. Constant coordination meetings are needed involving all main stakeholders.

·       While clear activities and indicative activities are important and need to be agreed upon by the main beneficiary (NSS), the beneficiary must be committed and actively participate along the way and, in fact, drive the intervention given the many players involved.

·       Feedback from the main beneficiary and other participants is essential and needs to fit into future steps or activities.

·       Publication of the data was important so that the government, academia, private business, NGOs and others can clearly see the results. The promotions workshop (December 2016) contributed to this effort, as does the NSS web site access.

·       While collection, organization and publication of the data is essential, there appears to be the need to concentrate more on the use of the data though there is anecdotal evidence (3 months after data publication and dissemination) that the data is being used.

·       Overall project satisfaction is high according the NSS and other stakeholders. There is an interest on follow-up work to extend the progress made to date (further elaborated in Section IV).


3.

First, the in-country interviews have shown that the stakeholders have appreciated the project intervention to support the agricultural census. The main counterpart, NSS, had developed a good working relationship with project management and facilitated and used the expertise provided by the FAO consultants. The activities were delivered in a timely manner and supported the NSS and the census at critical milestones- both by creating key milestones and accomplishing them. Other stakeholders, namely the MoA and MoTA also rated the project as positive and are starting to see what the census data can provide to help in sector research and policy making. There were also positive comments regarding the support to the census and the actual census results by representatives of academia (National Agrarian University), the private sector and NGOs.

 

Second, the success of the interventions can be measured by the actual use of the data. While the data was only fully disclosed and made available in December 2016, there is anecdotal evidence that the data resulting from the census is being used. The data is mostly being used as information and review purposes currently but will most likely eventually feed into the policy and other decision making processes. There are also evident opportunities to continue to strengthen data collection and verification as well as develop capacities to better use the data for policy analysis and preparation.


4.

There were no apparent gaps or weakness in the project design, implementation or management.

 

The project design was prepared with the input of UNDP and FAO with consultations of the NSS and others, including development partners who had various roles also in conducting the agricultural census. The key factors to determine the strengths or weaknesses of the project were: 1) the ability of the development partners to work together to complete an agricultural census within the expected timeline; 2) the collection of the appropriate data, and the publication of the data; and 3) having a cooperative and committed counterpart with committed management and staff. The inputs delivered by the project in terms of hiring census activity personnel, providing reliable and experienced technical experts, and either conducting or facilitating capacity development contributed to the strengths. Project management and the project’s sponsoring entities (UNDP, FAO and ADA) were flexible enough to allow the project to proceed through the two phases by backstopping and make slight adjustments as needed.

 

Project management, as mentioned previously, was lean but effective. One may think that full time project management and staff were needed to add value but the part-time inputs as well as cost sharing with a similar projects most likely helped project implementation.


5.

·       A multi donor activity with a strong national partner and appropriate government support and legal framework can work for such a complex undertaking as a first agricultural census (and it’s pilot phase). Ongoing meetings and communications are needed to help collaboration and the delivery of inputs and results.

·        A large and complex project, program or other activity (such as a census), especially for a first-time endeavor, benefits from a pilot phase to work out approaches and methodologies, develop capacity and build relationships for a full-scale activity.

·       The resulting questionnaire proved to have the appropriate design. However, some adjustments were necessary to include, but not limited to: classification of the total areas of the holding according to land use and land tenure and adding a category for household members under 15 years. Consultations were undertaken with development partners as well as potential data users.

·       The definition of “agricultural holding” was problematic. In Armenia, besides holdings that correspond strictly to FAO definition of “agricultural holding” (operate agricultural land and hold livestock), there are also holdings, especially from the household sector, that hold, but do not operate or rent out their agricultural land. Thus, the enumeration only of agricultural holdings, according to the FAO definition, could have prevented the full census from obtaining comprehensive information on total agricultural land of the country.

·       For data entry and processing of census information, NSS designed the IT solution, based on CSPro, which was also used for processing of 2001 and 2011 population census data in Armenia. Some concerns were expressed regarding the possibility to use only CSPro to ensure the necessary processing of full scale agricultural census data, which (in comparison with population census data) need to support much more sophisticated arithmetical checking rules and tabulations.

·       The lists of pilot census units comprised only the information on the availability of agricultural land and livestock, without quantity information (land area and livestock number), which did not allow the elaboration of a complete census frame and timely checking and analysis of the discrepancies between census and administrative data.

·       There were discrepancies on total agricultural land at the community level between census results and cadastral information. Additional attention is needed to aligning statistics between the different GoA statistical agencies.

·       The monitoring procedure developed for the pilot census did not allow NSS to receive timely and comprehensive information on the pace of the fieldwork (with the exception of the number of enumerated units). Important information, such as the land area and livestock by main kinds of animals of enumerated agricultural holdings, were available at NSS only when census materials were received from Territorial Statistical Offices several weeks after the census end date. This delay did not allow NSS to assess and make necessary timely decisions to ensure the completeness of census enumeration.

·       NSS and census field personnel encountered difficulties caused by the lack of agricultural land property ownership certificates in some of communities. There were issues with the enumeration of absent and non-resident landholders with the lands located outside the locality of their residence. And, there were some cases when people were reluctant to give the census data, especially on livestock (e.g. the beneficiaries of state social benefits for people with low income).


6.
  • To support a complex multi-year one-time exercise into phases helps to build capacity over time through practical experience, brings awareness to a full cycle of activities rather than a one-off activity, and provides checks and balances to correct instruments and processes along the way. It also allows buy-in to develop more naturally with stakeholders and allows for more public awareness and education over time.
  • Multi-development partner collaboration is essential to address such large one-time efforts. While such an approach, with the UNDP/FAO support a major component, was successful for pre, actual and post census, the same collaboration will be needed to help advance from the actual census to further strengthen agricultural statistic use, updates (including post census surveys), and the next cycle of census activity.
  • It is important to emphasize that an agricultural census is not a one-time activity but part of a cycle of activity to build and maintain the agricultural statistics database that users can access and use for practical purposes.
  • Transfer of capabilities to government is happening and could be enhanced as more experience on data reporting and manipulation is obtained.
  • Linkages to an international and/or regional institution body, such as the FAO, are essential to help benefit from relevant other country experience, especially within the region, and obtain and adhere to international standards. The adoption of international standards, such as was done with the activities surrounding the census, helps to legitimatize the census activity and resulting statistics.
  • The use of consultants from within the region and with a similar development experience benefits project and, in this case, census, execution. The mission to the Czech Republic had the same result to learn from peers and exchange experiences and capacities.
  • Given the change in the source, compilation, verification and organization of the census data, education on data use is proving to be a key need. Education appears urgent at two levels- existing users in government, private sector and NGO community, and at the technical or university education level. The former is short-term for current information needs and the second is for medium and long-term sector development.
  • With the introduction of a new set of statistics (for any new statistic gathering exercise, not only for the subject agricultural census), there will be discrepancies between statistics previously used or gathered by others and those of the new set of statistics gathered according to international standards. These discrepancies must be worked out so there is one set of reliable date- or ‘one set of music’ that policy makers, researchers and others are reading from. This will take cooperation between the NSS and other statistical producers.
  • When carefully thought out, project management can be organized so project activities can be implemented in an efficient manner to contribute to related efforts. In the subject project effort, project management and staff had shared functions and salary costs with other projects that helped to conduct the subject project in an efficient manner, build relationships, and use common resources as well as obtain shared goals and objectives.

Findings
1.

The project was instrumental in the planning, conduct and reporting on Armenia’s first agricultural census. The design of each phase took into consideration the interests of the main partner, the NSS, and each phase was integrated with the work of other donors so that a comprehensive and concerted effort took place in a timely manner and within costs.

 

The design of the first phase allowed for the pilot census to take place and prepare the groundwork for the full census. Several issues were addressed that resulted from the pilot phase and the NSS and other partners gained experience that led to a successful census. Phase 2 activities surrounded the activities of the PES and then the documentation to supply the data provided by the census in a more user-friendly manner. Both phases addressed capacity development- mainly with the NSS and other participants gaining on-the-job experience as well as other training. The Phase 2 missions to Austria and the Czech Republic have contributed to improving data and its use, some of which the NSS and MoA are following-up on.

 

The project design, as identified in the Results and Resources Framework for each phase, had very clear outputs, targets, activities, indicative activities and measurement of those activities as shown in Annexes D and E. The measurements were mostly numerical and measured on a periodic basis. Where indicative activities were not achieved, reasons were provided with the different stakeholders informed prior to any changes and the acceptance of the main counterpart, other development partners involved and project the project sponsor. There were no cost overruns.

 

Project administration was lean and efficient. The two project positions were part-time given time share arrangements with other projects. The arrangement did not appear to impede project execution. In fact, given the placement of the project within a cluster of similar efforts under ENPARD, the collaboration may have contributed to project efficiency and results.


2.

All project targets for each phase were accomplished. There was a slight delay in the pilot census (from October to November 2013) that was outside of the project’s control. There was a “trade” of some indicative activities with the World Bank because of World Bank funding flow issues but the result did not affect the support to the pilot census or other intended targets. There was an over achievement of the schematic mapping as intended at the request of the main stakeholder. All changes were preapproved.

 

There were several comments that the census could have been achieved in a more condensed timeframe. However, given the structure of the phased approached, the multi-development partners support and involvement, and a first time exercise, it seems that more of an emphasis was placed on the quality of “doing it right” than trying to do it too fast. Also, there seems to have been the issue of increasing public awareness so that there was adequate and responsive support and participation- not an unusual occurrence for a first time agricultural census.

 

The project was effective in terms of contributing to the completion of the overall census. The verification, validation and organization of the resulting data through the PES and other Phase 2 activities contributed to the presentation of data in a more user-friendly way, as exemplified by the summary publication with all main data tables and web access to the main data tables. There is also anecdotal evidence of the NSS providing data on request.

 

The main beneficiary, the NSS, has expressed satisfaction with the project’s participation in the census activity. They believe that the project and the overall census was a success. The MoA, MoTA have also expressed satisfaction along with several representatives of development partners, academia, and private sector and NGO data users. There is anecdotal evidence that these entities are using the data for a variety of purposes to include research, marketing and other studies, comparisons to other countries, and policymaking. There is also the understanding that there are future steps to take to improve upon the data gathered and in its use and presentation.


3.

·       Close cooperation between project administration, consultants and stakeholders (mostly NSS in this case) will lead to the satisfactory implementation of activities as well as building relationships and capacity.

·       Keeping fixated on the project’s activities contribute to meeting targets and fulfilling outputs for such a large and complex endeavor. The stakeholders knew what the project was to deliver and the project appeared to deliver what was expected.

·       Tracking of what was done on a timely basis adds to the ability to communicate to others what was accomplished, changed, or not being accomplished as planned.

·       Collaboration between development partners keeps all parties up-to-date on what is happening. There seemed to be professional understanding and expertise so the ‘pieces of the puzzle’ fit together.

·       Given the success, stakeholders are very receptive to future interventions by UNDP and FAO to further advance upon the progress made to date. Several areas of possible support were mentioned and further discussed in Section IV.


4.

·       Project design involving the main stakeholders and development partners helps to identify discrete interventions and commitments. It also helps with the identification of clearly defined activities, targets and outputs.

·       A technical and capacity development intervention can add more value if linked to an actual activity, such as an agricultural census, so real and practical experience can be gained.

·       Staying in step with the main stakeholder is important so that inputs don’t get ahead or behind what is planned or what the main stakeholder can absorb.

·       Periodic meetings of main participants contribute to cooperation and communication, especially for a large and complex effort like a first-time agricultural census.

·       A step-by-step progression of activities such as pilot census – census – PES – data formatting and documentation interspersed with capacity development and process development helps to provide support in a timely manner and accomplishing pieces of the larger effort (the completed census).

·       Project management and staff provision should be tailored to the project intervention. The administrative framework was consistent throughout the two phases and provided adequate planning, management and backstopping support.


5.

Most Phase 1 activities were implemented on a timely basis and within budget as originally planned in the project document. The implementation did confront some complexity to include the realization that the pilot agricultural census was the first such exercise in Armenia. Second, the pilot census effort, and its surrounding activities for preparation, data entry, compilation and analysis, and capacity development, involved mainly the NSS from the government and also a variety of development partners besides those involved in the project. Extensive coordination was needed with the different participants to execute the pilot census in a timely and cost effective manner. While there was a slight (1 month delay) in the actual conduct of the pilot census, it did not affect the conduct or results of the pilot census. The coordination of the census was also exemplified by the cooperation of the UNDP/FAO with the World Bank to swap inputs given fund flow issues but resulted in the same indicative activities being implemented at the same cost.

 

The pilot census did fulfill its role in providing a test of the census questionnaire and its instructions, methodology, and capacity that fed into the design and implementation of the full agricultural census. As shown in the 2-day February 2014 workshop, sponsored by the project, the pilot census findings and results were reviewed and reported on to make further improvements.


6.

All project activities were accomplished within the timeframe and budget planned in the project document and in line with full agricultural census activities and the resulting activities of the NSS and other development partners. The execution of Phase 2 continued and/or responded to some of the findings of Phase 1, namely, the coordination and collaboration of the government (mainly NSS) and a variety of development partners to execute a complex first time exercise such as an agricultural and applying the lessons learned, experience and adjustments from the pilot census of Phase 1.

 

Phase 2 also showed the importance of a project intervention to identify discrete activities to contribute to a larger exercise in a practical and realistic manner. In the case of Phase 2, the conduct of the PES was an essential step of the agricultural census cycle and helped to close the loop on data entry, quality control and verification, and data preparation for publication. Second, the assistance to publicize and make the census data more “user friendly” and widely available through the publication (in 3 languages) of the main statistical document and web site availability has contributed to the understanding of the purpose of the census exercise and, more importantly, the actual use of the data. Third, the capacity development efforts have proven instrumental in two ways: 1) there has been an obvious gain of practical on-the-job experience at NSS that will contribute to post census surveys and the next census cycle; and 2) the out-of-country missions to Austria and the Czech Republic helped to show the main stakeholders, NSS and MoA, the experience of other countries and provided information on future steps needed to help sustain the progress made to date.

 

There is now a more advanced discussion on the use of agricultural statistics for research, policy making and reporting. While it is too early to fully assess this impact, there is anecdotal evidence that the census data is being used by a variety of public, private and non-governmental entities.

 

Last, the end of the ‘census cycle’ shows that NSS and the GoA can undertake such a census according to international standards. However, there are necessary next steps to help sustain the experience gained to date.


Recommendations
1

The further interventions should be focused on the consistency of data

2

The future support should be focused on further improving the use of the Census data.

3

The further interventions should be focused on census cycle of activity.

1. Recommendation:

The further interventions should be focused on the consistency of data

Management Response: [Added: 2017/09/27]

This recommendation is in line with the priorities of the National Statistical Service of Armenia to further enhance the rural and agricultural statistics in Armenia. The implementation of the first-ever Agricultural Census ensured the availability of reliable, comprehensive, internationally comparable and up-to-date information on agriculture and the rural sector in Armenia. In order to keep the statistics up-to-date and further ensure the consistency of data, the clear understanding of inter census surveys and other related activities should be developed. Also, the NSS should be provided with the modern equipment ensuring the effective storage, back-up, protection and maintenance of Agricultural Census data.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
1.1. Organize workshops for central and regional NSS units to discuss the best practices for development of agricultural and rural statistics, establishing farm register, conducting surveys in between the Agricultural Census and other effective methods and techniques to be used by NSS in future years to keep the data up-to-date and ensure data consistency. The effective methods for data verification will also be discussed.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 25 August 2017 History
1.2. Procure electronic data storages for NSS to ensure proper storing, back-up, protection and maintenance of Agricultural Census data.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 31 August 2017 History
2. Recommendation:

The future support should be focused on further improving the use of the Census data.

Management Response: [Added: 2017/09/27]

This is a valid recommendation. There is a significant demand by government, NGOs, media, academia for the agricultural census data and it is already being used by a variety of end-users. But since the Agricultural Census has been implemented just recently, there is still the need to further improve the use of data by experts, non-expert users and general public. To ensure the improved and more user-friendly access to the agricultural census data, the online interactive statistical map should be developed to search the Census data by category, regions, communities. Also, to make the data more user-friendly and visual for wider audience, the leaflets on various topics of Census findings should be developed and disseminated to the large user community. The details on these and other possible actions are presented in Key Actions section below.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
2.1. Develop online interactive statistical map allowing to search the Census data by category, regions and communities to ensure the improved and more user-friendly access to the agricultural census data. This will contribute to the increased usage of Census data by experts, non-expert users and general public as well as to advance the use of data for policy making purposes.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 31 August 2017 History
2.2. Develop, translate, publish and disseminate leaflets on various topics of Census findings to the large user community. Where possible the gender segregated data will be used. The leaflets will help to make the data more user-friendly and visual for wider audience and will enhance the use of the census data for policy making purposes. The leaflets will be distributed during several thematic meetings to be held with the representatives of Ministries, educational institutions, students and researchers, as well as NGO representatives of agriculture and rural development sectors.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 25 August 2017 History
2.3. Discuss with the NSS management the NSS website design improvements. The improved website will allow to enhance the use of statistical data, including the agricultural and rural data.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 31 July 2017 History
2.4. Organize workshops for central and regional NSS units to discuss the best practices of effective use of Census data.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 25 August2017 History
3. Recommendation:

The further interventions should be focused on census cycle of activity.

Management Response: [Added: 2017/09/27]

This is a valid recommendation and in line with the NSS priorities to further enhance the rural and agricultural statistics in Armenia. The agricultural census is not viewed by NSS as a one-time event, instead the census is viewed as part of a cycle of activity. In order to keep the agricultural statistics up-to-date, accurate and relevant as well as to put the emphasis on census cycle of activity, the clear understanding of post census surveys, farm registry and other related activities should be developed. The cycle will be formulated into a timeline of activity to identify post-census surveys to keep data updated, and other activities so that the NSS and others are clear what needs to be done in future years.

Key Actions:

Key Action Responsible DueDate Status Comments Documents
3.1. Organize workshops for central and regional NSS units to discuss the best practices for keeping the agricultural statistics up-to-date, accurate and relevant as well as for the emphasizing the census cycle of activity. The topics will include establishing farm register, conducting post census surveys and other effective methods and techniques. The cycle will also be formulated into a timeline of activity to identify post-census surveys and other activities so that the NSS and others are clear what needs to be done in future years.
[Added: 2017/09/27] [Last Updated: 2017/10/14]
UNDP Armenia, NSS 2018/08 Completed 25 Aug 2017 History

Latest Evaluations

Contact us

1 UN Plaza
DC1-20th Floor
New York, NY 10017
Tel. +1 646 781 4200
Fax. +1 646 781 4213
erc.support@undp.org