“Big Data for Development: Opportunities & Challenges”

Published by Global Pulse, 29 May 2012

Abstract: “Innovations in technology and greater affordability of digital devices have presided over  today’s Age of Big Data, an umbrella term for the explosion in the quantity and diversity of high frequency digital data. These data hold the potential—as yet largely untapped— to allow decision makers to track development progress, improve social protection, and understand where existing policies and programmes require adjustment.  Turning Big Data—call logs, mobile-banking transactions, online user-generated content such as blog posts and Tweets, online searches, satellite images, etc.—into actionable information requires using computational techniques to unveil trends and patterns within and between these extremely large socioeconomic datasets. New insights gleaned from such data mining should complement official statistics, survey data, and information generated by Early Warning Systems, adding depth and nuances on human behaviours  and experiences—and doing so in real time, thereby narrowing both information and  time gaps. With the promise come questions about the analytical value and thus policy relevance of  this data—including concerns over the relevance of the data in developing country contexts, its representativeness, its reliability—as well as the overarching privacy issues of utilising personal data. This paper does not offer a grand theory of technology-driven social change in the Big Data era. Rather it aims to delineate the main concerns and challenges raised by “Big Data for Development” as concretely and openly as possible, and to suggest ways to address at least a few aspects of each.”

“It is important to recognise that Big Data and real-time analytics are no modern panacea for age-old development challenges.  That said, the diffusion of data science to the realm of international development nevertheless constitutes a genuine opportunity to bring powerful new tools to the fight against poverty, hunger and disease.”

“The paper is structured to foster dialogue around some of the following issues:

• What types of new, digital data sources are potentially useful to the field of international development?
• What kind of analytical tools, methodologies for analyzing Big Data have already been tried and tested by academia and the private sector, which could have utility for the public sector?
• What challenges are posed by the potential of using digital data sources (Big Data) in development work?
• What are some specific applications of Big Data in the field of global development?
• How can we chart a way forward?”

Click here to download the PDF:

Read about Global Pulse. “Global Pulse is an innovation initiative launched by the Executive Office of the United Nations Secretary-General, in response to the need for more timely information to track and monitor the impacts of global and local socio-economic crises. The Global Pulse initiative is exploring how new, digital data sources and real-time analytics technologies can help policymakers understand human well-being and emerging vulnerabilities in real-time, in order to better protect populations from shocks.”

See also: World Bank Project Performance Ratings. “IEG independently validates all completion reports that the World Bank prepares for its projects (known as Implementation Completion Reports, or ICRs).  For a subset of completed projects (target coverage is 25%), IEG performs a more in-depth project evaluation that includes extensive primary research and field work.  The corresponding ICR Reviews and Project Performance Assessment Reports (PPARs), codify IEG’s assessments using Likert-scale project performance indicators.  The World Bank Project Performance Ratings database is the collection of more than 8000 project assessments covering about 6000 completed projects, since the unit was originally established in 1967.  It is the longest-running development project performance data collection of its kind.”(1981-2010)

Rick Davies comment: There is a great opportunity here for a data mining analysis to find decision rules that best predict successful projects [Caveat: GIVEN THE FIELDS AVAILABLE IN THIS DATA SET] ”

See also: Good countries or good projects ? macro and micro correlates of World Bank project performance. Author: Denizer, Cevdet; Kaufmann, Daniel; Kraay, Aart; 2011/05/01, Policy Research working paper ; no. WPS 5646 . Summary:”The authors use data from more than 6,000 World Bank projects evaluated between 1983 and 2009 to investigate macro and micro correlates of project outcomes. They find that country-level “macro” measures of the quality of policies and institutions are very strongly correlated with project outcomes, confirming the importance of country-level performance for the effective use of aid resources. However, a striking feature of the data is that the success of individual development projects varies much more within countries than it does between countries. The authors assemble a large set of project-level “micro” correlates of project outcomes in an effort to explain some of this within-country variation. They find that measures of project size, the extent of project supervision, and evaluation lags are all significantly correlated with project outcomes, as are early-warning indicators that flag problematic projects during the implementation stage. They also find that measures of World Bank project task manager quality matter significantly for the ultimate outcome of projects. They discuss the implications of these findings for donor policies aimed at aid effectiveness.”

 See also: A Few Useful Things to Know about Machine Learning.  Pedro Domingos. Department of Computer Science and Engineering, University of Washington, Seattle, WA , 8195-2350, U.S.A.pedrod@cs.washington.edu