Ania Calderon, Craig Hammer, Fiona Smith, Joel Gurin, Katelyn Rogers, Lejla Sadiku, Maurice McNaughton, Muchiri Nyaggah, Nancy Salem, Nnenna Nwakanma, Shaida Badiee, Stefaan Verhulst, and Teg-wende Idriss (Tinto).

Abed Khooli, Ali Rebaie, Amanda Smith, Amy Guy, Anca Matioc, Ania Calderon, Caleb Rudow, Claire Schouten, Claudia Schwegmann, David McNair, Eric Swanson, Francesca De Chiara, Giuseppe Sollazzo, Hatem Ben Yacoub, Jacqueline Klopp, Jean-Noé Landry, Jenna Slotin, Joshua Powell, Julian Tait, Keitha Booth, Krishna Sapkota, Krzysztof Izdebski, Leigh Dodds, Maya Forstater, Mollie Hanley, Omenogo Mejabi, Oscar Montiel, Paul Walsh, Paulina Bustos Arellano, Pyrou Chung, Raed M. Sharif, Rafael García Aceves, Riyadh Al-Balushi, Rob Kitchin, Rosario Pavese, Satyarupa Shekhar, Tom Orrell, Willow Brugh, and Yan Naung Oak.

Carlos Iglesias, Enrique Zapata, Arjan El Fassed, Stefanie Felsberger, Alan Hudson, Kshitiz Khanal, Michal Kubáň, Carla Bonina, Devangana Khokhar, James McKinney, Khairil Yusof, Pierre Chrzanowski, Adrián Pino, Ana Brandusescu, Andrea Borruso, Anne-Marie Heemskerk, Bart Hanssens, Ben Parker, Christian Medina-Ramirez, Eduard Martin-Borregon, Elise Dufief, Eva Constantaras, Francois van Schalkwyk, Joshua Tauberer, Lindsay Read, Manuel Acevedo, Marina Godoy Crotto, Martin Noblecourt, Martín Szyszlican, Matteo Brunati, Paige Kirby, Rachel Rank, Rupert Simons, Aaron Wytze, Adam Kariv, Alannah Hilt, Alla Morrison, Andi Pawelke, Andrea Ayres Deets, Andrew Nicklin, Andrew Therriault, Andrew Young, Anna Alberts, Anna Fleming, Anna Powell-Smith, Anton Ruehling, Antonio Jesús Sánchez Padial, Arturo Muente-Kunigami, Audrey Ariss, Bierta Thaci, Carole Excell, Chang Liu, Chipo Msengezi, Chris Taggart, Daniel Carranza, Darko Brkan, David Moore, David Rae, David Sasaki, David Selassie Opoku, David Wasylciw, Denice Ross, Dhanaraj Thakur, Dheeraj Ravindranath, Diego Cuesy, Duncan Edwards, Edafe Onerhime, Edward Saperia, Eliza Niewiadomska, Fabrizio Scrollini, Felipe Amaya Salazar, Feng Gao, Gabe Sawhney, Gabriel Mercado, Gabriela Rodriguez, Gaurav Godhwani, Georges Labreche, German Stalker, Gerry Tychon, Gwen Phillips, Hari Subhash, Hossein Maleknejad, Jason Lally, Jason M. Hare, Jay Daley, Jeff Geipel, Joel Natividad, Jonathan van Geuns, Jorge Florez, Jorge Umaña, Jose M. Alonso, Joshua Powell, Juan Ortiz Freuler, Juan Pablo Marin Diaz, Kate Vang, Katie Clancy, Krystina Shveda, Krzysztof Madejski, Kyle Copas, Laura Meggiolaro, Lisa Walmsley, Liz Dodds, Liz McGrath, Maciej Możejewski, Madeleine Ngeunga, Maggie Walter, Manuel Acevedo, Marnie Webb, Martin Bader, Martin Noblecourt, Matthew McNaughton, Michael Schnuerle, Mike Davies, Mikhail Parfentiev, Miles Litvinoff, Momi Peralta Ramos, Mor Rubinstein, Nadiia Babynsky Virna, Nancy Salem, Natalia Mazotte, Nikesh Balami, Nikhil VJ, Nino Macharashvili, Noémie Girard, Nora Lester Murad, Owen Boswarva, Pablo Cruz Casas, Paloma Baytelman, Paola Mosso, Paul Bradshaw, Paul Hindriks, Paul Stone, Paulina Bustos, Pedro Manrique, Philip Horgan, Pınar Dağ, Rachel Murray, Ruba Ishak, Scott McQuarrie, Selene Yang, Sidi Zakari Ibrahim, Stefaan Verhulst, Steven Adler, Sym Roe, Tara Susman-Peña, Thomas Lassourd, Tina Appiah, Tyler Kleykamp, Valentina Delgado, Virginia Brussa, Walter Palmetshofer, Will Skora, Yacine Khelladi, Yanina Bellini Saibene, Yohanna Loucheur, and Zukiswa Kota.

The editors also wish to thank and acknowledge Jean-Noé Landry and the team at OpenNorth for their invaluable support of the project’s administrative processes.

Special thanks are due to the entire African Minds production team: Simon Chislett, Leith Davis, and Tessa Botha, as well as African Minds Director, François van Schalkwyk, for his partnership in the publishing process. Finally, the project also owes a great debt to Nola Haddadian, the Publisher at IDRC, for her tireless support and patience throughout the publishing and editorial review process without which the project would not have been realised.

Ten years may seem like a short period of time, but, when technology is involved, it constitutes a generational age. Institutional memories are curiously short, and in the cultural context of open data where amateurs are often welcome and professional barriers to entry are low, it is easy for work to proceed with little awareness of the past. This last decade has seen many succeeding phases of activity, so we have encouraged our authors to take a comparatively long view (when set against other contemporary writing on open data) to document the past in order to lay stronger foundations for future research and action.

We have also sought to understand open data as a global movement. Although some accounts have a tendency to focus on the North American or European roots of open data, tracing histories back to the launch of data.gov under Barack Obama’s presidency, open data practice has been shaped by interventions from across the globe. To gain a vantage point on open data as a global movement, this collection draws upon the editors’ engagement with the Open Data for Development (OD4D) network¹ which has been closely engaged in regional networks in the Global South and involved in a range of global initiatives, including the Open Data Barometer (ODB), the Open Data Charter, the Open Government Partnership (OGP) Open Data Working Group, the Impact Map, and the Open Data Leaders Network.

Since 2015, OD4D has also been the permanent co-host of the International Open Data Conference (IODC), and the editors of this volume have been involved in preparing conference reports, including shared roadmaps for action, for the third, fourth, and fifth IODC meetings. We have seen how, over its five editions, IODC has shifted from a focus on open data, in and of itself, toward discussions that are thematic, sectoral, regional, and issue oriented, fostering critical debates on open data. The conference tracks and sessions at IODC have ultimately provided many of the chapter titles in this book, reflecting the many subcommunities of the open data field that have emerged. The debates at IODC over the last nine years also provide a useful proxy for debates across the wider field of open data, so a survey of the IODC conferences offers us one route to explore, in broad strokes, a history of how the focus of the open data movement has evolved.

The first IODC was hosted by the United States Department of Commerce and took place in November 2010 in Washington, DC.² At the same time, in London, a civil-society led conference, the Open Government Data Camp, was taking place.³ These parallel events captured the growing excitement about open data from both governments and civil society and marked the end of a year in which open data had moved from idea to initiative and from inception to the earliest stages of institutionalisation. Over time, the boundaries between government and civil society networks have become more fluid with both positive and negative effects. The focus of these early events was on showcasing the platforms that had been built and discussing the potential for open data across sectors. However, even at this early stage, questions were being asked about how the impact of open data might be tracked, and whether bold claims being made on the transformative potential of open data could actually be realised.

By the time of the second IODC, hosted by the World Bank in July 2012,⁴ the question of how to measure emerging impact was firmly on the agenda. At this point, open data was being discussed in the context of international development and the movement had broadened to include a number of open data leaders from developing countries. Yet, while many of the projects profiled were still platform-focused, it was becoming clear that simply releasing data was not enough and that the quality of data available was far from perfect. Early discussions turned to whether the potential returns of open data had been overstated and how to deal with the growing gap between rhetoric and reality. That early sense of an impact gap still pervades many of the chapters in this collection with several authors exploring the various reasons that could explain less than promised progress on transformative use. However, we note that the perception of an impact gap is rarely reflected by a similar level of difficulty in sourcing case studies of open data use, raising questions about the perception and the reality of progress on open data, as well as the influence of early conceptual models for open data impact on current critical practice.

By the time of the third IODC in Ottawa in May 2015, the focus had moved to an examination of how open data ideas and practices were developing in different sectors and regions.⁵ The conference captured a period of dramatic regional and sectoral growth of open data activity with increasingly diverse representation from across the globe. There was growing recognition that opening data alone was not enough to create impact. Instead, as many of the chapters in this collection explore, to secure outcomes from open data, clear goals need to be established and a series of strategic interventions identified. Policy design, intermediaries, and capacity building were all on the agenda. As more stories of open data in use to solve specific problems were shared, there was a growing recognition that impacts secured in one context or sector may not automatically translate to another. And with this recognition came an understanding that, rather than a single open data movement, there may be many overlapping, interwoven movements, drawing on particular elements of open data to address many different agendas.

The third IODC also made explicit the potential links between open data and sustainable development, highlighting that open data was no longer the only data game in town. Instead, in the context of international development, open data now had to find its place alongside renewed efforts to build the capacity of long-established statistical agencies, as well as newer initiatives seeking to tap into the potential of big data from proprietary private sector providers.

The fourth IODC, held in Madrid in October 2016, was framed in terms of “Global Goals, Local Impact”, reflecting increased consolidation of global advocacy and a continued focus on shared global principles, which was evolving in parallel with the growth of subnational and thematic initiatives.⁶ Although the open data agenda had matured and become well-established as part of global policy-making, discussions explored concerns that it risked becoming a niche issue, destined to be the focus of only a small group of the “usual suspects”. Issues of privacy, gender equity, diversity, inclusion, and Indigenous data rights, all competed for space on the agenda, along with a new space for more critical discussion of how open data impact might be realised and the potential for more nuanced approaches to open data practice.

These critical threads continued into the fifth IODC that was held in Buenos Aires in September 2018.⁷ New on the agenda were discussions related to artificial intelligence (AI), and the conference saw a stronger focus on data standards and open data infrastructure. Although these later issues have long been discussed by a small but dedicated element of the open data community, there was increased recognition that they are not just technical issues. They also involve questions of data governance with political choices embedded in the use of data standards and structures, having substantial consequences for who can use and benefit from data.

In 2018, for the first time, the IODC agenda also featured a session on “Open Data Under Threat”, capturing a sense that continued progress was by no means assured. Against the backdrop of a deepening crisis of diminishing government support for openness around the world and much more public debate around the positive and negative potential of technology, concerns voiced over open data were no longer solely about a perceived impact gap. They also involved a deeper questioning of when and where openness can be safely practised and whether open data should be a priority for donors, advocates, and activists in the future.

A look at the 2018 IODC agenda also illustrates sectoral and regional sessions going deeper into the specific concerns of their fields and localities. In this, we find a reflection of the increasing diffusion of open data ideas, representing both a marker of success but also a potential risk to any future coherence of open data activity. In putting together this collection, while drawing on the OD4D network and IODC as a starting point, we have been conscious of the need to move beyond to capture wider activity on open data and to explore how an early open data movement has now become many overlapping movements. By working with a diverse community of authors, encouraging them to draw on both published literature and their own domain-based networks, as well as on wider online outreach to the community, we have looked to capture insights into the open data world from far beyond the core IODC community.

Culture and temperament inevitably shape any qualitative review of progress. As with any invested community, a substantial number of people and organisations engaged with open data have a tendency toward critique. For many, the idea that data should be open was ultimately born out of a critical opposition to the way governments were handling data and an ambitious imagining of an alternative future in which access and capacity to gain benefit from data is more evenly distributed. Coupled with the differences in pace between rapid technological change and comparatively glacial governmental reform, this critical approach combined with well-meaning ambition can lead to the progress of the last decade being underplayed. Challenges on the horizon ahead can too often serve to mask the steps that have been taken in order for those challenges to become visible.

In looking across the chapters that follow, we are struck by the extent to which open data ideas have become established across the globe. For instance, in Chapter 28 (Multilateral organisations), Hammer describes how, from 2010 onward, global development banks have integrated open data into their own methodologies, helping to popularise open data initiatives in developing and developed countries. In Chapter 29 (Private sector), Gurin, Bonina, and Verhulst illustrate the private sector’s widespread use of open data with examples from Asia, Africa, Latin America, Europe, and America. And since the Sustainable Developments Goals were adopted in 2015, robust, comparable, and open data has been emphasised as a critical tool to both inform and monitor development efforts. Across the entire section on Open Data Sectors and Communities, examples of open data being used to drive socioeconomic benefits or to shape policy debates are too numerous to mention here.

The adoption of open data as a central tool used in a number of major global policy initiatives of the last decade is particularly notable. The OGP, the International Transparency Initiative, the Extractives Industry Transparency Initiative (see Chapter 8: Extractives), and the Global Legal Entity Identifier Foundation which was created to respond to the last financial crisis (see Chapter 3: Corporate ownership), have all embraced open data within their work. Within the OGP in particular, commitments related to open data have been some of the most popular and successful.⁸ As Chapter 17: Algorithms and artifical intelligence explores, even as public attention shifts from open data toward a new wave of excitement about AI, open data ideas appear firmly established as a foundation for governmental AI policy.

So why is the current period for open data one of re-evaluation, rather than of celebrating progress? Put simply, the adoption of open data as part of the global development toolbox has opened it (rightly) to substantial scrutiny. How quickly are efforts to open up data leading to change? What is the return on investment from open data-related reforms? What are the factors that shape whether or not open data leads to impact? And finally, how does work on open data interact or integrate with other core issues of sustainable development, such as gender equity, Indigenous rights, and good governance? Questions such as these have received increasingly detailed attention over the last few years. Although hardly any of these questions have simple answers, by looking at both progress and challenges, this volume seeks to bring together evidence, examples, and analysis that can support efforts to address them more clearly than before.

For all the steps forward described above, as we look to the horizons of open data, we are confident in stating that policy excitement about open data has peaked. Ten years in, we are past the peak of a hype cycle and past the point where promise has to give way to evidence of practical impact. As a result, many open data communities are fast approaching their difficult teenage years with a deepening identity crisis.

Over the last decade, debates around the role of data in society have moved to centre stage, but arguments for openness now have to share the spotlight with newer excitement over the economic potential of big data, machine learning, and growing fears about the negative impacts of data stemming from data-driven manipulation of politics or the corporate invasion of personal privacy. Although early narratives around open data may have been able to present increased access to data as an unalloyed public good, contemporary advocacy must confront a much more complex landscape in which power, politics, and the question of who gains or loses from unfolding regimes of data access cannot be ignored.

This presents a number of key challenges with which the following chapters attempt to grapple. As open data has spread globally, the way in which open data ideas have manifested across different sectors, communities, countries, and stakeholder groups has increasingly varied. Regional distinctions of emphasis have developed, with, for example, some downplaying the importance of open licences (see Chapter 37: Sub-Saharan Africa) and others talking of innovation rather than of openness in order to avoid political resistance (see Chapter 34: Middle East and North Africa). As sectoral efforts deepen, it is domain or subject matter experts, rather than data specialists, who drive activity forward, so that the challenges of creating cross-sectoral linkages and building shared data infrastructure become even greater. Increased emphasis on inclusion places a substantial demand on problem-centred initiatives, which, in light of low levels of data literacy, must choose whether to focus on data for expert communities or to actively pursue the promise of open data as a tool of wider popular empowerment. When the focus shifts from calling for access to data to creating data infrastructure and putting data to work, the divergent goals of those who formed an initial open data movement come clearly into view and managing the tensions that emerge can be complex.

It was in mid-2017, as these tensions were becoming more apparent, amid a sense that overall momentum for open data may be faltering, that the State of Open Data project was conceived. Our objective:

To critically review the current state of the open data movement, assessing its progress and effectiveness in addressing challenges related to social and economic development and democratisation around the world.

Based on such a broad stock-taking of open data activity, we may not be able to fully resolve questions about the future of open data, but we can provide an account that helps practitioners, policy-makers, and community advocates to step back from their own position to gain a view of the wider landscape. By doing this, we hope to offer a rich and timely perspective and the groundwork for constructive debates that will shape the next decade of open data.

The open data field already benefits from a number of semi-regular quantitative studies of the progress of open data, such as the ODB⁹ and Open Data Index,¹⁰ both also supported by OD4D. To complement these, the approach to the State of Open Data project was designed, from the outset, to be more qualitative and narrative in style, involving a five-stage process.

1.Selection. Working with the OD4D network, potential chapters were identified based on open data communities, regions, stakeholder groups, and cross-cutting issues. Authors were then invited to lead on creating these chapters. The introduction to each section of this book provides details on the selection of chapter topics.

2.Engagement. Authors were asked to create an initial “environment scan”: a community brainstorming of issues, evidence, key actors, and events related to their topic. Scans were posted online for public comment and additions to gather more examples, case studies, articles, and input from beyond the authors’ own networks.

3.Writing and review. Responding to a common set of questions and prompts, authors then completed full chapter drafts, drawing on the input received from the environmental scans. These draft chapters were sent for peer review by independent reviewers and by members of our editorial board. Reviews were sent to authors who completed chapter revisions based on the input received.

4.Public drafts and discussion. Public drafts for the majority of the chapters were posted online ahead of the IODC in Buenos Aires in September 2018, where emerging themes were discussed. Panel discussions on themes from the work were also held at the OGP Summit in Tbilisi, Georgia, followed by additional opportunities for revision.

5.Synthesis and recommendations. Based on a collective review of all chapters, the editors have worked to draw out key findings and recommendations, which are summarised in section introductions and the book’s conclusion, including recommendations for research, funding, policy-making, and practitioner communities.

The authors and contributors to this project have been drawn from a wide range of backgrounds. Some have been active in the open data field for many years, while others are relative newcomers. Some are advocates and activists, while others are observers or academics. Some are open data generalists, while others specialise in a particular field. Many draw upon a range of different roles and positions.

When considering all of the authors, contributors to the environment scans, independent reviewers, and the editorial board, input has been received from over 220 individuals from around the world. Representing the diversity of the open data community with regard to gender, diversity, and global inclusivity has been the key principle underlying our approach to this volume. The goal was to achieve a 50–50 gender split in terms of authorship, although we fell short of this with a 58–42 split in favour of men.

One of the notable features of open data is the way in which it has been adopted and shaped by so many different stakeholders. Unlike “big data”, for example, which appears to be primarily a corporate concept marketed to governments and civil society, networks around “open data” have always been much more diverse, fluid, and cross-sectoral. More than anything, this breadth and fluidity lies at the root of the impending identity crisis of the open data movement. For a long time, it may have been possible to manage the tension between different interests via a short-term focus on simply gaining access to more data. However, when stakeholders turn their focus to data use and the need to quantify the return on their investment of time and resources, a broader open data coalition is much harder to sustain. Determining what the open data movement can (and should) yield moving forward, how to maximise every investment made, and how to take on the challenges of mainstreaming and sectoralisation simultaneously, is at the core of the movement’s identity crisis. The cracks that may appear need not lead to crisis. Rather, they should serve to highlight in relief where realignment and rethinking are needed for the future.

In editing this collection, we have sought to work with all of the authors to address the needs of four main groups: researchers, funders, policy-makers, and practitioners.

For researchers, each chapter draws upon available academic and grey literature, providing detailed citations and suggesting further reading. The hope is that researchers will use these chapters as a primer on open data within particular contexts to identify critical research gaps in need of further attention. In particular, the inclusion of further reading is designed to assist the use of these chapters in a teaching context.

For funders, we have sought to highlight key organisations and stakeholders in each sector and region and to point out instructive examples of what is being done with open data, noting, where appropriate, gaps in the available resources needed to develop new ideas or to scale what works in more locations for larger impact. A dedicated chapter on donors and investors (see Chapter 25) also considers the need for greater coordination of funding, and, as with most chapters, points to current areas of underinvestment, particularly around the infrastructure needed for sustainability and high-quality data delivery, as well as capacity building, to create a widespread culture of data use.

For policy-makers, we have encouraged authors to address both progress and challenges in the implementation of open data. In many cases, you will find more on the persistent challenges, reflecting not so much a lack of progress but rather the shared critical and progressive mindset of our authors who seek ambitious social change through the application of open data. We have sought, however, to keep chapters focused on a relatively small number of issues, prioritising those that most deserve policy attention at present.

For practitioners interested in detail on open data projects, whether focused on data publication or use, we have sought to provide them with both critical reflection and inspiration. The hope is that by reviewing chapters related to a specific sector from multiple perspectives, practitioners will discover new ways of framing old problems and practical ideas about how to move forward in using open data as a tool of entrepreneurial development or social progress.

Crucially though, we do not know how many of the readers of these essays will, in the future, associate themselves with the label of “open data practitioner” or “researcher”, or whether they will simply perceive their role as someone who engages with open data as one tool among many. This is perhaps core to the identity crisis the movement may be currently experiencing and to the corresponding adjustments that open data communities will need to make in the second decade of open data. Is there still a need for a sustained movement that identifies the technical and licensing regime around open data as its core objective? What ethical and normative approaches need to be integrated into any future engagement with open data? Is it a good thing for the debate to move on from openness to adopt other narratives related to “good data”,¹² “data justice”,¹³ or “data rights”¹⁴? We will return to these questions after our review of the state of open data offered in the following chapters, when we will be better placed to discuss what stands to be gained or lost in the years ahead.

1https://www.od4d.net/

2https://web.archive.org/web/20101128112407/https://www.data.gov/conference/

3https://web.archive.org/web/20101218004117/http://opengovernmentdata.org/camp2010/

4https://web.archive.org/web/20120821060331/http://www.data.gov/communities/conference

5https://web.archive.org/web/20150716100733/http://opendatacon.org/

6https://web.archive.org/web/20161104210854/http://opendatacon.org/

7https://web.archive.org/web/20190127062831/https://www.opendatacon.org/

8Khan, S. & Foti, J. (2015). Aligning supply and demand for better governance: Open data in the Open Government Partnership. Open Government Partnership, 1 January. https://www.opengovpartnership.org/resources/aligning-supply-and-demand-better-governance-open-data-open-government-partnership

9https://opendatabarometer.org/

10https://index.okfn.org/

11Digital Science. (2018). The state of open data report 2018. London: Digital Science and FigShare. https://digitalscience.figshare.com/articles/The_State_of_Open_Data_Report_2018/7195058

12Daly, A., Mann, M., & Devitt, S.K. (2019). Good data. Amsterdam: Institute of Network Cultures.

13Taylor, L. (2017). What is data justice? The case for connecting digital rights and freedoms globally. Big Data & Society, 4(2). https://doi.org/10.1177/2053951717736335

14Tisne, M. (2018). It’s time for a bill of data rights. MIT Technology Review, 14 December. https://www.technologyreview.com/s/612588/its-time-for-a-bill-of-data-rights/

There are very few sectors where open data might not have a role. However, to provide a broad overview of open data developments, the focus chapters in this section were selected based on an analysis of the agenda and discussions at recent editions of the International Open Data Conference (see Introduction), as well as themes identified in the 2015 Sustainable Development Goals (SDGs)¹ and the categories of high-value data identified in the G8 Open Data Charter² and global measurement tools (see Chapter 22). We have sought to select sectors at varying stages of progress, ranging from government finances (Chapter 10) where budget and subsidy datasets have had a pivotal role in shaping early work on open data through to telecommunications (Chapter 14), a sector largely overlooked to date as an area of focus for open data initiatives. Our coverage is by no means comprehensive, and, inevitably, there are different choices that could have been made on the scope of each sector. Water and air quality, for example, could arguably have been addressed as sectors in their own right, although, in this volume, they find their place as sub-themes within the essay on the environment (Chapter 7).

The key advantage of a sectoral approach in a review of open data is that it requires us to take a step back and to understand open data in context. Understanding and intervening in the struggles around land ownership data (Chapter 12), for example, requires an appreciation of the different systems related to land ownership and a recognition of the role that records and data play in securing land rights. Progress on opening up corporate ownership data (Chapter 3) can also be better understood in the context of the global financial crisis and the search for policy responses at that point in time when “shovel-ready” open data approaches were available to draw on. Sectoral engagement with open data is far from inevitable but instead relies on the right combination of advocacy, infrastructure, and backing at key opportunity points. These opportunities can evolve quickly from external events, as in the 2008 financial crisis, or from the alignment of different stakeholder interests over time, such as with agriculture (Chapter 2), where a case can be made for opening up new pre-competitive space and a sectoral shift from closed to open models of data production and use.

The histories and horizons of open data vary from sector to sector. We have worked with the authors of each chapter to identify key dates in the development of open data in their sectors. These timelines are published as part of the online companion to this book. Taking this long view helps us to understand the way in which open data ideas enter into an existing landscape of data systems, political attitudes, stakeholder relationships, and programmes of action. In the crime and justice sector, for example (Chapter 4), the history of open data might have started with interactive crime mapping in 2005, but new technological approaches have to contend with long-established and localised legacy ICT systems and the conservative ethos of many judicial institutions. The crime and justice chapter also draws important attention to the way open data work unfolds between different branches of government, encouraging us to consider government stakeholders beyond just the executive branch.

A sectoral approach also allows us to look beyond the “usual suspects” who self-identify with open data to locate other important stakeholders who have, to date, been on the periphery of the open data discourse. In the health chapter, for example (Chapter 11), the creators of an open source health management information system (HMIS) emerge as central players whose actions, in tandem with national-level policy activity, can contribute to improvements in the availability of aggregated open health data. Chapters on education (Chapter 6) and geospatial data (Chapter 9) also identify key stakeholder groups (the open education working group and open geospatial community, respectively) who have had relatively weak links to wider open data communities in spite of their relevant expertise and knowledge. A sectoral approach also reveals common influences across sectors. Eleven of the sixteen chapters in this section, for example, mention either the Open Data Charter³ or the Open Government Partnership⁴ as an influence on open data advances, and nine chapters draw on evidence from the Open Data Barometer⁵ to understand progress.

Finally, a sectoral lens can help us to assess open data maturity and explore how embedded open data has become across a sector. To comprehensively assess the state of open data in a particular sector might require looking at the proportion of data generated in that sector which is ultimately available as open data, or it might involve an audit of use cases, identifying how far open data approaches have been adopted in addressing key sectoral challenges. While the chapters that follow are indicative rather than exhaustive, they show very different states of open data adoption. For example, the chapter on development assistance and humanitarian action (Chapter 5) suggests that the idea of open by default has become reasonably embedded in the sector, allowing stakeholders to shift their focus to developing and embedding more mature data-use practices. However, the chapter authors also note the ongoing challenge of building a data, and open data, culture in the sector, particularly given complex relationships between international, national, and local stakeholders. In the extractives sector (Chapter 8), work on governance, looking at issues such as contracts, tax, and royalty payments, has progressively integrated open data over the last decade, resulting in increased data availability and use. Yet, at the same time, the wider sector has seen a vast growth in proprietary data collection by commercial firms using emerging technologies, meaning that while the absolute quantity of open data available may have grown, the relative proportion of open to closed data has likely declined. A similar issue appears to be at play in the transport sector (Chapter 15), where route-planning apps have been a poster-child of the open data movement, but where the authors report that only a fraction of the data used to drive these apps is actually provided as open data. Even when open data is available, it may only cover a limited portion of the transportation experience. If a small group of stakeholders have access to superior but restricted-access application programming interfaces, the ideal conditions for innovation in the development of solutions will not develop.

One factor evident throughout the chapters in this section (and indeed throughout this volume) is that while open data has a technical foundation, progress relies upon policy, people, and collaboration. Open data tends to enter the discourse of a sector through the actions of one or more small groups that are able to enrol a wider group around them to develop and explore the application of open data. These are the open data communities that this section also attempts to bring into focus.

The original working title for this section of the book was “Open data communities” rather than “Open data sectors and communities”. Yet, it became clear that for most chapters, there was an open question as to the extent to which a coherent and recognisable community could be said to exist around the chapter subject. For most, the idea of community invokes a group with some degree of shared values, attitudes, and goals, and whose members have some degree of interaction. Although there are many successful “thematic” open data communities, in some sectors there are many different groups, each with distinct agendas, and with varying levels of interconnection, whilst in other sectors the sense of a distinct open data community is much more nascent.

By looking at the extent of community networking within, and across, sectors, we bring into focus a number of the drivers for community cohesion, including levels of collaboration, learning, and progress on securing impact from open data. For example, in the broad accountability and anti-corruption field (Chapter 1), we find strong connections have been made between distinct communities of investigative journalists, open contracting and procurement specialists, and individuals acting under a “follow the money” banner. While often meeting separately, these groups also benefit from a high degree of fluidity and the exchange of ideas through events, multilateral meetings, and field-building publications. By contrast, although the crime and justice chapter (Chapter 4) identifies many individual projects looking at open data, there is little evidence of a sustained global or regional community pushing open data forward in this sector, and instead the landscape is made up of ad-hoc initiatives by governments or other stakeholders without the evidence of substantial community development. Using a community lens can highlight how differing sectoral cultures, and different levels of investment in community coordination, impact on the degree to which action has been mobilised to address open data.

A community lens also brings to the fore questions about the people involved in steering and shaping open data activity within particular domains, inviting an exploration of whether communities are diverse or whether they are globally representative. Ultimately, all of the chapters serve to illustrate that community building requires intentional effort and sustained investments of time, resources, and energy. For example, substantial efforts have gone into outreach and to providing travel support to enable participants from lower-income countries to participate in open data events, such as the International Open Data Conference,⁶ the GODAN Summit focusing on agriculture,⁷ Open Contracting global events,⁸ or meetings of the International Aid Transparency Initiative’s Technical Advisory Group.⁹ We should also note that global community building often requires bridging language barriers, and the flow of learning and conversation between different linguistic open data communities is worthy of further investigation.

Lastly, a community lens can be used to examine the position of an open data community within a wider sector as a whole. Are open data specialists simply talking to each other or are they reaching out to shape wider sectoral work? The picture is varied, although, in almost all cases, there are opportunities to improve the integration of open data practitioners into existing sectoral communities of practice and to leverage open data to broaden those communities. A level of cultural adaptation is generally required as open data communities interface with existing communities of practice. For example, the national statistics chapter (Chapter 13) calls for improved connections between open data and national statistics offices (NSOs), recognising the need to focus on building mutual respect and understanding between statistics professionals and open data communities. The urban development chapter (Chapter 16) also illustrates the challenges of inserting an open data community into the mainstream of the sector, where, although open data has become a central topic in community discussions of resilient cities, within the commercial-led smart-cities marketplace, open data is treated as a minor tool rather than a transformative agenda.

The chapters in this section identify hundreds of different organisations engaging with the open data agenda and many different projects opening data and putting it to use. However, they also reveal that increasing open data adoption and impact across a sector is by no means inevitable. The process of making data open and ensuring that datasets can serve a much wider range of use cases than those for which they were originally created has resulted in a myriad of issues around data quality and interoperability that are only now starting to be addressed. Many chapters also point to major bottlenecks caused by endemic capacity gaps around data analysis and use, as well as the limited deployment of strategic actions to connect data analysis with policy change. In many sectors, the full potential of open data is being missed, in part, due to a shortage of sustained specialist work on technical and policy challenges and difficulty in finding non-profit or for-profit models that can bring the extended focus needed to move beyond pilots into long-term projects and programmes.

What is clear, however, is that although, in 2009, open data was promoted as a general reform, today, it is primarily seen as an asset to be used in meeting specific goals (including the SDGs). This raises many new questions for the open data movement as a whole, including whether it can be said that there is even a single overarching open data movement or whether we have many divergent sectoral movements and communities. How can open data be used to go deeper into sectoral problem solving while still maintaining cross-cutting learning and connections between communities? The chapters that follow are intended to address these questions and more.

An established international field working on anti-corruption and accountability has existed only marginally longer than the open data movement itself. Open data for anti-corruption holds great potential, but efforts often face the common challenge that data availability does not automatically translate into effective data use.

Strategies employed by reformers to address corruption and anti-corruption include strengthening the capacity of different local stakeholders to work with open data and tailoring the implementation of technical solutions to the institutional and political dynamics of particular contexts.

Research indicates that the relationship between transparency and accountability is not necessarily causal or linear. Anti-corruption practitioners continue to debate how to best address the challenges at the heart of corruption problems.

Future efforts need to focus on strengthening the connections between open data and anti-corruption practitioners, and ensuring the sharing of evidence and lessons learned.

The expectations that open data might serve as a strategic tool for reformers around the world to improve anti-corruption and accountability results has been a key driver behind the push by open data advocates for more and better open government data. The underlying theory appears straightforward: open data “can reinforce anti-corruption efforts by strengthening transparency, increasing trust in governments, and improving public sector integrity and accountability by reinforcing the rule of law through dynamic citizen participation, engagement, and multi-stakeholder collaboration”.¹

Excitement over the promise of open data has been shared by large and small organisations alike. The G7 and the G20 have recognised its value, and multilaterals, such as the World Bank and the Inter-American Development Bank, have invested heavily in programmes to support open data. Bilateral aid agencies, including the Department for International Development (DFID) in the United Kingdom (UK) and the United States Agency for International Development (USAID), and philanthropic foundations, such as members of the Transparency and Accountability Initiative,² have also supported open data work. Additionally, multi-stakeholder initiatives like the Open Government Partnership, the Open Contracting Partnership (OCP), and the International Aid Transparency Initiative (IATI), among others, have facilitated and promoted efforts by government agencies, civil society, and media organisations across the world.

Current evidence about the impact of this work is relatively scant. Some argue that open data efforts have proven successful in “improving government by tackling corruption and increasing transparency, and enhancing public services and resource allocation”, and in “empowering citizens [...] by enabling more informed decision making and new forms of social mobilisation”.³ Yet, at the same time, others have pointed out that open data has not been widely used in corruption investigations.⁴ Other research questions the linearity and simplicity of the assumption that data availability leads to results, arguing that “transparency, information or open data are not sufficient to generate accountability”.⁵ It is fair to conclude that challenges exist in measuring the impact of open data to improve accountability and anti-corruption results. This raises questions about whether, and how, the open data community can convince the general public that greater access to open data is key to achieving results.

One reason why the evidence is patchy is that the relevant literature lacks common definitions of accountability and anti-corruption.⁶ Definitions are often overly broad, defining accountability as the combination of answerability, the obligation to inform and justify public decisions, and enforceability, the ability to sanction or remedy contravening behaviour.⁷ Corruption, in turn, is often used as an umbrella term to group behaviours related to the abuse of entrusted power, ranging from bribery and embezzlement to clientelism.⁸ Both accountability and anti-corruption are about preventing, detecting, and disrupting abuses of power. Open data is a very powerful tool to reduce information asymmetries that lead to a power imbalance; however, more open information is not enough to actually negate the institutional and political dynamics that allow those in power to abuse it and remain impune.

Open data activists often assume that the solutions needed to strengthen accountability and to reduce corruption are already known by specialists, and that open data will increase the effectiveness of those working to implement such solutions. However, international development work focused on anti-corruption and accountability has been around only marginally longer than work on open data,^9,10 and the communities working on these issues have not yet reached consensus on several issues. Debates related to anti-corruption and accountability revolve around: concerns over how to prioritise and address corruption challenges in different contexts;¹¹ exploration of how to design, monitor, and implement interventions;¹² questions related to understanding and tracking changes in the political and technical dynamics that shape institutional reform and behavioural change;¹³ discussions regarding how to identify and assess impact;¹⁴ and ways to ensure that interventions actually empower marginalised groups and provide them with the means to improve their lives.¹⁵

Reflection on the overlap between the open data and the anti-corruption and accountability agendas offers important opportunities to methodically test underlying assumptions about the impact that power abuses have in practice and the role opening information can play in addressing these abuses. However, up to this point, such work has often been done by “pioneers” with little collaboration across agendas and with little attention given to the movement from simple data availability to using it strategically to address systemic or sectoral problems and achieving real impact.

This chapter will highlight the challenges, gaps, and progress made on key issues at the intersection between open data, accountability, and anti-corruption.

In the mid-2000s, reformers pushing for open data began to demand the publication of data by governments in reusable formats that could be accessed by the general public. This effort later evolved toward identifying and then closing gaps in the publication of datasets,¹⁶ with an additional focus on the implementation of data standards and data interoperability. Advocates have been successful in framing the open data agenda, advocating for standards, and convincing civil society, governments, and, to a lesser extent, the private sector to engage.

Open data initiatives have tended to focus on the release of data summarising existing government processes, while paying little attention to uses and users of the data, often treating open data as an end in itself. This has created momentum for the publication of datasets, but has also led to some governments focusing solely on transparency around selected issues without paying attention to opening up the underlying processes behind that data which are used internally to support transactions and decision-making. Open data and open government advocates have labeled these types of efforts as “passing off the release of inconsequential government-held data as transparency”¹⁷ or “open-washing”.

The mostly implicit theory of change in many open data initiatives is that more information will (almost) automatically lead to its use by those working on anti-corruption and accountability and enable them to produce better outcomes and achieve impact. However, while information and technical improvements are great tools to better understand accountability and corruption challenges, they are not sufficient to address entrenched power structures that oppose governance reform and generate systemic changes.

In 2016, the Open Data Barometer found that a number of datasets relevant to anti-corruption work (e.g. budgets, company registries, spending, contracting, and land ownership) “still tend to be highly opaque, and often the least open”, and that important differences persist within and across regions.¹⁸ A review of key datasets in five G20 countries also indicates that these relevant datasets are often not yet published, that public officials lack the skills to leverage open data, and that initiatives to strengthen citizen engagement using open data rarely link to anti-corruption or sectoral areas.¹⁹

In 2017, the “Open up guide: Using open data to combat corruption”²⁰ identified 30 key datasets²¹ for fighting corruption (see sample in Figure 1), as well as standards that can make these datasets interoperable. The guide was tested in Mexico,²² which produced evidence on the value of the guide for enabling government officials to open key datasets. It also highlighted the need to define clear data governance frameworks and to promote dialogue between data users and producers in government and civil society.

Progress in the publication of data, even if uneven and patchy, has raised important questions about who will use that data, how they will use it, and what results can be achieved. There are no silver bullets when it comes to promoting the use of open data by local stakeholders to address corruption and accountability challenges. The approaches that have been used to bridge the gap between data production and use can be classified into three overlapping groups: those focused on data standardisation and technological tools, those focused on engaging users and particular problems, and those focused on changing government processes and practices.

First, those initiatives that have focused on standardisation and technological tools have paid great attention to the development of data standards and their implementation by governments. They aim to improve the quality and comparability of published data and enable the development of tools that can be adapted according to the needs of audiences in different contexts. These efforts have targeted a variety of areas from democratic processes to resource flows and, to a lesser extent, development results. Examples⁵² include the IATI standard,⁵³ Fiscal Data Package,⁵⁴ the Popolo data specification,⁵⁵ the OpenCorporates schema,⁵⁶ and the Open Contracting Data Standard.⁵⁷

The development and management of data standards related to accountability and anti-corruption have shown a similar trend to that of the broader open data space. Initially, standardisation was focused on finding ways to better present the information that was produced by governments, but later those leading the standards began to pay greater attention to data users’ needs, moving beyond representing government processes into using data to reshape those processes. Important challenges still remain in terms of the technical features and tools needed to make the implementation of data standards more useful and in relation to ensuring that stakeholders have the capacity to use the standards to address locally relevant challenges. Increased collaboration between standard developers, implementers, and data users at the global and national level is needed to develop technical solutions in a way that is sensitive to local capacities to produce the data and to put it to use within complex political systems.

Even though there are a number of stakeholders working to implement data standards, promote interoperability, and develop tools to facilitate data use, the actual use of open data has not increased proportionally. New projects that pay greater attention to supporting users trying to use data presented according to data standards are now emerging with strategies to promote data use⁵⁸ and to explore the use of open data to fight corruption in particular countries.

Second, those initiatives that have paid greater attention to engaging users and achieving particular outcomes have shown important results. A clear example is the work of journalists at the national and international levels involved in collaborative networks such as the International Consortium of Investigative journalists (ICIJ). Recent scandals, such as those exposed by the Panama⁵⁹ and Paradise⁶⁰ Papers, have not only uncovered corruption, but have led to the consequential launch of prosecutions and the resignation of public officers and even presidents.⁶¹ After publishing such stories, data has been made available in open formats that can enable the work or analysis by others. While these examples could be used to question the value of open government data on politically salient issues when compared to data obtained through leaks, the disparity in outcomes may indicate more about the differences in the way this data is being produced, treated, and used.

Leaked data often includes full versions of documents that are then used to stimulate collaboration among networks of journalists, both online and offline. These networks review the data thoroughly to organise it, clean it, and make sense of it. The same networks then use the data to find leads that are further corroborated and developed through other sources, including open government data, documents, and on-the-ground research. This intense work is not focused on merely making the information available; it is aimed at making the information useful to further identify and expose illegal activities carried out by those in power.

Lastly, there are a number of initiatives that have focused on fostering and supporting changes in government processes and administrative practices. Some of this work relies heavily on data to explore the value of new technologies like machine learning, blockchain,⁶² and algorithms;⁶³ however, using these tools to analyse open government data has not yet reached a widespread level of popularity.⁶⁴ Interest in these still emerging technologies leads inevitably to a range of challenges with regard to potential violation of privacy, the possibility of reproducing and increasing existing biases, and the threat of using automation to hide questionable decisions and practices.⁶⁵

Other important work to promote change in government practices through the use of open data is led by multi-stakeholder initiatives on procurement, international aid, extractives, and public infrastructure. Even though these initiatives are at different levels in their uptake and maturity, all of them seek to alter long-established government processes. While some initiatives use formal multi-stakeholder forums for the production, verification, and use of open data, others promote the integration of open data into government processes beyond the simple publication of data. These initiatives have led to important, if not yet widespread, results,⁶⁶ ranging from identifying money flows in the extractives sector⁶⁷ and the misuse of public resources⁶⁸ to achieving savings and better service delivery through improvements in the planning and implementation of government processes (see the box opposite).

The wide variety of approaches by government and civil society to address anti-corruption and accountability challenges should not be read as an attempt to identify the single best strategy to achieve results. Instead, the open data community needs to distill, share, and debate the lessons emanating from both successes and failures, reflecting on what these lessons mean for developing and implementing further projects moving forward. Additionally, it is not only a matter of choosing between approaches focused around a particular technology, stakeholder group, or government reform. It will be necessary to identify and explore, in practice, how a combination of these approaches can help to address particular corruption and accountability challenges within specific contexts.

As discussed above, more data does not necessarily lead to a proportional increase in either the use of data or anti-corruption and accountability results; however, increased access to standardised, machine-readable, and reusable data has enabled sharper investigations into instances of corruption and abuses of power, additional research to identify inefficiencies in the use of public resources, and greater awareness of systematic biases against particular groups. Nonetheless, current advances are generally insufficient to address the root causes that underpin corruption and accountability challenges: the ways in which power is distributed in a given society and the subversion of existing (democratic) institutions for private gain.

There are several emerging efforts to improve openness beyond the executive branch^74,75 and address corruption and accountability challenges⁷⁶ in other branches of government, including activities at the heart of the democratic process, such as monitoring elections and the undue influence of money in politics through campaign and party financing. Some initiatives, like those of organisations in the Openingparliament.org⁷⁷ network, have paid particular attention to the legislative branch.⁷⁸ These efforts to open and communicate information about legislators, how they perform their duties, and about legislation itself⁷⁹ have been at the centre of work to strengthen democracy, with consequent benefits for anti-corruption work. Yet, these emerging efforts often face important challenges in relation to the availability of data in machine-readable formats to support the accountability of members of parliament, as well as in relation to implementing lobbying reforms. While legislatures may be happy to see accessible data on aspects of government operations, they may be more resistant to opening up structured data on their own activities and interests.

A number of national governments have also been subject to interesting efforts to open up data about the judiciary⁸⁰ and oversight bodies, such as audit institutions. Their aim is to get a more complete picture of how cases are assigned to judges and how those cases progress until judgment is rendered. However, these efforts are not yet widespread and often face claims that they may hamper due process during trials.

Crucially, even when data is made available, initiatives tend to remain limited in their focus on particular branches of government or processes and generally have weak formal connections to the institutional systems in which they operate (e.g. the functioning of democratic institutions, the use of public resources, and the application of effective sanctions against those who engage in corrupt practices). This makes it challenging to follow cases of corruption from identification through to final resolution and sanctions, and, ultimately, hinders lasting impact and influence on future activities. Without ongoing scrutiny of democratic systems of power to enforce anti-corruption measures, individuals and institutions are able to continue to act with impunity, and the consolidation and replication of corrupt networks is facilitated.

The theory of change behind the idea of using open data for anti-corruption and accountability also highlights the potential value data can have in empowering citizens and enabling social mobilisation. Some organisations have used data to pursue an activist approach to crafting stories, uncovering wrongdoing, and identifying entry points that enable others to get involved. However, these approaches can put activists in peril, and, as of today, there are no established safety networks for this work, such as those that exist to protect human rights defenders or journalists. The absence of such safeguards, and the weaker links to established mechanisms for protection, may lead activists to take unnecessary risks and expose them to legal, reputational, or physical attacks.

Despite the emergence of various activist approaches, it is generally organisations that focus on governance, transparency, participation, and accountability that most frequently lead initiatives to address accountability and anti-corruption. These organisations play an important role, but still need to find effective ways to engage other key stakeholders, such as organisations working in particular sectors or territories, those working on rights protection, those active in social movements, or those working through other alternative mechanisms, such as strategic litigation. The minimal connections that often exist between open data initiatives and a broader range of stakeholders can deepen the challenges related to the usability, and actual use, of the data and hinder real impact in addressing problems that affect citizens.

The assumption that an improved capacity to identify instances of corruption leads to activating institutional oversight mechanisms is not necessarily wrong; however, assuming that those mechanisms will actually deliver results in the form of successful reforms, grievance redress, or sanctions without additional effort is, at the very least, an oversimplification. Open data can be a tool useful not only to identify instances of corruption, but also to engage, challenge, and reform the institutional designs and practices that enable corruption. Turning this potential into reality requires the use of approaches that consider the institutional and political environments in which data is produced and used. Adopting such approaches will enable sharper thinking about how the use of data can be more effective in practice, how to counter the forces that oppose openness (be they for private gain or from an aversion to change), and how to build stronger bridges between advocates for open data, activists working on sectoral and systemic challenges, and the democratic forces that can act on the findings and evidence obtained from the use of open data.

The open data community needs to explore and test innovative ways of using data that take all of these slowly acquired insights into consideration: to effectively challenge institutional mechanisms and practices that perpetuate impunity, inefficiencies, and the abuse of power; to reach out to unusual stakeholders by finding ways to integrate their needs and interests; to tap into existing social mobilisation processes; and to link the efforts of the different stakeholders engaging on reform with government branches and institutions.

Over the past decade, reformers have used open data to create ripples and, in some cases, waves, in uncovering and prosecuting corruption. In a few cases, these efforts have resulted in the reform of systems where corruption had been the norm. Through their work, these reformers have generated insights that can help us to understand how to use open data more effectively to fight against corruption going forward. One of the key insights open data reformers have started to embrace is the value of adopting a problem-driven approach to the publication and use of data in order to address much more specific corruption and accountability challenges. These approaches also call for more collaborative models that are more grounded in the environmental context in which they are to be implemented, building on the needs and interests of local reformers and moving away from the replication of generalised practices toward the development of tailored “best-fit” solutions.

This shift in thinking on how to best use open data for accountability and anti-corruption does not represent a break with the ideals at the core of the open data movement, such as “open by default”, but it does call for some refinement of our thinking around how to articulate advocacy goals, learning aims, and the desired impact. There is, and will continue to be, value in demanding that governments open up data on key issues related to accountability and anti-corruption; however, these demands should be based on a clear understanding of the users and usefulness of data, as well as the technical, political, and institutional environments in which it will be used.

To grapple with the implications of these insights, stakeholders would benefit from engaging with each other to develop non-linear approaches to better address particular corruption and accountability challenges. Learning about other perspectives and approaches will provide useful insights to improve how we devise and test methods, monitor progress and results, and spur dialogue on how and why specific approaches might yield better results. In particular, it is important to explicitly address the following questions:

1.How can the field facilitate and strengthen the work of local champions, including government, civil society, and the private sector, to generate and use data evidence to demand accountability and to lead in the fight against corruption?

2.What are the needs of different local stakeholders with regard to using open data? How can these insights help to tailor technical tools and methodological approaches to better support stakeholders in different sectors and contexts?

3.How can stakeholders build stronger and more effective connections among those working on open data, accountability, and anti-corruption, and those who work in sectors on specific issue areas?

4.What are the potential risks associated with using emerging technologies, such as machine learning, and artificial intelligence, in relation to accountability and anti-corruption? How can these tools and methods be combined with the social mobilisation and institutional mechanisms needed to generate and sustain change?

5.How can actors link the technical capacities needed to use open data with the political strategies needed to effectively change systems and ensure sustainable results?

Success in addressing these questions over the next decade will enable reformers to achieve significant accountability and anti-corruption results, but future work will require the community to develop holistic theories of change and the willingness to test them, implementing interventions in an iterative manner that enables reformers to ensure that open data is useful and used, while strengthening collaboration among stakeholders to achieve systemic reform and explicitly addressing entrenched power dynamics. In addition, the community must move beyond simple dichotomies that either highlight the production or the use of data toward models that start with identifying a specific problem to be solved, include the identification of the opportunities and challenges faced by local champions, and embrace learning and adaptation to develop solutions that are a better fit in specific environmental contexts.

High level leadership, private sector engagement, and academic networks have put open data on the agenda across the agriculture sector.

Issues of ethics, ownership, power, culture, and capacity all need to be addressed before the sector is “open by default”.

Mapping information flows through agriculture value chains can help policy-makers and practitioners to identify pre-competitive spaces for open data sharing and to understand the implications of opening data more broadly.

Donors and governments have a key role to play in establishing the policy framework for openness and supporting the infrastructure needed for a sustainable open data commons for agricultural research and practice.

Goal 2 of the Sustainable Development Goals (SDGs) commits United Nations member states to both achieve food and nutrition security and to promote sustainable agriculture. The world population is projected to exceed 9 billion people by 2050,¹ and the corresponding growing demand for food is exerting massive pressure on the use of water, land, and soil, which is further exacerbated by global warming. The majority of the world’s food is still harvested by smallholder farmers,² many of whom are poor and food insecure themselves.³

Agriculture is a knowledge intensive industry. Government and private sector-supported research and agricultural extension work (e.g. farmer education) is central to improving crop yields, understanding and implementing sustainable practices, and getting food to market. However, it is only in the past two decades that the agricultural sector has valued data as a tool for generating, sharing, and exploiting knowledge to improve yields, reduce losses, and increase overall agricultural business outcomes.

Rapid internet and mobile phone penetration, especially in the developing world, the accessibility of satellite and remote sensing data, and new data collection and analytical approaches all play a role in the “datification” of agriculture. While data-related opportunities are increasing, challenges still exist in the policy, ethical, and data standards domains, and key datasets remain absent or inaccessible. This is especially true in terms of nutrition-related data, which is largely under-utilised in the field of agriculture. Despite some progress in raising consumers’ awareness of the nutritional value of the food they consume, demand has not been significantly redirected to the production of more nutritious food, especially in the developing world.

As we have seen, agriculture is diverse, as is the potential for applying open data to support a range of activities in the sector, from providing remote sensing data for precision agriculture applications to bringing farming extension advice to smallholder farm owners. Although the stakeholders may look very different, overarching sector goals remain mostly unchanged: to grow nutritious food as efficiently as possible, balanced with the need to secure the basic livelihood of people everywhere, using successful business models. As outlined in this chapter, the burgeoning ecosystem for open agricultural data is only beginning to address a myriad of issues as evidenced by the series of discussions that took place at the GODAN Summit in 2016 (see Figure 4). In the light of a growing world population and ever-increasing pressures on resources, we need technological improvements and innovative approaches in many areas of agriculture and nutrition to meet this goal, and data will be central to that effort.

To date, the private sector has shown only minimal interest in publishing their data openly for reuse. A much greater emphasis on incentives and business models that encourage the release of open data at all levels of agricultural value chains is necessary. Both researchers and companies need to undergo a cultural shift from closed and proprietary to shared and open, recognising the value of open data in promoting innovation, cost-sharing, and improved value chain efficiencies. The extent to which the FAIR principles have caught on, at least in the rhetoric of the sector, is encouraging and highlights the value of communicating open data ideas as part of a broader normative agenda for advancing agriculture.

In 2018, the meaningful sharing of useful (both anonymised and identifiable) on-farm data was often curtailed by legitimate privacy concerns raised by farmers and their organisations, or by farm machinery and farm management systems that operate in a proprietary space. The open data community needs to increasingly involve stakeholders who are trusted by farmers, such as farm cooperatives, in order to promote innovation using on-farm data. Right now, for those wanting to innovate with data, obtaining large satellite datasets from governments or agents who have already adopted an open data policy is a lot easier than opening up on-farm data or nutrition data from surveys. Yet inclusive innovation also requires remote sensing to provide ground-truth data, highlighting the need for ongoing efforts to secure granular data about farms with the acceptance and support of farmers and their communities. As the agri-food industry increasingly needs a “licence to operate” from the public, they have begun to release more data on their sustainability performance. Early examples of this data publication and of the private sector’s involvement in tracking SDG progress is promising in that regard. In addition, open data for agriculture has almost exclusively focused on food security, but, thus far, has neglected to consider textiles and forestry, which bear a large environmental cost and should be priority areas for future focus.

The seeds are sown for the growth of open data in agriculture, but, as yet, the evidence of lasting impact is limited. Creating the right ecosystem will need more than awareness raising. It will require all stakeholders to grapple with challenging ethical issues by turning debates and discussions into consensus, capacity development, guidance, and common approaches that can be deployed at scale.

The availability of standardised and openly licensed corporate data “at source” from corporate registries is limited, but through intermediaries like OpenCorporates, significant open data can be accessed and reused.

Big strides have been made over the last decade in laying the technical and policy foundations for more open data on corporate structures, ownership, and control, but although progress has been made on the balance between openness and privacy in corporate data, there are still issues to resolve.

Evidence suggests open corporate data can be a key tool in improving risk management and holding the powerful to account, but progress may also bring increasing hostility to openness from some entities in the private sector.

A concerted effort will be needed in the coming years to build on the foundations laid to date in order to deliver a global, robust, and reliable supply of open data on corporate identity and ownership.

Basic corporate data is essential for understanding our world. The name of a company, its legal form, registration number, formation date, the identities of its directors, and the registry where this information is held are all fundamental to knowing who we are doing business with and who our employers are, as well as which entities should be taxed and in which jurisdiction. Access to that data over time allows us to assess the performance and structure of the economy as businesses form, merge, break apart, and fail. Another layer of analysis opens up if we move from simply identifying corporate entities to identifying their owners and those who ultimately control them, a concept referred to as beneficial ownership. The more jurisdictions that require corporate ownership data to be open, the easier it becomes to navigate through a myriad of shell companies, regardless of where they are located, to identify the actual owners.

Information provided as open corporate data is of interest to public, private, and civil society stakeholders, and has a universal geographical applicability. G20 leaders have discussed the need to use corporate data to improve financial stability and efficiency, to combat corruption, and to improve the exchange of tax information between jurisdictions.¹ These same goals are reflected in the Sustainable Development Goals (SDGs). SDGs 16 and 17 address the need to reduce illicit financial flows, ensure the return of stolen assets, reduce corruption, promote investment, and develop capacities for domestic tax collection, all of which are supported by improving the availability and use of corporate data.²

Much progress on open data related to corporate identity and ownership has been made. While it might appear that the amount of open corporate data is still relatively limited, much of the infrastructure required for success, such as policy, legislation, and technical architecture is now in place or being developed. Moreover, progress to date has involved a broad base of stakeholders from government, multilateral institutions, civil society, and the private sector. Great advances have been made in creating a shared understanding that we need to be able to unambiguously identify legal entities using universal identifiers, which has led to an emerging ecosystem of links to complex corporate data. A more recent development is that the policy argument in support of beneficial ownership seems to have been won. We have seen early examples of registers providing ownership data as open data, as well as the legislative basis for many more to come in the near future mostly as the result of a civil society movement committed to opening up asset ownership as part of economic transparency. However, the withdrawal of the US from the EITI on the basis that it is a “burden for business” demonstrates the fragility of the achievements to date and the difficulty of maintaining corporate disclosure as a desirable goal.⁴⁴ For open corporate data to fulfil its promise for the SDGs, any momentum that has been created in Europe and North America will need to be recreated across the Global South.

Some of the earliest open data experiments revolved around crime data and were driven by public and journalistic interest in local crime data; however, the open data community related to crime and justice data remains one of the least developed.

Open data work in the crime and justice domain faces particular challenges related to privacy, legacy systems, and interoperability, and often involves working with some of the most conservative institutions.

Donors and international organisations have increasingly recognised the potential links between open data and the crime and justice sector, but there are many cultural and coordination barriers to be overcome.

A strong and sustainable judicial open data ecosystem has the potential to create more transparent and accountable judicial institutions and to improve the quality and effectiveness of judicial public policy, leading to greater access to justice and safer environments for all.

In May 2005, a month before the official Google Maps application programming interface (API) was launched, ChicagoCrime.org was launched, a pioneering experiment that took crime data from the Chicago Police Department and presented it on an interactive map. Not only did this inspire a plethora of diverse mapping mash-ups,¹ but it also sparked many other data-driven crime maps and acted as a key reference point for early open data arguments.² Yet despite this strong beginning, crime and justice probably remains one of the least developed sectors for open data. Outside of the publication of administrative data by police forces, usually at the local level to enable crime incident mapping, the release of open data from judiciaries or other entities within the justice system remains rare, and governments are generally more reluctant to open up crime and justice data to the public. According to the last edition of the Open Data Barometer, only 17% of surveyed governments had, by 2017, made any crime data available to the public as open data.³

There is, however, a growing awareness of “open justice” as a public good and the need to apply open data principles to enhance transparency, accountability, and citizen participation related to the activities of the government agencies dealing with crime and justice matters. Evidence of this movement toward “open justice” can be seen in the growing number of worldwide judicial commitments included by member countries of the Open Government Partnership (OGP) in their National Action Plans. In 2011, only two out of a total of 170 commitments delivered by member states related to the judiciary, while the last three years have seen an increase to 63 (16 in 2015, 25 in 2016, and 22 in 2017). Of the total of 100 justice-related commitments delivered within the OGP system since 2011, 24 are based on the use of open data (see box, Examples of OGP justice-related commitments based on the use of open data).

The importance of open crime and justice data derives from the need to reinforce transparency and accountability. Crime and justice institutions have historically been seen as rather aloof institutions, detached from social influence; however, these institutions take actions and make decisions that should really not be considered any differently from other public institutions with regard to the need to be transparent and subject to constant public scrutiny.⁴

Government is often divided into three branches: legislative, executive, and judicial. Open government data programmes have predominantly focused on the executive, which has responsibility for the delivery of government services and the implementation of legislation, leading to the release of crime data and crime mapping from police institutions or ministries. However, there has been much less focus on the availability of open data from the judicial branch of government.

This must be addressed as a core issue of democracy as the justice system should be a citizen-centred public service, where decisions are actually made by civil servants who are entrusted with the task of observing the law but are in no way above it.⁵ Greater levels of open data from the judiciary can help the system become not only more transparent and accountable, but also more efficient. Open data should inform judicial public policy. While, at present, policies are often designed from the top down and can result in poor quality services, the use of open data to build sound judicial policies through data analysis and citizen engagement will provide more efficient judicial services. Jimenez-Gomez recently described the worldwide state of the art of “open justice” initiatives more broadly and identifies the use of open data as a core element that should be taken into account to enhance the accountability of the courts.⁶

There are many elements that go into open justice, often involving the innovative use of technology in the crime and justice space. This chapter is intended to be an examination of the evolution of open data specifically pertaining to crime and justice, and is not intended to include the additional analysis of policies related to civic participation or technology other than those related to open data.

The most common sources of open data in this domain are the government agencies responsible for delivering services and implementing policies related to crime and justice. Hence, it makes sense that a vast majority of open data initiatives are national projects driven by institutions of the executive branch and the judiciary, with a few of them carried out by international organisations such as the European Union (EU) or the United Nations Office for Drugs and Crime (UNODC). While there is a not negligible amount of crime and justice data collected by private sector institutions, such as law firms, the potential contribution of these sources is yet to be realised.

Bargh, Choenni, and Meijer have accurately identified “three typical challenges” for the implementation of open data in the judicial field, highlighting privacy, legacy, and interoperability as significant challenges that should be taken into account in further development.⁹

Privacy relates to the required balance between the transparency of data and privacy for the real-life persons whose sensitive attributes, such as names, birth dates, crime types, or judgments, must be protected by removal or anonymisation.

Legacy refers to the very nature of legal data and the semantic evolution over time caused by continuous changes to rules and regulations. New crimes under the law need to be codified and old crimes can have their names changed or be redefined. Managing legacy data also becomes a challenge when government reform initiatives involve switching to new IT landscapes, requiring the migration of large amounts of accumulated data (data historically stored on paper) and then transferred to newer electronic systems. In order to make this data open and reusable, the importance of effective independent management of legacy systems cannot be underestimated.

Finally, the challenge of interoperability alludes to the necessity of ensuring that different sets of data, gathered by a large number of different agencies, be collected, stored, and then released using standardised criteria and processes, allowing the data to be integrated and combined with data from external sources. The justice system also needs to advance the use of unique identifiers that would make it easier to connect data across institutions and avoid redundancies in data collection between multiple partners across the judiciary who may be recording the same information.

The structure of the judiciary within federal governments deserves a special mention as the existence of national and sub-national levels (involving different judicial systems within one country) requires complicated inter-institutional coordination, making synchronisation and interoperability especially difficult to accomplish. Collaboration between different branches of government also presents challenges to reforms driven by the openness agenda as the transversal interaction required is perceived, in some cases, as a threat to the separation of powers (i.e. judicial independence).

Political and cultural barriers are still common hurdles for the implementation of the open data agenda in the public sector. Those barriers tend to be even higher in the case of institutions dealing with crime and justice (law enforcement agencies, the judiciary), which are traditionally some of the most conservative and independent institutions. As Roberto Gargarella notes, this is based on a conception of impartiality that holds dear the idea of isolated reflection by an individual (or a small elite of individuals) as a requisite for making correct or unbiased decisions.¹⁰

Another challenge is created when police and justice institutions lag behind in terms of technological capacity (i.e. hardware, software, skills, and expertise). Universities and scientific agencies could, and should, play a key role in building capacity for the use of data and emerging technologies in the police and justice sector.

Additionally, the involvement of civil society is still very limited in this field. Few civil society networks have projects looking specifically at open data in the crime and justice domain, with even fewer being large enough to be known internationally. However, there are some emerging examples of civil society organisations (CSOs) working independently or in collaboration with government agencies, including Measures for Justice in the United States (US), OpenGiustizia in Italy, La Nación Data in Argentina, and the Justice Data Lab¹¹ in the United Kingdom (UK).

Measures for Justice¹² is a civil society initiative launched in 2011 that has developed a data-driven set of performance measures aimed at assessing and comparing different aspects of the criminal justice system in state jurisdictions of the US. The analysis, using data extracted from administrative case management systems, covers three main categories: fiscal responsibility, fair process, and public safety.

OpenGiustizia¹³ was a project focused on organisational innovation and optimisation for the Court and the Public Prosecutor of Napoli, developed by three Italian universities and financed by the EU’s Social Fund between 2007 and 2013. Among the project’s objectives was the creation of interoperability within the system’s databases and the provision of tools for accountability and performance evaluation.

La Nación Data¹⁴ is a data journalism initiative which has been underway since 2012 by one of the main newspapers of Argentina. It consists of a news portal and blog based on data collected from various sources. It makes an intensive use of open crime and justice data, delivering content on themes such as femicide, high-profile judicial cases, and the penitentiary system.

Justice Data Lab¹⁵ is a service run by the Ministry of Justice of the UK and New Philanthropy Capital. Set up in 2013, it is aimed at organisations providing offender rehabilitation services. It uses administrative data on re-offenders to conduct on-demand impact evaluations, so that these organisations can assess the actual impact of their work through data-based evidence.

For most of these initiatives, data availability and interoperability remain a challenge. Measures for Justice, for example, covers just six US states at present. The next section will explore the different kinds of data that could or should be available on crime and justice, and the various actors involved in creating and using it.

The primary focus of opening data within the crime and justice sector is on three main categories of information:

1.Case data: information on judgments and court rulings issued by crime and justice institutions (e.g. courts, tribunals, etc.).

2.Jurisdictional data: performance and activity data from crime and justice agencies, such as statistical data related to cases, reported crimes, arrests, citizen complaints, etc.

3.Structural data: information on the internal characteristics of crime and justice agencies, such as their organisation, their internal processes, how they allocate their budget, infrastructure, rules of procedure, staff and salaries, procurement, etc.

Data produced or collected by institutions within the crime and justice system is generally made available in three main ways:

1.As primary data (i.e. unprocessed, as it was collected at the source) in downloadable datasets or files (CSV, XML, DOC, XLS, PDF).

2.As aggregated statistical data (i.e. as processed, and anonymised if necessary, data) in the form of downloadable files or datasets.

3.In aggregate form, but as graphical presentations either in static visualisations of statistical data or through the use of user-facing data visualisation and analysis tools.

As the box below illustrates, while jurisdictional and structural data may originate with either the judicial or executive branches of government, case data tends to be solely within the province of the judiciary.

The way in which open crime and justice data are combined and delivered to users can vary substantially. Table 1 gives 18 examples of data projects, although, for the sake of this chapter, detailed analysis is restricted to four illustrative cases: Openjustice (US), Data.police.uk (UK), Datos.jus.gob.ar (Argentina), and ECourts (India).

Openjustice¹⁸ is an open data project developed by the Office of the Attorney General of the Department of Justice of California (US) to establish a criminal justice data portal which was first launched in 2015. It currently delivers jurisdictional data from all enforcement agencies across the State of California, covering crime, deaths in custody, hate crimes, homicides, juvenile court and probation, citizen complaints, and the use of force. Structural data on the portal includes lists of law enforcement and criminal justice personnel, as well as county-level contextual data (educational attainment, income, poverty, and unemployment levels) for each county. The data is made available as downloadable datasets and through visualisation tools.

Data.police.uk¹⁹ is an open data portal maintained by the Home Office of the United Kingdom that provides data about crime and policing in England, Wales, and Northern Ireland.

It was launched in 2013³⁴ and delivers jurisdictional data on reported crimes and all kinds of police activity, including drug seizures, the issuance of firearms certificates, breath tests, or the setting up of cordons under the Terrorism Act. It also contains structural data on the police workforce, procurement, salaries, etc. Both primary and aggregated statistical data is available, and the update frequency is either quarterly or annually depending on the subject matter.

Datos.jus.gob.ar³⁵ is the open data portal of the Ministry of Justice and Human Rights of Argentina, containing overall data on the country’s justice sector. The portal was launched in 2016 and offers data on a range of jurisdictional activities, such as the delivery of pre-judicial mediation and the provision of access to justice, as well as information on criminal policy, the prison system, and structural data on institutions of the judicial branch and the Ministry of Justice. Primary and aggregate data are available as downloadable datasets as well as via visualisation tools. The update frequency of datasets depends on the subject matter, ranging from daily or monthly to annually.

ECourts³⁶ is a service provided by the Ministry of Law and Justice and the Supreme Court of India. Online since 2013, it contains real-time judicial data for all jurisdictions subject to the Indian judiciary. It aims to serve as a dynamic source of information on the judicial system. It is based upon a “National Judicial Data Grid”, which works as a nationwide data warehouse for case data and aggregated data delivered through visualisations.

Additional examples of open crime and justice data initiatives are highlighted in Table 1, classified according to the categories mentioned previously.³⁷

Open crime and justice data is expected to play a key role in measuring and delivering progress in terms of social and economic development. Although the UNODC has been working on crime and justice statistics for many years, the United Nations 2030 Agenda for Sustainable Development places a fundamental importance on open data at all levels to promote accountability and inclusive decisions, to support reductions in crime and violence, and to improve access to justice for all over the next 11 years. Although data will play a vital role in showcasing national progress toward over 169 global targets encompassed within the Sustainable Development Goals (SDGs), it will also allow decision-makers in all three branches of government to be able to rely on quality information for the design of public policies, based on evidence, to achieve those global targets. Key SDGs for crime and justice data include SDG 16, aimed at the reduction of violence, the reduction of organised crime, the development of effective, accountable, and transparent institutions, and ensuring access to justice and public information, and SDG 5, which focuses on gender equality and the total elimination of violence against women and girls.

With regard to SDG 5, two specific examples of effective initiatives should be noted: the provision of primary open data on sexual offences by the Colombian government through their Open Data Portal³⁸ and the specific section on gender issues of the Open Judicial Data Portal of the Ministry of Justice of Argentina, datos.jus.gob.ar,³⁹ where primary data is made available on femicides, human trafficking, and assistance granted to victims of violence.

The potential of open crime and justice data with regard to the SDGs will probably have a crucial impact in the future allocation of resources and funding for associated projects and initiatives. International organisations, such as The Hague Institute for Innovation of Law or the Latin American Open Data Initiative (ILDA), are already orienting their funding priorities in this direction as are other significant actors like the Open Data Institute, Transparency International, and the Open Society Foundations. Crime and justice open data is also increasingly on the agenda for key international organisations that are pushing for open government-oriented reforms in the public sector, including the International Development Research Centre, the OGP, and mySociety, among others.

An enabling environment is currently emerging for the spread of open data by and between crime and justice institutions. International organisations and governments have begun to consider crime and justice data as a raw material to use in implementing and evaluating public policies. At the same time, data journalists, academia, and CSOs are learning how to use open data to promote more transparent and accountable judicial institutions.

There are still, however, many barriers to the implementation of quality open data initiatives and specifically to the extended use of judicial open data. The main barriers are traditional cultural and political forces against openness within the judicial system, the lack of adequate financial and human resources invested in capacity building, and current inadequate or restrictive legal frameworks, including those that create a barrier to publishing data from judicial cases. Another hurdle to overcome is the difficult but necessary coordination of the various public institutions producing judicial data, as well as the lack of consistent standards around the production and publishing of judicial information and the relatively weak expertise of civil society and other actors in analysing the data.

Ultimately, two main actions are required in order to strengthen the judicial open data ecosystem. First, at the institutional level, governments should include the judicial sector within their access to information laws and open data policies and regulations. At the same time, justice institutions should commit to country-wide and transversal open data strategies. These strategies must take into account open judicial data intermediaries, such as academia and CSOs. It is recommended that one judicial institution in the country (e.g. the Ministry of Justice, the Supreme Court, or the Judicial Council) takes on the leadership role and coordinates the development and implementation of open data policies and plans. Additionally, judiciaries should set goals, targets, and indicators for justice delivery, and the resulting performance data should be available and evaluated through open data.

Second, concerning the use of open judicial data, each judicial system must establish a balance between publication and privacy protection. Privacy should not be used as an excuse for avoiding openness. Governments and international organisations should promote the use of open judicial data through different tools (curricula, hackathons, journalism, etc.) and public participation mechanisms should be put in place to assess and set priorities for the data release process. We also recommend setting up open judicial data portals to contain the totality of available data for each judicial system. Open judicial data should not only be provided in the format of open datasets, but also through visualisations and data stories in order to reach a wide variety of users. Advancing the interoperability of the data held in the systems of numerous institutions producing open judicial data is a must. Governments, working with leading international organisations, should promote the definition and adoption of specific standards for open judicial data. International organisations should also promote the creation of open judicial data networks and working groups, as well as the participation of open judicial data experts and leaders in relevant conferences and debates wherever they are taking place. With success in these endeavours, the next decade should see the ongoing development of a strong and sustainable judicial open data ecosystem that can enable more transparent and accountable judicial institutions while delivering more effective access to justice and safer environments for all.

From the mid-2000s, organisations and individuals working in the field of development assistance and humanitarian action have identified significant gaps in the data sharing needed to support effective coordination of funding and operational work. Early adopters of open data, from 2008 onwards, have worked to fill these gaps and have continued to pioneer open data projects.

Availability and accessibility of open data have increased substantially, often outstripping the capacity of organisations to reliably use this data, and more work is needed to ensure that data sharing reflects the principles of data protection.

Greater investment is needed in joining up data and establishing common languages and standards for aid-related data. Open data approaches have a key role in breaking down silos between aid, budget, and demographic data.

Research must now move beyond qualitative case studies to rigorous testing of theories of change through quantitative longitudinal studies.

Bureaucracies like to “hug” data¹ for many diverse reasons, and international development aid and humanitarian agencies are no exception. For decades, the complex aid regime has been plagued by information silos and technical, political, and cultural barriers to data sharing. This legacy presents a distinct challenge to effective global assistance in an era of unprecedented humanitarian crises and persistent poverty, especially in conflict-ridden states. Doing no harm and ensuring protection are key principles in development assistance and humanitarian action; therefore, ensuring data protection must also be a principle. Sharing data requires a delicate balance of effective coordination and protection of the most vulnerable. Organisations involved in aid and humanitarian action have limited funding allocated to upskilling staff and developing infrastructure. Gaps in technology and digital literacy are often barriers to building open data processes within the complex aid delivery structures.

According to the international transparency movement’s theory of change, open data is the key to unlocking the potential of international aid. Opening data related to development assistance and humanitarian action will improve donor coordination, improve the efficiency of humanitarian action, facilitate a faster response regarding relief assistance and development spending, better inform resource planning and management, and empower stakeholders and communities to push for greater participation.^{2,3,4,5,6,7,8,9} Open data, simply put, will make development aid and humanitarian action more accountable and effective. But how far have we come in realising this potential?

This chapter will provide a brief overview of the state of open data in the development and humanitarian space, focusing on data collected and published by development agencies, private philanthropic organisations, and humanitarian relief organisations. It will also supply a critical assessment of the progress and pitfalls in the global transparency movement. We find that there have been significant achievements in building consensus, standards, and technical platforms around open aid data. Yet, the supply of open data has not always matched the demand, nor has the open data revolution incited the expanded use of data in the area of international aid that may have been expected.

The key challenges lingering today involve the need to improve the quality and consistency of available data. This is difficult insofar as the data models, infrastructure, training, and business risk analysis/workflow for open data in aid and humanitarian action are often insufficiently funded. At the same time, we need to build broader awareness and expand the use of open data with the objective of building data literacy and improving (and proving) the impact of open data on decisions and outcomes. Likewise, we need to make open data accessible and useful to all stakeholders, while also addressing difficult issues, such as data privacy, protection, and responsible use. Finally, to sustain the momentum behind this data revolution, we need to garner greater evidence of impact to demonstrate the benefits of open data in the field of development and humanitarian assistance.

In the context of development assistance, the open data agenda has grown out of larger debates on aid accountability and effectiveness. Since the Second and Third High Level Forums on Aid Effectiveness in Paris in 2005 and Accra in 2008, several definitions and standards on aid transparency and open data have emerged, as well as numerous efforts to construct monitoring and verification systems around compliance with international agreements and transparency guarantees. At the Fourth High Level Forum on Aid Effectiveness in Busan, South Korea, in November 2011, most major donor countries and agencies, including many from the Global South, committed to reporting their aid information according to a common standard that combined three complementary systems: the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) Creditor Reporting System (CRS++),¹⁰ the OECD DAC Forward Spending Survey (FSS),¹¹ and the International Aid Transparency Initiative (IATI).¹²

The open data movement in international development has seen the development of a rich set of supranational initiatives,¹³ national-level policies, and international non-governmental organisations (NGOs), and networks devoted specifically to the advocacy and production of transparent and open aid data. Today, the principals and goals of open data are embedded in the United Nations (UN) 2030 Sustainable Development Goals (SDGs). In 2014, the UN’s Independent Expert Advisory Group (IEAG) published A world that counts: Mobilising the data revolution for sustainable development.¹⁴ The report called for investments in new technologies and capacity building to improve the quantity and quality of data to address the inequalities in data access between countries and for donors to promote the use of data in decision-making, participation, and accountability.¹⁵ Similar commitments were made in the 2015 African Data Consensus,¹⁶ the 2016 G8 Open Data Charter,¹⁷ the Grand Bargain for the Global Humanitarian Agenda,¹⁸ and, more recently, the March 2018 UN Statistical Commission’s 49th Session on “Better Data, Better Lives”.¹⁹ The open data movement as it pertains to international development and humanitarian aid has shared a similar trajectory in terms of the evolution of influential policies and activities.

The growth and support of open data as applied to humanitarian action is often tied to large-scale humanitarian crisis events. This work often starts with determining workflows and best practices for sharing data that will not do harm, and the first data that needs to be shared is most often geospatial data. The Global Facility for Disaster Reduction and Recovery (GFDRR), created in 2006,²⁰ has been instrumental in advocating and piloting open data for both resilience and disaster recovery, primarily through its OpenDRI initiative established in 2011.²¹ The GFDRR has connected key humanitarian actors with technical communities. Open data, including OpenStreetMap (OSM),²² has become more central for humanitarian action after its use during the response to the 2010 Haiti earthquake. By engaging volunteers, the global OSM community can quickly contribute essential geospatial data, such as location data on buildings and roads. Having the most up-to-date data can provide those involved in delivering humanitarian aid with the information needed to make strategic decisions. The UN Foundation sponsored Disaster relief 2.0 report outlined the potential impact of this kind of information sharing. The GFDRR, the World Bank, the United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA), and government agencies collaborated with the open data community during this response.²³

Many other emergency response activities have included organised efforts of global open data advocates within the humanitarian network or within digital humanitarian networks like the Digital Humanitarian Network or CrisisMappers.²⁴ The Humanitarian OpenStreetMap Team (HOT),²⁵ founded in 2010, has worked to coordinate technology communities, mappers, and humanitarians to deliver geospatial data for both international aid and humanitarian action. Missing Maps, founded in 2014 by the American Red Cross, British Red Cross, Medicine Sans Frontiers/Doctors Without Borders UK (United Kingdom), and HOT, promotes the use of open map data for humanitarian action from disaster responses to health programming.²⁶ The UN OCHA’s establishment of the Humanitarian Data Exchange in 2014 builds on years of effort by multiple humanitarian groups to open data.²⁷ UN Global Pulse, the United Nations International Children’s Emergency Fund (UNICEF), the World Food Programme, the United Nations High Commissioner for Refugees (UNHCR), and other UN agencies all work with open data. In the humanitarian space, the CrisisMappers Conference and the State of the Map events²⁸ have convened businesses, technologists, researchers, open data enthusiasts, funders, and governments. Burgeoning support for open data has also been reinforced by the proliferation of work by civil society organisations (CSOs), NGOs, technologists, businesses, and researchers, much of which has been initiated as a result of global and regional events, including the annual International Open Data Conference,²⁹ Open Data Day,³⁰ and the Data for Development Festival.³¹

In the development aid space, key leaders in the open data movement include the Members’ Assembly of IATI, PWYF, Development Initiatives, the World Bank Open Aid Partnership and Mapping for Results team, Development Gateway, AidData, the International Development Research Centre, the Transparency and Accountability Initiative, Interaction, and the Open Data Research Network. These actors have been central to establishing the broad momentum for open aid and establishing the methodologies and platforms needed to provide open aid data within developing and emerging market economies (through country-owned aid information management systems, such as Development Gateway’s Aid Management Platforms),³² bilateral and multilateral aid donor dashboards,³³ and international datasets (including the IATI registry, Development Initiatives’ Development Data Hub, and AidData’s project-level aid datasets).³⁴

As discussed above, one clear success in the open aid data movement is the emergence of a clear consensus on the need to open data and to establish robust policies to ensure the provision of standardised aid data by development and humanitarian organisations, national governments, and supranational institutions. There has been considerable progress in developing the infrastructure, in particular the systems and standards needed to collect, store, and publish open data, such as the IATI XML standard and the Humanitarian Data Exchange.³⁵ To reinforce the transparency movement, monitoring and rating systems have been established to oversee aid donor performance, including one aid-specific index, PWYF’s Aid Transparency Index, and others with a broader focus on open data, such as Open Data Watch’s Open Data Inventory, Open Knowledge Foundation’s Government Open Data Index, and the World Wide Web Foundation’s Open Data Barometer.

More recently, the open aid data movement has introduced innovations to improve access to data in forms useful to stakeholders and decision-makers. This has produced platforms that enable interactive use and easily downloadable data. Perhaps, more critically, the collection of data has taken on more inclusive approaches. From mapathons to hackathons, the entire data lifecycle has changed with new mobile and community engagement programmes. This improves the timeliness and usefulness of data and increases awareness and community buy-in. Simultaneously, there is growing attention to the need to “join up” open data across sectors (e.g. open aid data with open budget data) to increase its usefulness to key stakeholders.

Despite the progress described above, there remain numerous challenges to realising the promise of open data in international development and humanitarian action. There are four main issues: persistent problems in providing consistent, standardised data across a proliferating number of sites; concerns about privacy and data protection; a lack of organisational investment in technology; and the lack of clear evidence of the cost benefits and impact of open aid data.

The past decade of the application of open data to development assistance and humanitarian action has provided critical lessons for moving forward. We offer four key recommendations to the international open data community on how to address the key challenges faced in making open data work in the delivery of development assistance and humanitarian action.

1.The release and use of open data faces organisational hurdles. This may include a lack of resources and infrastructure needed to ensure quality and timely data collection or a lack of a data culture that encourages data use. Data is only useful if it is seen by end users as central to information products, evidence, decisions, and knowledge sharing. Open data advocates need to ensure that the mechanisms designed to supply open data are informed by, and integrated into, organisational structures in ways that are consistent with local data cultures and existing capacities. A common language is needed to develop an understanding between data consumers and data producers. The key to success is understanding the culture and context, then building capacity and usage with early adopters. The talking points about “why open data matters” need to incorporate and acknowledge the barriers and aim for opportunities that show true impact.

2.More investment is needed to support joined-up data initiatives. The evidence we have to date suggests strongly that stakeholders want open data around aid and humanitarian assistance, but would find it more useful if such data was more effectively integrated across sectors, especially with respect to domestic budgets and essential demographic information. We need to break down silos and manage open data with a comprehensive, holistic approach.

3.Successfully addressing data privacy, protection, and responsible use will continue to be critical to the success of the open data movement. Setting minimum data standards is the starting point for data sharing. Improving education on the impact and value of data sharing while still adhering to data protection and responsible data use will require a constant balance. Open data and data sharing can occur if data-driven projects are built with privacy protection by design. Data controllers, data producers, and data consumers will need to plan and manage risks and benefits by incorporating proven practices into standard operating procedures.

4.The open data community, and the broader community of donors engaged in international development and humanitarian action, need to invest more in basic research on awareness, use, and impact. Investment in technology and business analysis will also aid the implementation of open data practices. To sustain momentum for open data, we need to rigorously test the theory of change and hypothesised effects on outcomes, such as aid accountability, effectiveness, donor coordination, improved budget management, and timely and inclusive decision-making in the allocation of scarce resources.⁶¹ These studies need to go beyond static, qualitative case studies to include more longitudinal studies that are capable of capturing the larger societal costs and benefits and the long-term impacts of open data.

Open data can help researchers and policy-makers understand the education landscape, provide information for parents and children about education facilities and their performance, and serve as a key element in the creation of open educational resources (OER).

Attention must move beyond the simple availability of data on education to also question how the data is contextualised, presented, and used to ensure it does not result in the reinforcement of pre-existing biases and social divides.

There has been relatively limited intersection to date between the open education and open data communities. There are opportunities for future strengthening of these links, increasing the use of open data as a key educational resource, and supporting more applied civic education.

According to United Nations (UN) Sustainable Development Goal (SDG) 4,¹ states must “ensure inclusive and quality education for all and promote lifelong learning”. In this chapter, we consider the ways in which open data can support the achievement of this goal. In the education sector, open data released by governments and educational institutions, as well as by national and international organisations, can support a wide range of interventions, including strategies to improve the quality of education, the design of effective education policies, the creation of educational resources, and the development of the key literacies needed to operate and participate in today’s “datafied society”.²

The education ecosystem is made up of a complex network of systems and practices developed to address a wide range of sociopolitical and economic issues. Despite the enormous efforts made by countries to guarantee equal access to quality education, there are still challenges to overcome for which open data can provide insight, perspective, and a wide range of tools to further our understanding of core educational problems and to support the development of solutions. It has also been argued that open data can be used as part of a series of quality indicators to help people to make better decisions related to educational opportunities and methodologies and to choose among education providers. More overtly, open data used in the development of open educational resources (OER) can be considered a key tool in promoting citizenship and democratic values and developing the transversal literacies that citizens require in order to participate in a datafied society. Figure 1 indicates three main ways in which open data and the broader education sector intersect. You can also think of this in terms of how open data use intersects with the three main education stakeholder groups: policy-makers, parents and learners, and educators.

In this chapter, we will explore both the opportunities and challenges that open data presents across the education sector, drawing upon examples from around the world, and wider critical arguments and studies related to open data.³ We are aware that while open data can promote public participation and social innovation, it can also reinforce pre-existing biases by connecting performance with the poor and vulnerable in an unfair manner, helping to further marginalise those who cannot choose where to live or study. The evidence we have gathered suggests that although impact to date has been mixed, there are many opportunities to substantially strengthen existing networks and activities around open data and education in the future.

Understanding the current state of education and identifying ways to improve education, are vital tasks for policy-makers. Davies,⁴ Niemi,⁵ Burns, Köster,⁶ and the 2017 EU Eurydice Report⁷ argue that policy-makers need better access to evidence in order to address policy issues. Data that describes achievements, attainment, enrolment, or the distribution of learning are all important to determine whether educational systems are working or not. The United Nations Educational, Scientific and Cultural Organization (UNESCO)⁸ has indicated the need to ground policy on reliable evidence to ensure that educational policies are effective, efficient, and implementable. They argue for the use of comparable indicators and for ensuring that data is available disaggregated by gender, administrative area, geographical location, sociocultural groupings, education level, and type of provider to enable a comparison between the different groups and to identify those who are educationally disadvantaged.

Motivans,⁹ in exploring data availability to monitor the SDGs, also calls for educational data that is relevant, valid, reliable, timeless, punctual, clear, transparent, comparable, accessible, affordable, consistent, and with potential for disaggregation. There has been some progress on making this data available (and open), but major gaps remain. Notably, educational data from countries such as Kenya, South Africa, Ecuador, or Montenegro¹⁰ is scarce and neither widely nor openly available, making it difficult to assess their progress in relation to SDG 4.

While some states have had standardised testing since the 1950s, it is only in the last 20 years that standard national assessments have become the norm in Europe, and the majority of the world’s population still resides in countries without such testing.¹¹ International initiatives have stepped in to fill the gap. The best-known example of performance data provided at the international level is the Organisation for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) test¹² initiated in 2000, providing data about learner performance in science, mathematics, and reading. The results of this standard test, linked to sociodemographic data, enable comparative analysis regarding differences in performance among diverse groups of learners, taking into account gender, social background, migrant learners, and ethnicity. In 2015, 72 countries participated in the PISA survey, generating data that is commonly used in evidence-based policy-making to help educational stakeholders to target specific problems guided by clear information. Individual (anonymous) student results from the study are published in downloadable structured data formats for common statistical software.

When open data is available as disaggregated data then a wide range of actors can get involved in its analysis. Academics are clearly major users of education-related data, but private consultancies and non-profit organisations have also taken advantage of available datasets. For example, in the United Kingdom (UK), the FTT Education Datalab¹³ was established by a nonprofit education services company to help policy-makers improve educational practice. International organisations, such as the OECD, UNESCO, and the World Bank, make use of data (combined with qualitative research) to contribute to the international collection of policies, presentations, policy tools, and frameworks intended to support evidence-based policy-making. Van Schalkwyk (2017) has also drawn attention to the way in which institutions providing performance data (in particular, higher education institutions in South Africa¹⁴) take advantage of cross-institution comparisons for benchmarking and how making more granular information available as open data has provided “a new fuel for transformation”.¹⁵

However, when approaching educational data for research and policy purposes, there are at least two important considerations to keep in mind. First, the privacy of educators and learners must be protected when using or sharing data, particularly administrative and statistical data containing personally identifiable information. Surfacing and addressing patterns of educational disadvantage requires a careful balance because it is important that educational data can be disaggregated by gender, sociocultural background, educational level, and type of school. In the UK, controversy has emerged a number of times over the intrusiveness and level of data disclosure from the National Pupil Database.¹⁶

Second, it is important to consider the capacity to create and use data, not just its availability. In this area, one project to watch is the CapED initiative.¹⁷ This project, active in 25 of the least developed countries (LDCs), aims to connect national education policies with data sources, and to support states in their use of this data in the development of their national action plans to achieve SDG 4. As each national CapED project works with UNESCO’s Institute of Statistics to implement a data component, there may be opportunities to further emphasise open data approaches.

When microdata cannot be disclosed, the design of indicators that describe the data landscape is also of crucial importance. At the national level, one example that demonstrates this is the Data Chile education indicators site¹⁸ that provides information from the National System of Performance Evaluation (SNED). SNED has been constructed using six indicators: school effectiveness, improvement, initiative, improvement of working conditions, equal opportunities, and the integration of teachers, parents, and guardians (see Figure 2). In an open data context, it is important to think about who gets involved in defining the indicators that will shape the sources of data that will be available in future.

In summary: demand is high for data across the education landscape, but supply varies. When open data is available, established policy-makers can be joined by new actors, including entrepreneurs and journalists, to debate and shape education performance and policy; however, even in the absence of globally comparable data or the use of that data by policy-makers, datasets on educational institutions can also drive change through parent and pupil behaviours.

In many countries, parents and/or pupils have some degree of choice over educational institutions. Statistics have long played a role in decisions related to the selection of learning products, programmes, and providers. With the availability of open data, a range of interactive platforms have emerged that use institutional or third-party assessment data to inform parents and learners, providing them with indicators and information they can use to make informed choices.^19,20 The data made available about educational institutions tends to focus on performance (e.g. university ratings) by using standardised metrics, but also may provide detailed information on programmes and prerequisites.

The last decade has seen the launch of numerous portals around the world that provide the means to compare the quality of education at different institutions using data provided by national and local authorities. Some examples include the Identicole portal in Peru, MIME from the Ministry of Education in Chile, JedeSchule run by non-profit organisations in Germany, the mobile app-based Conozca su escuela in Costa Rica run by Programa Estado De La Nación, and Scholen Keuze and Scholen op de Kaart in the Netherlands.²¹

A number of platforms go beyond using data to encourage “shopping around” in the selection of schools. For example, Mejora tu escuela in Mexico,²² created by El Instituto Mexicano para la Competitividad (IMCO) with funding from the Omidyar Network, places an emphasis on gathering feedback from users of the platform and equipping them to advocate for improvements to their existing schools. In the UK, School Cuts,²³ created with the backing of major teachers’ unions, places the emphasis on how funding cuts in education are impacting individual schools and was used as an advocacy tool in the last election. One of the unions funding the project claimed it helped to change “750 000 votes during the election and resulted in the government stumping up another £1.3 billion for schools in July”.²⁴ However, the vast majority of platforms focus on maps and rankings. Figures 3 and 4 show two further examples from the UK. The first one, School Atlas, was developed by the Mayor of London and showcases the impact of income deprivation on children in London. The second example is a map of schools in London developed by a private firm, Locrating Ltd, which places the emphasis on school quality, cross-referencing data from Ofsted (the inspector of schools) and the Department of Education (UK). It showcases schools by area, displaying school quality as “inadequate”, “requires improvement”, “good”, or “outstanding”; however, if we look at the data from a critical perspective, we can note the biases this information may portray by reinforcing preconceived notions of privilege and disadvantage.

Both examples offer an illustration of how the quality of education can be portrayed, but, even with contextual data, there is a risk that such information could stigmatise pupils from schools rated as inadequate or in low-income areas. We need to consider critical ethical questions when making data available about schools or, at the very least, ensure performance data is accompanied by contextualised information about the socioeconomic challenges faced by the relevant community, such as poverty, integration, and inclusion.

While school information portals are popular and may support more informed decision-making by learners faced with a complex mix of educational opportunities, there is limited empirical evidence to date on whether they ultimately improve education as much as advocacy-oriented efforts aimed at holding governments accountable or at ensuring proper funding for quality education for the most vulnerable in our society. When it comes to data on educational institutions, we have both ample open data supply and demand, as well as active intermediaries who are able to sustain their platforms. While there may be cases of individual impact for particular learners, the net social impact is difficult to determine.

The final application of open data in education is its direct use in the development, or as part, of OER. OER are defined by UNESCO²⁵ as “any type of educational materials that are in the public domain or introduced with an open license”. Open data used as OER can allow students to learn and experiment by working with the same raw data researchers, governments, civil society, international organisations, and policy-makers generate and use. They can form a key component in research- and scenario-based learning activities, and in supporting students to develop informational, statistical, scientific, media, political, and critical-thinking skills. By working with real-world data, students can develop storytelling and research skills, and can apply analytical, collaborative, and citizenship skills in using data to solve real-world problems.

This idea of using open data in education is recognised in the sixth principle of the Open Data Charter²⁶ on open data for inclusive development and innovation, which states that it is key to “[e]ngage with schools and post-secondary education institutions to support increased open data research and to incorporate data literacy into educational curricula.” Although it is not clear how much emphasis has been placed to date on this point by countries and cities adopting the Charter, the groundwork to support the use of open data as OER has been laid in a number of projects.

In 2015, the Open Education Working Group of the Open Knowledge Foundation, established in 2013, published Open data as Open Educational Resources: Case studies of emerging practice²⁷ in which a series of authors presented activities that could be adopted by educators at schools and universities to promote the use of open data in research-related activities. The book provides examples and best practices, showcasing how to use real data from research and from national and international data projects to foster educational activities to develop data literacies and critical thinking through collaborations among students, researchers, and academics. One of the practices portrayed in the book is A Scuola di OpenCoesione in Italy,²⁸ an educational challenge, designed for Italian high school students. It was funded under the open government strategy on cohesion policy in partnership with the Ministry of Education and the Representation Office of the European Commission in Italy.

Other practical examples of the use of open data as OER²⁹ can be found at the Open Data School in Russia, which provides a series of lectures and seminars from experts on open data topics. The Open Linked Data project at the Universidad Técnica Particular de Loja in Ecuador presents the results of a study on Linked Data technology for students, researchers, and educators, and Data Science Fundamentals in Palestine offers an online tool to enable students to follow the Foundations of Data Science training course developed by students and academics from Birzeit University. Finally, Monithon, also from Italy, offers an example of applied learning through open data, which citizens and university students, alongside researchers and policy-makers, use to monitor development projects. However, even with these notable successes, many initiatives focused on the use of open data as OER have been relatively short-lived, and the connections between the open education and the open data communities are still relatively weak with only a few points at which the communities intersect.

Supporting use of open data as OER is closely linked to work on data literacy (see Chapter 19: Data literacy). Recently, the Latin American Initiative for Open Data (ILDA) has developed a training programme for academics in the use of open data for teaching and learning³⁰ to support them in developing the capacities needed to live and work in the datafied society, including learning to construct knowledge and analysing information critically from a wide range of data sources.³¹

Following Uhlir and Schröder’s argument³² that “[s]tudents may be less effectively educated and trained if they are unable to work with a broad cross-section of data”, and Davies’³³ assertion that “there will be greater need in future for capacity both in state and society to be able to debate the meaning of data, and to find responsible ways of using open data in democratic debate”, we consider that the inclusion of open data in curricular activities is key to ensuring that both educators and learners acquire the skills they need to participate in contemporary society.

Over the last ten years, open data availability has grown, including data about education and data that can be used within education. Looking for school performance information may have involved using tables published once a year in newspapers ten years ago, but now many countries have interactive websites offering analysis and visualisation: ranging from official government sites to private sector-managed portals. Schools and post-secondary education institutions no longer need to rely on tables in textbooks, but can go to real-world updated datasets for teaching and learning; however, many challenges remain.

Although open data can provide evidence about problems that need to be addressed at the policy level, it can also be a key component in the development of the literacies needed in a datafied society, as well as in enhancing and promoting civic participation and understanding of the media and the sciences. However, it cannot be considered as the panacea for all educational problems.

Data is never neutral and it is ultimately a political instrument. Data and the algorithms used to analyse it can prompt stigmatisation, segregation, and discrimination. Mainstream narratives may place the blame for poor quality education on the children that perform poorly on standardised tests based on their economic or social background, instead of pointing at the authorities who have failed to provide the policies, programmes, and funding needed to improve the schools those children attend.

Arguments for opening data in education have tended to focus simply on the importance of access to data. Such arguments can gloss over the non-neutrality of data and the potential threats inherent in data-driven decision-making, where the context for data collection and presentation is opaque or where data “consumers” lack the critical thinking skills needed to interpret the data. They often also ignore the impact of trends toward the marketisation of education. We do not believe that it helps to approach open data as innocuous and benign per se. As Kitchin³⁴ states, “if open data merely serves the interests of capital by opening public data for commercial re-use and further empowers those who are already empowered and disenfranchises others, then it has failed to make society more democratic and open”. However, as we have seen above, with examples like SchoolCuts.org, it is not only private interests that can deploy data for implicit or explicit political ends and there is potential for critical action.

Ultimately, while there are many challenges around the use of open data for education, it is through wider education about the creation and use of open data that these risks can be best addressed. The wealth of open data on all topics that could be applied to OER can be part of this. In conclusion, we recommend that:

In the use and development of education indicators, it is important to prevent analysis exclusively through the use of algorithms as these may reflect biases and can foster the stigmatisation of vulnerable students.

When governments open up educational data, they must ensure that it is anonymised to prevent the identification of individuals and collectives and, in addition, consider the potential uses of this data by public and private stakeholders to prevent this data from being used unethically.

When institutions, civil society, and private sector organisations build tools using educational data, they need to consider the potential impact and use for students, educators, and educational communities.

And finally, to foster data and citizenship literacies, the open education, open data, and open science communities must collaborate to develop educational materials and curricula to support educational institutions and programmes at all levels, including training for educators and educational communities.

Data and information underpinning environmental knowledge is recognised as a form of power.

Vast quantities of environmental data are available online through many dedicated local, regional, and international data portals. This reflects long-established norms and practices of data-sharing within the environmental research community.

Emphasis must be placed on increasing the volume and geographic coverage of open water and air quality data.

Making connections between datasets across borders and thematic silos is essential to support greater understanding of a changing climate, to address air quality, to manage water resources, and to sustain biodiversity. However, there is often a disconnect between academic and official data initiatives and open-source, grassroots/citizen-science open data projects.

Context-aware open data approaches and well-resourced data infrastructures are crucial to avoid loss of data, missed opportunities, and duplication of effort.

As the amount of environmental data from sensor networks increases, there will be major inequalities in global data coverage to address with developing countries often being more poorly represented.

Since the early 1960s, we have seen an increasingly vocal response to unmitigated anthropogenic impacts on the environment.¹ Although there were earlier activists and movements, the 1960s marked the period when disparate voices started to coalesce. Environmental activists started conceptualising environmental problems as political matters, and, in doing so, using scientific knowledge as part of their armament. This led to a significant change in policy-making with regard to the use of scientific outputs and knowledge as supporting evidence. Data and information have become forms of power that are used to drive or change political discourse on issues affecting the environment. Knowledge derived from science, coupled with activism, played a major role in getting governments to endorse the Declaration of the United Nations Conference on the Human Environment in Stockholm in June 1972.² It was at this conference that governments accepted that anthropogenic impacts on the environment were a reality and that more research was needed to understand the causes, impacts, and mitigation measures. Since that time, we have had subsequent international environmental engagements that rely on scientific knowledge to guide activism, decision-making, and policy development.

The 1990s brought the digital revolution. Data generation and exchange became easier, and, by 1996, the internet had become mainstream, allowing for easy digitisation and the dissemination of data. Environmental data became easier to acquire and to share. Although access to environmental data, information, and knowledge is not a recent phenomenon, over time the emphasis for open access has shifted from information and knowledge as products to include the underlying elements: the data that comprises these products.

Environmental concerns are all-encompassing, ranging from microbial research through to large planetary weather systems research. Open data provides an opportunity to promote review, transparency, accountability, participation, and the identification of knowledge gaps. The growth in environmental open data portals to support research, advocacy, decision-making, and communication indicates the importance of sharing data on a range of environmental issues.

The following sections present an overview of the progress on open data in relation to four key environmental domains: climate change, air quality, biodiversity, and water resources.

Open data plays a crucial role in advancing our collective efforts to ensure sustainable management of all our natural resources. It has fostered collaboration that would not have been possible 30 years ago. It has allowed scientists to review the veracity of their work and hold them accountable for their conclusions, as it does politicians for their decisions. Furthermore, it has also supported instances of greater civil participation in the public and private sector spheres with the potential to give poor and marginalised people greater power through knowledge. Open data has also helped to drive the development of innovative products and services, not only in developed countries, but also in developing countries, addressing issues of environmental conservation, skills development, and economic growth. Overall, open data has shown revolutionary potential, although the measurement of impact remains difficult.

However, there is still much effort needed to ensure that environmental data becomes fully accessible to address environmental challenges. The advancement of the environmental open data agenda must happen at both the macro and micro levels. At the macro level, changes are necessary on an institutional scale to challenge closed governments to open their data. The collaboration between thematic sectors must be encouraged to avoid data duplication and gaps, as well as to maximise the value of open data. A coherent and collaborative approach must be adopted to address data gaps, specifically in developing countries. These gaps can be filled through adopting vendor and ‘donor agnostic’ data management systems, integrating data sharing agreements for funded programmes, and establishing formal data sharing programmes with the private sector without compromising personal information or trade secrets. The development of case studies is a powerful mechanism to encourage sharing as it can illustrate effective processes and the value of open data.

At the micro level, institutions should develop formal or structured data management strategies that can proactively lead to open data. Data management strategies must always be focused on organisational needs and address standards, quality, applications, and capacity building.

Environmental open data has helped shape national and international policies and decisions. Notwithstanding the challenges of getting governments and private sector entities to share data, the volume of open data is increasing. Our task is to ensure that the release of environmental open data is needs-based, user friendly, and of sufficient quality to address the local, regional, and global challenges in developing a sustainable future.

During the last decade, data has rapidly become more available across the extractives sector. Civil society, researchers, and journalists have responded by finding new ways to examine natural resource revenues, locations, production statistics, and corporate filings, drawing on data which, until recently, was only available to the companies involved or locked up in databases of proprietary data providers.

The Extractives Industry Transparency Initiative (EITI) adopted an Open Data Policy in 2015 and has since introduced a database of revenue payments for its 51 members, demonstrating a shared view that open data can serve as an enabler of accountability.¹

Open data principles have also been gaining traction within government-led extractive industry reporting regimes, including requirements to submit structured and standardised data. However, experience shows that unless reporting as data is made mandatory, companies prefer to provide unstructured PDFs.

New extractives open data has, in some cases, allowed for vibrant and timely evidence-based debate on taxation in resource-rich countries, offering a public space for review of various public policy options. However, it is important that analysis, journalism, and evidence-based advocacy reaches policy-makers in order for it to achieve lasting impact.

Since the 1990s, academics, civil society organisations (CSOs), and multilateral institutions have paid increasing attention to the impact of oil, gas, and mining operations on human development in resource-rich countries. The apparent paradox that abundant natural resources have not, in many countries, translated into economic growth and human development² has sparked considerable work toward shining a light on how extractive industries operate. Questions of policy and taxation, of the local impact of extractive operations, and of governance and corruption, have all found their way onto the agenda.

At the start of this millennium, substantial international advocacy efforts toward greater transparency for the extractives sector started to make up ground with the creation of the Extractives Industry Transparency Initiative (EITI), Revenue Watch Institute (RWI), which later merged with the Natural Resource Charter to form the Natural Resources Governance Institute (NRGI) in 2013, and the Publish What You Pay (PWYP) global coalition among others. A focus on the disclosure of information about payments made from extractives companies to governments and on contracts and concessions have led, over the last 15 years, to a number of new legal disclosure requirements, national-level multi-stakeholder partnerships, and voluntary disclosure schemes. In the last few years, these reforms have started to yield new flows of documents, and, in some cases, open data.

However, persistent problems of unequal access to data and poor data quality in the extractives sector are far from solved, and it has long been acknowledged that a lack of accessible data affects everyone from grassroots civil society groups and national anti-corruption watchdogs through to multilateral institutions engaged in economic planning. This chapter will examine how stakeholders in the extractives sector have engaged with open data over the last decade, working in parallel to secure incremental policy change on data publication and to put the data that is already available to use. It will also explore how the broad community of practice around open extractives data has supported a cross-pollination of ideas and research methods, helping to break down silos between different development disciplines and to more rapidly facilitate informed and evidence-based debate.

Understanding the extractive sector requires data. A country’s fiscal regimes provide detailed rules on how oil and minerals can be extracted, how extracted resources will be priced, the deductible costs for extractors, and how revenues are to be collected.⁸ Further rules and environmental regulations regarding where and how extraction can take place are mapped through mining cadastre datasets. Information on the companies, corporate structures, the public–private partnerships involved, and the mechanisms through which they are financed, can only be understood through data analysis. Global commodity traders manage the transfer of oil, gas, and minerals around the world using complex data systems.

Although transparency efforts in the extractives industry in the early 2000s were focused primarily on the publication of documents ready to be checked and reconciled through an audit process, by the start of this decade, there was an increasing emphasis on disclosure in the form of data. This coincides with the emergence of the open data movement on the global stage, marking the extractives transparency sector as an early adopter of open data methodologies.

Notably, work on transparency in the extractives sector has focused as much on requiring the private sector to open up data as it has on opening up government data. PWYP has described how this framing was a strategic choice, noting that “At the time of the launch [of PWYP] there was little recourse at the global level to push for disclosure of revenues by resource-rich developing country governments”, but “mechanisms to require disclosure by companies which are listed on stock exchanges and subject to accounting regulations were available and could be amended”.⁹ There is, within this work, an ambitious programme of re-imagining how a global market should function, with concerted work to rebalance the line between public and proprietary information. For some, this is simply a corrective action in a market where governance has not kept up with the globalised industry. For others, such as Berlin-based social enterprise OpenOil, launched in 2011 and operating under the tagline “imagine an open oil industry …”,¹⁰ there may be a deeper vision at play of transforming the way natural resources are managed and the role that policy-makers and citizens have in their exploitation.

As government agencies gain new data, both public and private, from companies, the attention of civil society and researchers has turned to the lack of cross-agency data sharing. Various government agencies tend to obtain different types of data which can all be of value when, for example, assessing tax payments and the other contractual obligations of companies. But if the data from different agencies is not brought together, opportunities to use it may be missed. In a survey of government officials in African resource-rich countries, OpenOil identified, in particular, that project costs, reserves, and production data were identified as areas where the gap between the perceived need for joined-up data and data availability was the most pronounced.¹¹ More research is necessary in order to determine the appropriate limits for how much data should be made public, but, in the meantime, models for data sharing between trusted agencies could be further advanced. In support of this, recent research in the mining sector noted that “Revenue authorities could improve their analysis of risks through sharing production data, findings from cost audits, mining agreements, and information on beneficial owners as a matter of course rather than just before a tax audit.”¹² Concretely, the African Tax Administration Forum has indicated that such work is being explored.¹³

Critical to the use of extractives data is an emerging ecosystem of platforms, infomediaries, and capacity-building activities.⁴⁸ Organisations, such as OpenOil, have deployed the Aleph platform that enables the search of security exchange filings from extractives companies, which helps data users to find the needle in the haystack of disclosure. In 2016, PWYP, in partnership with OpenOil, launched a global Data Extractors programme dedicated to building skills among CSOs working on extractives accountability issues,⁴⁹ and the WRI has released mapping platforms, Global Forest Watch and Resource Watch, which provide access to raw data and visualisations from public domain mining datasets.⁵⁰ All these activities are illustrative of the comparatively (to other sectors) well-resourced environment for data use in extractives. The following sections outline four different ways in which data is being put into use.

In this chapter, we have highlighted trends in data collection and use which are starting to have an impact at both a global and country level. We see more robust analysis, more data-driven experimentation, and improved cross-sectional work to build skills among extractives-focused NGOs.

The extractives sector remains a highly volatile and capital-intensive sector. While there are different opinions on the pace, there is broad agreement that the sector will continue to invest heavily in new technologies, digitisation, and automation in the future.⁸¹ The World Economic Forum has also identified a wide area of technologies which could impact the extractives value chains, including artificial intelligence, robotics, privately owned trading platforms drawing on distributed ledgers (blockchain technology), and automation across the supply chain.⁸² This indicates that, while progress has been made in the last decade on data openness, private sector investments in data collection and management across extractives operations is likely to expand. At this stage, however, it is unclear how CSOs, activists, journalists, and policy-makers will be able to keep up with the pace of change to maintain access to data and to scrutinise a rapidly changing market.

As the extractives transparency movement heads into its third decade, and extractives open data enters its second, there is a need to draw on learning so far to sharpen our focus on clearly defined problems. Improvements are needed in providing reliable data, generating analysis which can address timely policy challenges, and finally getting the analysis and policy recommendations into the hands of broad-based civil society campaigns, journalists, and, ultimately, policy-makers. There are major benefits to be sought from linking up the sector with other “Follow the Money” efforts,⁸³ building alliances to trace funds across extractives, budgets, contracting, aid, and service delivery. While substantive research is yet to emerge beyond initial mapping,⁸⁴ there are opportunities here to go beyond breaking down silos in a single sector and to build bridges between groups concerned with fundamental questions of how public resources are managed. There is also a need to strengthen links with sustainability and climate change networks. Notable efforts are being made to connect analysis of extractives revenues, fiscal policy, and implications for climate change, for example, in areas such as fossil fuel subsidies⁸⁵ and through the Green Fiscal Policy Network.⁸⁶ Lastly, the emergence of the Task Force on Climate-related Financial Disclosures has led to the development of a framework for company disclosure of climate risks with commitments from 1 800 major companies.⁸⁷ Moving forward, these climate disclosures will be an important source for analysis.

As this chapter has shown, when machine-readable open data is available, infomediaries, civil society, and governments can leverage it to develop analysis and evidence-based policy recommendations. Yet, it is important to recognise that the road from data acquisition to analysis and policy impact remains highly complex and dependent on many factors, such as the political context, the capacity of civil society and other potential users, and opportunities for decision-making based on the results of analysis. The extractives sector will be able to continue to learn from peer research across the open government space.⁸⁸ Scaling up the use of extractives data is not necessarily about building mass-movements or substantially increasing investment in open data, but it does involve supporting the development of specialist skills among key stakeholders and leveraging those skills effectively, for example, by expanding work on fiscal modelling, which, if enabled by open data, would bring extensive domain expertise to bear on specific weaknesses in extractives governance. The future of open data in extractives will also rest on how well regulations for disclosure are implemented in practice by national agencies and how far multi-stakeholder initiatives advance interoperable disclosure requirements. The importance of getting the technical definitions and details right cannot be underestimated, and this will demand much deeper collaboration between open data and standards specialists, policy advocates, and policy-makers.

Overall, if the multi-stakeholder approach that has characterised extractives governance work over recent years can be sustained, and open data specialists can be further integrated into the process, then the next decade should see open data become an unprecedented tool for extractives governance.

Approximately 80% of all government data contains some reference to location.

Opening up geospatial data was a key early driver of open data advocacy, and there has been significant progress in opening up this type of data. However, much of government geospatial data remains under restrictive intellectual property agreements.

Work on open geospatial data technology and infrastructure pre-dates the concept and implementation of open data, yet there are relatively weak links between the open geospatial and other open data communities. Stronger links could build critical capacity for spatial analysis within open data communities.

Mapping visualisations are a popular way of presenting open data, yet the spatial analysis carried out is often unsophisticated. Relationships that appear on a map may not be statistically significant. It is important to recognise that geographic relations can be shown in other forms, such as tables and charts.

Fifteen years ago, most users experienced online maps much as they might their paper counterparts: flat non-interactive images for browsing geography. In 2005, Google Maps changed that, giving rise to enthusiasm for the mapping mash-up, where data (often taken from public datasets) is located on an interactive scrollable and zoomable map. A year later, OpenStreetMap was launched, providing a platform for the collection and display of mapping data, unencumbered by intellectual property (IP) restrictions, and launched in response to ongoing frustration at the lack of open geographic data in the United Kingdom (UK).¹ The move from large proprietary desktop Geographic Information Systems (GIS) to increasingly open access to geospatial² data appeared to be underway.

Mapping visualisations have been strategic assets in the popularity of open data and they remain one of the public entry points to engage with open data. A typical mapping portal from the City of Phoenix³ in the United States (US) demonstrates the type of geospatial data (and prepared maps) available through a typical North American municipal data portal, including property boundaries, zoning information, traffic volumes, and recreation areas (see Figure 1). A similar site can be found for Manchester, England,⁴ although geospatial data and map access come with terms and conditions that restrict how that data can be used.

Both the potential demonstrated by mapping mashups and user interfaces and the desire for access to valuable geospatial datasets held by governments and government agencies can be seen as driving forces in the development of the open data movement. But what of geospatial data today? Is the data now widely open, accessible, and used? And what progress has been made in unlocking the potential of geospatial data for analysis and improved policy-making?

While much progress has been made in the availability of data, and in the development of tools to visualise it, substantial work is needed to better connect geospatial and open data communities, to equip creators and users of geospatial data with the critical skills (and technical platforms) needed to move beyond simply mapping, and to gain the full benefits of geospatial data analysis. There also are significant risks from the wider use of geospatial data that need to be more directly addressed. Ultimately, advances made in terms of sheer data availability and infrastructures are currently counterbalanced by significant stalemates in terms of analytical approaches to geodata, as well as ownership and privacy risks.

It is estimated that 80% of all government data has some reference to location.⁵ Almost every chapter in this volume touches upon geospatial data in some form. Geospatial content can be found in datasets on subjects as diverse as parks, refugee camps, financial transactions, natural resource distributions, and socioeconomic statistics. Many uses of open data rely on being “mapped” (i.e. attached) to basic geographic framework data.⁶ For example, socioeconomic statistics, like population, may be mapped on top of administrative boundaries. Data on soil quality may be attached to digital elevation models to model erosion, and that same soil data may be compared to geographically intersecting data on land ownership and land subsidies. Without their geospatial component, many open datasets would have much reduced impact.

Mapping generally involves presenting geospatial data alongside a geographic layer. Geographic layers are datasets that are essentially outlines and may or may not be open data themselves. These layers include jurisdictional boundary files (e.g. country, city, school catchment areas, and watershed districts) or linear features like rivers or roads. For completeness sake, there also are geographic point layers, such as centres of cities or locations of known elevation like mountain peaks. Geographic layers may also include remote sensed imagery. Imagery can function as a backdrop onto which geospatial data is overlaid (e.g. logging operations in forested areas). Like other geospatial data, remote sensed imagery can be analysed alone or in combination with other open datasets to identify areas of drought, land use, or pollution.

Many practitioners working with open data consider geography primarily in terms of x and y coordinates, usually expressed as latitude and longitude, respectively. It is important to recognise that there are numerous types of “coordinates”. These include direct location references such as latitude/longitude, postal addresses, or GPS traces. There also are indirect references to location, such as place names (e.g. colloquial neighbourhood names, or official country or region names) that can be turned into a set of coordinates using a gazetteer or a lookup database.

The vast diversity of geospatial data may be more or less open along a number of dimensions. Data may be free to browse but not to download. Or data may be free to download but provided under restrictive licences that limit reuse. Or data may be openly licensed but only available in formats that require proprietary software or that use proprietary referencing systems. To understand open geospatial data, we need to ask: What kind of data is this? and How open is it?

The last decade has seen substantial strides in opening up geospatial datasets. Evidence suggests this has brought significant social and economic value. For example, in 2013, the Government of Denmark, through their Basic Data programme, released digital mapping data free under an open licence. A follow-up study in 2017 estimated that this had led to DK 3.5 billion (approx USD 495 million) in socioeconomic value in the preceding year.⁷ It is estimated that making the US LandSat satellite imagery freely available in 2009 accrued USD 1.8 billion annual value to the economy; whereas, charging for access would lead to substantial inefficiencies and loss of value.⁸ In the UK, open data policy has led to new datasets being made open from their mapping agency, the Ordnance Survey. The release of geospatial data responded to advocacy that focused on gains to the economy from a more open approach to this data.⁹ It has long been argued that Canada suffered significant losses due to government’s early reticence to open geospatial data,¹⁰ which is being remedied.

In the US, efforts to open up federal geospatial data pre-date most consideration of open data worldwide. The federal government, as well as subnational jurisdictions of the US (states, cities), tends to publish geographic datasets as integral parts of their open data portals. The reason that geospatial data is arguably the first open (government) data is due to the establishment of national or subnational spatial data infrastructures (NSDIs), the first one being the Australian Land Information Council in 1986.¹¹ NSDIs are outgrowths of “the technology, policies, standards, and human resources necessary to acquire, process, store, distribute, and improve utilisation of geospatial data”.¹² Geospatial data infrastructures tend to require high levels of interoperability in terms of standardisation to function. These datasets likely originate in different agencies with varying practices of data collection, update schedules, and definitions. Full standardisation requires geospatial data to be at the same geographic projection with the same coordinate system, spatial extent, updates, and data definitions. It is by no means easy to coordinate data so that layers “lie on top” of each other in alignment.

Spatial data infrastructures did not necessarily originate as open platforms. Many were designed as government-to-government data sharing platforms, although several promoted the idea that the data should be accessible to a range of applications and support economic development. Openness of geospatial data remains uneven across the world. The latest Open Data Index¹³ identifies just 12 countries where governments provide fully open national geospatial data, and only one (Brazil) is not in the World Bank’s “High-Income Economies” category. There is movement among numerous countries to increase openness (e.g. Indonesia’s widely discussed One Map initiative). Progress has been slow and mostly focused on rationalisation of geospatial data management. Opening up geospatial data is not simply a matter of applying a licence to existing datasets, but also involves the adoption of policies, standards, and human resources specific to geospatial data.

Encouraged by the International Open Data Charter, and noting the value of an “open by default” approach, the Group on Earth Observation adopted open data principles in 2016,¹⁴ seeing this as the natural step forward from their existing data sharing regime (established in 2006) and justifying this shift on the basis of the economic, social, governance, education, research, and innovation value.¹⁵ The European Union’s (EU) INSPIRE¹⁶ directive has driven the inclusion of geospatial data features in a number of national data portals and extensions for geospatial data to the open source CKAN software.¹⁷ Many NSDIs have had little integration into the open data landscape. However, the EU’s initiative demonstrates how governments may integrate parallel tracks of activity between the open data and geospatial communities.

Gaps in geospatial data are increasingly addressed through the use of cross-border satellite imagery available on digital earth mapping platforms. Some of this data is sourced from government. The launch of the Africa Regional Data Cube in May 2018 resembles many features of an NSDI in terms of standardisation and provides access to free satellite imagery for Kenya, Senegal, Sierra Leone, Ghana, and Tanzania. It builds on an open source “data cube” platform that compresses pre-processed imagery to reduce the otherwise prohibitive costs of data transfer, storage, and analysis.¹⁸

Government data also is being augmented by the private sector and civil society, and some of these new geospatial datasets could become open data. Firms like DigitalGlobe provide imagery derived from commercial satellites. Whereas satellite coverage may be universal, street mapping remains limited by either the availability of non-proprietary street-mapping data or volunteer contributions. Much of this data is licensed to proprietary platforms like Google Maps. Users can zoom into most places on Earth and see road layouts or satellite imagery. To access the same data on other platforms to support applications or analysis can often be prohibitively expensive. For instance, software application programming interfaces (APIs) may be available but based on per-access pricing,¹⁹ or sudden price changes may leave data out of reach of users seeking to map open data coordinates or build open data-related applications and businesses.²⁰ It is important to remember that free to use, but non-open, platforms are subject to prevailing business models of tech industries. Parts of Microsoft’s Bing mapping division were sold to Uber in 2015, and Google increased prices for its mapping APIs up to fourteenfold in 2018. There is a precariousness to basing one’s mapping applications on a specific non-open platform. Fortunately for data consumers, the last decade also has seen the emergence of tools like Leaflet,²¹ which enable digital mapping using a variety of geospatial data providers. Companies like MapBox²² provide a commercial offering but are committed to building on top of open source tools and data.

Open geospatial data also is being created through crowdsourcing. The largest platform, OpenStreetMap, “is built by a community of mappers that contribute and maintain data about roads, trails, cafés, railway stations, and much more, all over the world”.²³ By comparing CIA World Factbook data on road length in a country with OpenStreetMap data, Maron and Channell found that some countries have 100% coverage of major roads.²⁴ In Asia and China coverage is more limited. In India, for example, only 21% of the road network has been digitised on OpenStreetMap.²⁵

Use of private or crowdsourced data reflects the costs of collection and maintenance of geospatial data and related infrastructures. When geospatial data is funded directly from government budgets, rather than through cost-recovery (i.e. charging users for use of the data as a method of supporting government data collection and maintenance), access is at greater risk of budget cuts.²⁶ This can lead to pressure from agencies working with geospatial data to develop or retain financing regimes. The cost of data collection has led a few governments, particularly in North America, to explore partnerships with private sector firms to collect data through projects, such as Google Waze, Strava Metro, and Uber Movement.²⁷ Ironically, these datasets frequently originate from civil society or individual citizens, but ownership is claimed by the firms providing the platforms for data collection. This can introduce new sources of proprietary data in spatial data infrastructures at the same time that other aspects of those infrastructures may be opening up. Additionally, the inclusion of privately sourced or crowdsourced data invariably shifts control from government in terms of data accuracy, coverage, and timeliness of edits and updates. This will increase the risk to governments (real or perceived), particularly if that data is central to government operations.²⁸

For all the progress that has been made in terms of data openness, four issues present notable challenges for work with open geospatial data.

First, numerous countries face challenges in opening key datasets due to IP restrictions. The UK’s mapping agency, the Ordnance Survey, and postal service, Royal Mail, have long been restricted in how they can open up their geospatial data due to Crown Copyright. Ownership of all or part of the IP was further complicated when the management of the postcode database was outsourced to a private firm. The situation shows signs of improvement with a 2015 open data policy supporting a “presumption to publish”.³¹ However, efforts to create an open address register for the UK have been put on hold, which places this critical lookup dataset out of the reach of many open data projects.³² CanadaPost has maintained strict IP protections on its postal code database. In Canada, a one-person firm, Geolytica, built an application that would reverse engineer Canadian postal code boundaries using computational geometry and crowdsourcing. It was done as a proof-of-concept, but the database was also opened up to the public. Geolytica’s efforts led to it being sued by CanadaPost for violating the latter’s ownership of the phrase “postal code” and the underlying content.³³

The value of spatial data as IP means that firms are often interested in acquiring exclusive rights to it. Another example from Canada illustrates this. The Ontario-based firm, Teranet, purchased the rights to land registries (cadastres) around the world. In exchange for those rights, the firm maintains the registry datasets and then licenses access back to local and regional governments.³⁴ This represents not just private provision of the service but private ownership of the data. There is a paucity of reliable data on how many countries have substantial private ownership of IP in their spatial data infrastructure, yet this is likely to be an important area to track over the coming decade if further gaps are to be avoided in the open geospatial data landscape.

A second key challenge relates to privacy and security. When it concerns data about individuals, location data can often pierce privacy protections and enable surveillance. A combination of just three variables (i.e. gender, birthdate, US zip code) has been found sufficient to identify individuals by name in the US.³⁵ Individuals increasingly leave geographic data traces on the web through their use of fitness trackers, location-stamped photographs, or a myriad of other location tracking apps. The existence of this data can jeopardise the anonymity of other datasets that might contain coinciding location and timestamps. Methods exist to maximise privacy while preserving the ability to analyse data (e.g. through geographic masking).³⁶ However, the ability to deanonymise data will only improve as artificial intelligence and machine learning are applied to open data.³⁷ Whereas open datasets generally do not describe individual persons, the growing availability of geo-indexed data needs to be accounted for when creating, sharing, and using open datasets.

Standardisation presents a third major challenge for greater interoperability in the world of geospatial data. The most commonly used standard for geography is the “atomic standard” of the coordinates, latitude and longitude. Multiple alternatives exist to lat/long (e.g. polar coordinates are better for people near the poles). Considering coordinate systems requires contemplating standards in geographic projections. Inconsistent projections prevent one dataset from correctly being overlaid onto other data layers and may inhibit other operations like calculating travel distances. Polygons like jurisdictional boundaries also generate complexity related to standards. The schema.org standards for place, which contain at least ten different relationships of containment, overlapping, intersection, and equality between areas, provides a sense of how complicated it is to structure geometries beyond simple point locations.³⁸ Maintaining the quality of geographic data and ensuring standards are adopted correctly is not trivial. Unlike other sectors, the problem is not the availability of standards (e.g. the Open Geospatial Consortium maintains over 30 open standards for geographic data).³⁹ We need an educated understanding about their adoption. Instead of creating an integrated world of geospatial data, open data initiatives could lead to a soup of misaligned points and polygons that are difficult to distinguish.

This leads to the last challenge: the lack of interaction between open data communities and the communities that traditionally work with geographic data. Open geospatial data (via WAIS servers, NDSIs, and Al Gore’s articulation of a Digital Earth⁴⁰) predate the concept and implementation of open data. Open data advocacy in several countries was sparked by a desire for geospatial data as in the UK FreeOurData campaign⁴¹ and Canada’s DataLibre.⁴² Nonetheless, there has been a gulf between the early open data movement with its focus on quantity over quality and the geography/geomatics community, which by 2010, was already well established and considering issues of standardisation and data management. We have seen plenty of missed opportunities to bridge the gulf, which has resulted in a bifurcation in skills for geospatial data handling that impedes both the opening, and the effective use, of geospatial data. In particular, this has led to the open data world’s focus on mapping but very little focus on geographical analysis. There remains considerable potential for increased interaction between the two communities to enhance skills and analysis.

Mapping is undoubtedly important, but visualisation of data is just one strategy of many. There has been a tendency among open data practitioners to map and make inferences based on visual inspection of geospatial datasets. However, these ostensible relationships are often not statistically significant. The ability to map open data in the absence of the critical skills to analyse it correctly can lead to problems and even incorrect policy prescriptions. Expanding skills for detailed spatial statistics and analysis, to allow conclusions to be drawn from open datasets and to create new, improved maps based on the results of that analysis, should be a high priority in the open data community. General data literacy capacity has grown, but the availability of tools, resources, and outreach to promote geospatial data literacy is much more limited. The current lack of analytical capacity represents a critical bottleneck to the effective use of open geospatial data.

For example, one large part of open geographic data handling concerns what is known as “feature geometry”. Most open data containing geospatial attributes is point-based. That is, an entity’s location (e.g. a park, a government transaction, a building project, or a refugee settlement) is represented by a single x, y coordinate. The choice of which points to use is not always obvious. Should the location be a headquarters of a local relief agency or the location where activities are occurring? Many of these points reflect what is called a central tendency or the centroid (a geometric centre of an area). Depending on the shape of the area (e.g. a crescent), a centroid could actually appear outside the area. The simple consideration of which location is mapped can affect the message a map communicates.

Numerous forms of analysis should not rely on point location at all. Many features, such as the geographic distribution of poverty or of crop types, are not natural distributions, easily interpreted through the use of latitude and longitude, but are shaped by politics. Such features are more appropriately described by areal measures. For example, poverty should be reported by the political boundary of a township. Unlike geographic points, working with jurisdictional data can be difficult because boundary file availability and discoverability are limited and there may be disputes over borders. Tools for working with containment (polygons) are less user-friendly, in many cases, than those for generating point-based online maps. Similar issues exist for raster datasets (e.g. satellite imagery), which are especially important for rural areas.⁴³ Working with raster data, whether it is satellite data or drone data, generally requires more extensive experience and expensive software than other types of data.

A common alternative to mapping by jurisdiction is through aggregation and clustering. Two popular aggregation methods are hexagonal binning (hexbins) and rectangular grids, which rely on the use of regular artificial areas into which points are counted. A different approach is clustering points through hotspot analysis, which infers the geospatial extent of a phenomenon (e.g. a cluster of disease outbreaks) and differentiates statistically significant clusters from non-significant clusters. Many tools can now automate aggregation and clustering, but tools need to be accompanied by a critical understanding of the way the choice of approach affects analysis. Geographers have widely discussed the modifiable areal unit problem (MAUP)⁴⁴ whereby aggregation units are understood as definitionally artificial and the results of data aggregation depend on the choice of the unit. Results (e.g. counts, rates, densities, and correlations) are influenced by the shape and orientation of the unit (e.g. slight tilting or enlarging of a rectangular grid), as well as by the way the units are combined (scale). O’Loughlin et al. (2014), for example, use open data on a rectangular grid to map violence, heat, and precipitation across the African continent.⁴⁵ They note limits in the data and its aggregation, even as they perform analyses at a finer aggregation than previously conducted to better understand climate conflicts. Tools exist to improve data literacy with regard to problems introduced by spatial aggregation.⁴⁶ The challenge is promoting their adoption outside the geography community and within the much wider community of open data users who may otherwise adopt naive analytical strategies. No aggregation is perfect, including those using jurisdictional boundaries. It is important to broaden critical understanding of the malleability of aggregations in the results they deliver.

This noted, we must be aware that improving the quality of analysis of geospatial open data can be knowledge and resource intensive. For example, AidData’s infrastructure for sophisticated geospatial analysis of international aid patterns is expensive to maintain and requires substantial annual resources.⁴⁷ Although Google has instituted a business model for Google Maps, organisations like AidData cannot rely on similar mechanisms of support.

As we look to the future, opportunities lie in better connecting the open data and geospatial data communities. The latter has been working on improving open source geospatial data tooling for many decades. Even though much of this work has been focused in particular professional contexts, critical and community geographers have long been working on ways to open up access to, and support popular engagement with, geospatial data. The extensive learning and thinking within this field should not be ignored in the rush to open up data and excitement over the latest commercial tools and simplified mapping platforms.

Major advances have been made in open geospatial data. However, numerous gaps remain related to IP, standardisation, privacy, and analytical capacity. In the next decade of open data, we need to ensure greater coordination between the geomatics/GIS and the open data communities so better maps can be produced and greater value can be demonstrated from the wealth of geographic content within the open data released in the last decade.

More than anything, anyone working with geographic open data should approach it with a critical eye and ask two questions. Which choices have been made in creating this data? What lessons might there be from the existing geospatial data community to help with the analysis of this data?

Opening up data on government finance has been a major focus of open data advocacy with projects like OpenSpending bringing a data-driven approach to work on fiscal transparency.

Opening up public finance data requires a whole set of conditions for success, including government capacity, access to technical platforms and standards, and in-depth engagement from civil society, to help make sense of complex financial data.

When better connected to grassroots advocacy, open data approaches to government finance can help re-energise global budget transparency work.

Working to ensure the transparency of government finances has a long history. By 1850, many countries in Europe had already enacted constitutional requirements that government budgets or accounts be published, leading to what Irwin¹ refers to as an “avalanche of data” that was sparked, in part, by “rulers’ need to persuade creditors to lend and taxpayers’ representatives to approve new taxes”. However, this avalanche of annual accounts, published in printed paper reports, seems miniscule when compared to the data on government finances that could be made available today. When the East Asian financial crisis hit in 1997, fiscal transparency was firmly placed on the global agenda, and principles were put forward calling for disclosure of information across government operations, not just budgets.² And as the open data movement has developed over the last decade, it has brought a particular focus on transparency in government finances, adding a particular digital spin to advocacy and calling not only for data but for machine-readable data that is ready for public analysis.

Public finances are ultimately at the heart of government activity, constituting one of the main levers of public action through which governments shape society. The study of public finance may historically have been regarded as a question of simply determining the income and expenditures of governments. However, since the middle of the 20th century, this has expanded to recognise the role that taxes and spending play in shaping the wider economy (e.g. taxing activities that may have negative consequences and spending that may stimulate economic development and trade, including research grants or development aid). As such, citizen scrutiny and a clear understanding of all aspects of public finances is crucial. Debt, taxation, contracting, grants, and subsidies are all topics to be covered within the context of fiscal transparency, alongside more obvious themes of budgets and expenditures. With the right mechanisms in place, improved citizen understanding of the state’s fiscal behaviour can encourage greater civic participation and oversight, can promote public accountability, and, most importantly, can potentially enhance the effectiveness and efficiency of public budgets and spending.^3,4

From the start, the open data movement has placed an emphasis on government finances with projects such as the 2007 “Where Does My Money Go” prototype (see box below) that demonstrated the potential of open data in this sector. Over the last decade, civil society and government-led projects around the world have sought to make public finance data more accessible with initiatives on almost every continent. However, the latest findings from the Open Data Index⁵ and Open Data Barometer illustrate that just 10% of surveyed governments publish fully open budget data (12 countries in total) and only 3% publish disaggregated open spending data (just 4 countries).⁶ In some countries, such as the United Kingdom (UK), an early publisher of spend data, reliable data availability has not been sustained, and it is not clear how far citizens have engaged with the data that has been made available.⁷

A decade into the new wave of open data-driven financial transparency, it is important to take stock of progress and to ask whether efforts to open up financial data have delivered results or whether activity is beginning to stall. This chapter takes a look at the arc of activity since 2005, taking stock of the state of initiatives, issues, and communities related to open government finance data.

In 2005, a trio of data journalists launched FarmSubsidy.org with the goal of facilitating access to information on the subsidy payments under the European Union (EU) Common Agricultural Policy (CAP). The platform, constructed with data accessed via freedom of information (FOI) requests to governments across Europe, made structured data accessible to search and explore, providing detail not only on subsidy payments, but also the details of the companies who receive subsidies. Danish journalists were able to use this information to challenge the dominant political narrative that the CAP supported primarily the poorest farmers by showing that it was actually large landowners and agri-businesses that received the most funds.⁹ By 2009, EU member states were mandated to publish their subsidy data, removing the need for FOI requests, although, even now, the data is not always available in machine-readable formats. The growth of the project played a key role in demonstrating the value of data-driven public finance journalism and attracted interest from a range of funders, including the Hewlett Foundation.¹⁰

In 2007, the Open Knowledge Foundation’s Jonathan Gray developed the idea for “Where Does My Money Go”¹¹ as a visual breakdown of the UK budget, tapping into a growing appetite for both data visualisation and open data ideas (see Figure 2). In 2008, the project was a winner of the UK Government’s “Show Us A Better Way” competition and had soon secured grant funding from government to develop a working prototype.¹² Further funding from a UK state broadcaster (4IP), the Open Society Foundation, the Knight Foundation, the Hewlett Foundation, and the Omidyar Network enabled the evolution of the project into the global Open Spending platform,¹³ which now hosts elements of fiscal data from at least 70 countries. The nascent community related to the project was not comprised of accountants or public finance experts but rather civic hackers and citizens interested in making complex government finance more accessible and supporting wider citizen engagement.

By 2010, more governments were starting to explore the direct publication of machine-readable budget data, reducing the need for citizens, organisations, and projects to manually scrape data out of documents and PDFs. The United States (US) government’s USASpending.gov, originally created in response to legislation passed in 2006 requiring all “federal contract, grant, loan, and other financial assistance awards of more than $25,000 to be displayed on a publicly accessible and searchable website to give the American public access to information on how their tax dollars are being spent”,¹⁴ went through a number of relaunches in 2009 and 2010 with increasing emphasis placed on the availability of downloadable open data and enhanced granularity. Although the site had provided an application programming interface (API) since 2007, it was the addition of downloadable open data in subsequent versions that gained it an increased profile.

Intense policy competition between the UK and US during this period may be behind the UK government’s 2010 publication of the COINS (Combined Online INformation System) dataset,¹⁵ providing detailed “fact tables” that presented disaggregated spending data from across the public sector. The Guardian newspaper was one of the early users of this data, creating a public data explorer interface to help citizens search the large dataset and working with Open Knowledge Foundation to use citizen research and FOI requests to fill gaps in the data, particularly around individual supplier names.¹⁶ The Guardian went on to write a number of stories based on their analysis of the COINS data and used its release to explore gaps in the quality of public financial management in the UK.¹⁷ In parallel, government departments and local authorities were asked in a letter from Prime Minister David Cameron on 31 May 2010 to publish details on all expenditures over GBP 25 000. The letter also committed to the online publication of information on all new central government contracts and all international development project spending over GBP 500 from January 2011 onward.¹⁸

Latin American governments also took a lead during this early wave of government activity. In Mexico, the first budget dataset was published in 2011 by the Ministry of Finance as part of a project under Mexico’s Open Government Partnership (OGP) action plan.¹⁹ The portal that was created published basic information about federal programmes with quarterly updates on the money spent, information on external evaluations, and a matrix illustrating progress toward planned and achieved goals. The intent of this project was to provide a place where both citizens and decision-makers could find government finance data in a unified format. An OGP case study credits the portal with generating “commitments from the Federal Public Administration to make progress on public projects and initiatives which [had] fallen behind”.²⁰ Although such portals could theoretically be created without using open data, taking an open data approach helped to provide Mexico with a common format for aligning data from different departments and agencies, supporting the integration of information that originated from many different IT systems.

Although Brazil launched a National Transparency Portal in 2004, Beghin and Zigoni (2014) documented²¹ that it was not until the passage of an Access to Information Act in 2011, establishing procedures for federated entities to follow in the disclosure of information, that access to government finance data increased. However, they note that, in 2014, there was still a long way to go before all budget and spending data would be accessible in machine-readable form.

It is no surprise then that a World Bank study in 2013 cited the UK, Mexico, and Brazil as members of a small pioneering group of countries working to provide good access to reliable open budget data from financial management information systems.²² The full list of countries noted included Brazil, Germany, South Korea, Mexico, New Zealand, Spain, Sweden, the UK, and the US. They all have a high Open Budget Index score (above 60)²³ and OGP commitments to promote fiscal transparency in common.²⁴

In this pioneer phase, we can see how the interaction between select journalists, civil society, and governments spurred action to make more granular and machine-readable data available on government finance. But whether or not this data can be used to answer questions like “where does the money go?” and whether these early publication projects are sustainable depends on a much wider network of actors and activities.

While many of the most prominent actors working in the area of open budgets are intergovernmental organisations, international NGOs, and multi-stakeholder initiatives, according to Gray’s analysis of the open budget data landscape, the active grassroot community is composed of a myriad of international and local CSOs involved in open government, government transparency, aid transparency, open data, and related topics.²⁵ The way these groups are making use of public finance data is innovative and ingenious and reflects a limited but dedicated citizen interest in understanding how public money is spent. For example, in Nigeria, BudgIT²⁶ has worked since 2011 on creating infographics that explain elements of the budget and, since 2014, has used Tracka to crowdsource information on the progress of development projects in local communities. Communicating through social media, mainstream media, and community outreach, BudgIT reports reach over 4 million Nigerians with their information.²⁷

In an effort to help scale up innovations, equip organisations with open source tools, and improve data literacy around spending data, the Open Knowledge Foundation launched the OpenSpending project in 2011.²⁸ Its vision was to provide a central database of budget and spending data, as well as to build a community of groups and individuals who could work together to acquire, use, and add their contributions to the platform. From its launch, the resources made available increased substantially as the project grew, including a spending data handbook,²⁹ an open-source CKAN data portal with extensions,³⁰ a visualisation library based on Where Does My Money Go,³¹ and a data specification called the Fiscal Data Package.³² As of November 2018, OpenSpending contains government finance datasets from over 80 countries, although at varying levels of granularity and timeliness.

The tools provided by OpenSpending have been used by different civil society projects and platforms to provide citizens with accessible and user-friendly budget information (e.g. the German project Offener Haushalt, the budget explorer tool in Kosovo, and the Open Budget platform in Ukraine).³³ Many other organisations have developed their own technology and visualisations. This is the case for the Open Key project in Israel, the Open Spending portal in the Netherlands, the Vuleka Mali project in South Africa, and the Dónde van mis impuestos platform in Spain.³⁴ A key driver for the editorial and technological choices of these projects has been the goal of building visualisations that reflect the needs of citizens and a desire to embed data within a pedagogical context that provides education on government finance.

Between 2013 and 2017, as the community grew, many more projects and platforms emerged from civil society organisations (CSOs), some of them with the specific objectives of using public finance data for investigation in journalism or to enhance civic participation. One notable data journalism project using public finance data is Spending Stories, a project by the former data journalism agency J++ that was developed in 2013 to allow comparisons between big and small amounts of money to give users a context to understand how money is being spent while referencing original news stories.³⁵ The Farmsubsidy.org network has also continued to play an important part in building data journalism capacity related to open financial data, giving rise to the annual European investigative journalism Dataharvest Conference that now brings together as many as 400 journalists, coders, and scholars from all over Europe each year.³⁶

As Gray’s map of the linkages between open budget data-related websites from 2015 suggests (Figure 3), it is also important to recognise different sub-communities working in the open finance data domain. As well as local groups, there are a number of overlapping global communities of practice, some with specific thematic areas of focus. Examples are the International Aid Transparency Initiative (IATI), the Extractives Industry Transparency Initiative (EITI), and others looking at particular sources of data, such as the Open Contracting Partnership which has, since 2015, developed a global network of governments, civil society organisations, and companies working with data on public procurement to enable a different way to “follow the money” that complements budget and spending data. Gray’s 2015 mapping does not, however, capture groups working in the area of tax justice. Since 2017, the Open Data for Tax Justice network has sought to put more focus on companies reporting the tax payments they make to government,³⁷ which, once again, fills in another part of the complex government finance picture.

Although they do not feature heavily in Gray’s mapping, we should also not ignore private sector actors. Firms like SpendNetwork³⁸ clean and re-package government spending data for firms interested in securing government contracts, and there is some evidence to suggest government spending data feeds into a range of other private sector products. This said, more could be done to understand the role of the private sector in this field.

It should be clear from the examples above that there is widespread interest in, and engagement with, open data on government finances. Networks like the FollowTheMoney network³⁹ host regular community calls to connect organisations working on different parts of the governance finance puzzle, and groups like the Global Initiative for Fiscal Transparency (GIFT)⁴⁰ place an emphasis on open data as part of wider fiscal transparency reforms. Yet there remain many shared practical challenges that mean the vision of timely, accessible, and accurate open data on government finances is far from fully realised.

Even though much progress has been made in opening public finance data, some gaps remain, including those related to policy and high-level commitments, technical platforms for data, linking data to decision-making, and challenges in encouraging the use of data.

In the years ahead, the key challenge will be to better connect the current wave of the open data-driven transparency movement with other grassroots advocacy networks and government decision-makers. When it comes to securing impactful results from open government finance data, the evidence suggests that projects will require unique partnerships between technologists, CSOs, and government. This is the model followed in South Africa with the Vuleka Mali project, a partnership between the National Treasury and a coalition of CSOs called Imali Yethu to make government budget data and processes accessible to all citizens and interested parties. Their motto, “We aren’t interested in transparency for transparency’s sake”,⁵⁵ should be one that more organisations place at the heart of their thinking. Technical work on government financial data also needs to connect with wider social agendas. For example, Carter notes that the potential exists to apply a gender lens to budget analysis;⁵⁶ however, we have not yet found open data projects that directly apply a gender lens to open budget data creation and sharing.

Given the long history of work on opening up government finance, we should not expect a complete transformation in less than 15 years of open data activity. The vision of current advocates for open government finances is an ambitious one – to provide more granular data than ever before. There are signs, however, that when data is used, and governments are willing to open up, substantial progress can be made. In their brief history with the DATA Act, Sunlight Foundation has described how “reporting bad data drove reform” to secure new legislation, better processes, and ultimately improved data.⁵⁷ Rather than waiting for perfect data, it is possible to publish data and then improve it with subsequent iterations.

The foundations laid in the last decade in terms of technology platforms and data standards, and in terms of networks and communities, is impressive. Long-term, opening up and securing the use of government finance data will require significant resources in terms of technology, financial and human capacities, as well as time and strong political support. Not all the organisations explored in this chapter will have the resources they need for sustainability, but all have demonstrated what could be possible in their local contexts, and they have collectively re-imagined ways to engage citizens on governance finances.

Government finance data has played a key role in shaping the early development of the open data movement. The challenge for the decade ahead is to see how far, and to what end, open data advocates and practitioners can shape a sustainable ecosystem of open government finance data.

There is relatively limited awareness of open data in the health sector, where, given the focus on patient data, the idea of “open by default” does not resonate. It is important for initiatives to understand that data exists on a spectrum from personal and closed to non-sensitive and open.

Privacy concerns, a lack of fresh data, disjointed source systems, and usability problems have all hindered nascent open data initiatives in health. Initiatives have often failed to identify the high-priority use cases, driven by demand from multiple stakeholders, that would sustain the attention and investment necessary to help them overcome early challenges.

Open data that originates from health facilitates as feedback from service users can be used to improve performance or support researchers as input into policy; however, if feedback is not connected to action or if input meets political and resource constraints, it is hard to create a virtuous cycle of data publication and reuse.

The development of large public databases by government ministries, departments, and agencies (MDAs) has been ongoing in earnest in many countries around the world since at least the 1990s. The most basic of these government data systems are registers, supporting a range of government services, such as health insurance, social security, vehicle and business registration, and census-taking among many others. These registers form the basis of numerous vital public services whether the services are delivered electronically or not. Other systems are layered on top of these registers in order to support decision-making, planning, and policy-related research. To function well, many of these systems reside behind rigid security and multi-level authentication and authorisation protocols as they regularly contain very sensitive personal information about citizens.

Data about health is often considered some of the most sensitive information collected and held by governments and institutions. Yet over the last decade, there have been a number of initiatives focused on open data in the health sector. Broad et al. describe open data as “data made available by governments, businesses, and individuals for anyone to access, use and share”.¹ Clearly this should not apply to the detailed personal information within health registers. So, when it comes to open data and health, it is paramount to understand the particular data held within each system, to think carefully about the levels of access that different stakeholders may want or need, and to determine how, or whether, the data may be safely anonymised prior to publication as open data. To do this, it is useful to consider a data spectrum for health, to enumerate the different stakeholders creating and using data, and to consider the challenges they must overcome before open data in the health sector can evolve from being a minor sub-community and enter the mainstream.

As explored in the previous sections, technology has been a key driver of e-government and has resulted in substantial growth in the amount of health data available. The coming decade could see further dramatic developments in the use of technology in healthcare, and, consequently, the rapid expansion of data availability, especially with the trend toward big-data enabled healthcare. Potential open data users must be prepared for this expansion, while also ready to address the critical need for information governance. Most importantly, the orientation of open data projects must move from analysis to action to create an evidence base that can reveal the different components needed to secure meaningful impact on the health system.