The Critical Role of Observations in Informing Climate Science, Assessment and Policy

Scientific evidence of climate change is unequivocal. Human-induced climate change is already affecting every region of the Earth, with many experiencing more frequent weather and climate extremes. This conclusion was reached by the Intergovernmental Panel on Climate Change (IPCC), Working Group I (WGI) in its Sixth Assessment report (AR6, IPCC, 2021) by drawing upon a variety of datasets and reanalysis derived from climate observations (Figure 1).

Figure 1 - Subset of Figure 2.11 from IPCC AR6 WG1 showing trends and timeseries for global mean surface temperature reconstructed from land and marine meteorological observations. The HadCRUTv5 product includes substantial interpolation over data void regions. (Source:  IPCC AR6 Figure 2.11)

Observations are our primary source of information about climate change. The available historical observations undertaken by National Meteorological and Hydrological Services (NMHSs), although known to be incomplete, underpin our understanding of key climate processes and climate change. Long-term records from land and ship-based meteorological stations (Figure 2), radiosondes, satellites and other observational instruments provide the necessary long-term data to understand our rapidly changing climate. These data have been analysed using a variety of techniques to provide the robust scientific basis upon which to undertake scientific assessments and monitoring activities. 

Figure 2 - Centennial station – Sonnblick (Austria). Old photo from 1886, new photo from 2001

Without historical observations, it would be impossible to draw any firm conclusion on climate change. However, in many parts of the world, the available historical and present-day observations are insufficient to adequately monitor and predict the climate at regional and local levels. This is especially true for climate extremes, which tend to be more localized and short term. The available datasets under ClimDEX, using the climate change indices developed by WMO, have large gaps over many critical areas of the globe. In those areas, monitoring requires a much denser network of observations at daily or synoptic report resolution.

An assessment of climate extremes in several IPCC regions was not possible in AR6  owing, in many cases, to the lack of data available to the scientific research community. This means there is no observational basis for verifying future projections of changes in impacts in those regions and hence no means to effectively plan the necessary adaptation measures. While oftentimes this may be due to the scarcity of historical observations, it is undoubtedly the case that the existing historical data record is not made available to the scientific community. 

The Global Climate Observing System (GCOS) defines a set of Essential Climate Variables (ECV) that cover atmospheric, ocean and terrestrial components of the climate system, including meteorology, hydrology and the cryosphere. Observations of ECVs have many uses: 

• For monitoring the climate, detecting trends and providing information about the occurrence of weather extremes
• To enable reanalysis to provide long time series of consistent climate data for the past
• To improve our scientific understanding of the climate and develop climate model projections
• To deliver the information products needed for adaptation.

The Paris Agreement of the United Nations Framework Convention on Climate Change (UNFCCC), approved in 2015, aims to limit the impact of climate change by asking all Parties to the Convention for voluntary commitments to reduce their emissions of greenhouse gases (mitigation) and to improve resilience to the effects of climate change (adaptation). NMHSs can support those goals by providing climate observations and projections as needed for adaptation and other climate services. To do so, NMHSs must depend on and contribute to a system of free and open data exchange.

Many parts of the Paris Agreement require access to both historical and ongoing meteorological observations: 

• The goal of “holding the increase in the global average temperature to well below 2 °C above pre-industrial levels and pursuing efforts to limit the temperature increase to 1.5 °C above pre-industrial levels” requires observations to monitor its achievement and the impacts of mitigation measures. 
• The adaptation goal of “increasing the ability to adapt to the adverse impacts of climate change and foster climate resilience and low greenhouse gas emissions development” requires observation-based predictions of a changing climate. The IPCC has identified a lack of access to data as a significant issue for adaptation in some parts of the world, particularly in the African continent. 
• Determining greenhouse gas fluxes from observations (based on measurements of atmospheric composition) can guide Parties in their assessments of progress, and support reporting under the transparency framework. 
• Observations of land cover and above ground biomass are fundamental to supporting efforts to conserve and enhance sinks and reservoirs, including forests.
• Parties should enhance understanding, action and support of the loss and damage associated with the adverse effects of climate change. Observations are necessary to identify, attribute and predict extreme weather and slow onset events and are an essential part of emergency warning systems. 
• Informing the public of the state and future of the climate system. 
• Supporting the Global Stocktake, by reporting on collective progress towards aims and goals of the Paris Agreement. 

A long record of climate observations is needed with sufficiently high levels of quality and consistency to allow the detection of long-term changes embedded in diurnal, seasonal and multi-annual variations. The fundamental climate data records of “original” observations must be preserved indefinitely, even if they are not often used directly without further processing. Global estimates derived from observations such as reanalyses and other high-level data products are more commonly used to monitor climate change, to support policy development and implementation and to inform the public. Those global datasets are often downscaled to higher resolution in order to support local climate services. The entire value chain from observation to climate services, however, crucially depends on global availability and free and unrestricted exchange of observational data as well as model output and reanalysis data. New and improved climate datasets, including reanalyses and other innovative information products, will come and go – each benefiting in succession from new insights and capabilities – but they cannot be produced without sustained access to the original observations.

It is of fundamental importance that we share and preserve historical observations, manage them robustly and effectively, and make them available to all. The current and future generations of researchers must be able use and exploit these data to provide the products and services required for effective climate decision-making. 

Reanalysis 

Reanalysis has benefits for Numerical Weather Prediction (NWP) as well as climate studies. It has an important role in providing high quality and detailed data on the climate of the past and present which are required to support decisions for adaptation. It accurately represents low-frequency variability in several ECVs that have been reasonably well observed globally since the 1980s (Simmons et al. 2017). These include surface-air temperature, tropospheric and lower-stratospheric temperatures, surface-air humidity and precipitation. Reanalysis data also provide useful information about several ECVs that are not well observed, such as tropospheric winds, soil moisture, river discharge, and runoff (Dunn et al., 2020). Reanalysis is critically dependent on the availability of global high-quality observations. 

Global reanalysis using NWP Infrastructure  

The evolution of the global observing system, including the necessary infrastructure and protocols to enable data exchange in near-real time, has made it possible to implement increasingly skillful NWP systems that use observations to initialize global forecast models.   Forecast skill strongly depends on the availability of both ground and space-based observations sensitive to key meteorological variables such as surface pressure, air temperature and humidity, and wind speed and direction. As the models evolve to more accurately represent various physical and chemical processes, many other types of observations – related, for example, to atmospheric composition, ocean biochemistry and land-surface processes – become important. Forecast products are updated several times per day as new observations become available, and are disseminated to users within hours. A critical step in this process is the initiation of a new model forecast based on adjustments made as a result of new information extracted from the most recent observations. The technical term for this continuous process of blending observations with model output is “data assimilation”.

Over time, a global NWP system will generate a long time series of meteorological fields for several geophysical parameters, based on observations and consistent with the laws of physics, covering the entire globe from the Earth surface to the stratosphere. When NWP systems became operational in the 1970s it was soon realized that such a digital representation of the atmospheric circulation, containing a history of weather events around the world, would be invaluable for research and development in atmospheric science. However, creating a consistent data record spanning multiple decades requires reprocessing, or “reanalysis”, of archived and quality-controlled observations using a fixed configuration of an NWP model and data assimilation system. Such a reanalysis needs to be repeated occasionally when forecast models, input observations and data assimilation methodology have improved substantially, and new computing capabilities enable higher spatial and temporal resolution of the data.

Reanalysis provides the academic research community with access to the vast amount of information generated by the global observing system, synthesized using state-of-the-art forecast models, in a form that is convenient to use. 

The NWP Centres are major users of reanalysis data, for example as a benchmark for evaluation of medium-range forecast performance. High-quality reanalysis data are indispensable for the development of seasonal climate prediction systems, which depend on the availability of a large database of hindcasts (i.e. re-forecasts of representative historic conditions) to enable statistical correction of the systematic errors that tend to develop in long-range forecasts. Reanalysis data are used to estimate global climatologies and probability density functions for various meteorological parameters that underpin a growing suite of probabilistic forecast products that can be used for risk assessment, emergency warning systems, planning and decision-making. These include, for example, maps that indicate locations where extreme weather is likely to develop in the near-to-medium range (Figure 3). 

Figure 3 - Extreme Forecast Index (EFI) map for 5 October 2021, showing where anomalous weather is likely to occur within the next seven days. The colours mark areas with likely high winds (magenta), heavy rainfall (green), high temperatures (yellow/orange) and low temperatures (light blue/blue). The EFI relies on climatologies and anomaly probabilities derived from reanalysis data. (Source: ECMWF)

Climate Reanalysis 

The use of global reanalysis data for climate applications has grown rapidly, despite the well-known effect of biases in models and observations on the representation of climate variability and change (Bengtsson et al., 2007). A reanalysis is a multi-decadal model simulation constrained by observations; any significant change in the observational constraint potentially interferes with the estimated climate signals. Modern reanalyses are better able to correct biases due to increased availability of high-quality controlled observations, better forecast models and advances in data assimilation. As a result, and reflecting the growing role of reanalysis in climate services, it is now commonplace to refer to “climate reanalysis.” 

Together with other climatological datasets derived from observations alone, data from climate reanalysis are now routinely used in annual State of the Global Climate reports published by WMO, the American Meteorological Society, and the Copernicus Climate Change Service. The use of NWP infrastructure for climate reanalysis has several important advantages: (1) data on climate change can be updated within days of observation; (2) ECV estimates cover the entire globe, including tropical and high-latitude regions; (3) the estimates are informed by quality-controlled observations from all available sources; (4) estimates for multiple ECVs are physically consistent with one another,  (5) meteorological observations are effectively reused for climate increasing the benefits of exchanging them.

The increasing awareness of climate change and its considerable impact on lives and livelihoods has led to a growing demand for science-based services for various industrial sectors. Climate reanalysis has a key role to play in their development. The re-insurance industry relies on reanalysis data to develop statistics and trends on windstorms, coastal flooding and other weather-related events to estimate future vulnerabilities and losses. Similarly, adaptation to climate change in the transport and infrastructure sectors requires information about trends and variability in temperature and precipitation, as well as other key variables affected by climate change such as soil moisture, sea level and sea ice. The energy sector critically depends on reanalysis data to provide the parameters needed to estimate the potential value of different renewable energy sources around the world, including wind, solar and hydro. In agriculture and forestry, reanalysis data are routinely used to map the movement of climate zones affecting crop planning and water supply. 

The current status of global historical meteorological data archives

As noted above, archival climate data and free and unrestricted access to climate data is of fundamental importance. Although there have been substantive recent improvements in the archival of climate data (Noone et al., 2020, Durre et al., 2018), many obstacles still exist:  

• Data may not be freely shared.  In some cases, observations may not be internationally exchanged, or only against payment. 
• Data may be freely available, but a lack of resources may hinder the process of sharing the data.  
• Data may be exchanged, but on a restricted basis, for example only for certain purposes or with certain groups. 
• Poor data stewardship can lead to data not being discoverable and thus not used, even if nominally available. 
• Climate data can be lost: paper records degrade, electronic formats becoming unreadable, lack of backup and proper archiving can all contribute. 

Historically, data preservation has fallen on a handful of institutions, and submissions of historical data have been piecemeal in nature. Collections are more complete since the advent of data sharing over the Global Telecommunication System (GTS), which has allowed these data to be captured and archived. 

Figure 4 -  Location of land-based stations with temperature observations. Plot shows number of land-based stations operational for temperature from 1750–2020 at each timescale using a logarithmic scale. (Source: Noone et al., 2020)

Marine observations have led the way. For several decades data have been collected and collated via International Comprehensive Ocean-Atmosphere Data Set (ICOADS) (Freeman et al., 2017). Data are held in a multivariate archive and all original sources are retained.  Undoubtedly, there remain national archives which could be further integrated improving coverage. The ICOADS effort is mature and the process well documented with good levels of community buy-in. 

The World Data Centre for Meteorology is maintained by the US National Oceanic and Atmospheric Administration’s (NOAA) National Center for Environmental Information (NCEI) in Asheville. In collaboration with international and national organizations, NCEI acquires, catalogues and archives global meteorological data which is freely and openly made available to the scientific community and the public via data portals and web-based services.  

NOAA’s Integrated Global Radiosonde Archive (IGRA) consists of radiosonde and pilot balloon observations from more than 2 800 globally distributed stations, retained as multivariate records. Data collection is principally based on data exchanged across the GTS, supplemented by dedicated data rescue efforts. IGRA is owned solely by NCEI in its role as the World Data Centre for Meteorology and has far lower visibility and buy-in than ICOADS. 

Land meteorological data holdings are in a far less advanced state and are generally shared at a mixture of synoptic, daily and monthly aggregations. Data management has often been structured per variable or per timestep and been project-based rather than sustained in nature, which means that data have been disaggregated (Thorne et al, 2017). Piecing the data back together again is difficult, since different data archives use distinct data formats and metadata. Figure 4 provides an indication both of the spatial distribution of presently acquired holdings and how the availability of data subject to various levels of temporal aggregation has changed over time. Further details of current status are given in Noone et al. (2020).

The  Global Land and Marine Observations Database (GLAMOD) is part of the Copernicus Climate Change Service (C3S). GLAMOD will be hugely important for reanalysis and producing climate services.  

Filling the gaps and enriching our knowledge 

Figure 5  - The hard-copy historical meteorological observations storage facility at NCEI, Asheville, USA. (Source: S. Noone, 2017)

Climate Data Rescue 

Hundreds of millions of weather observations made from the eighteenth through the early twentieth century are still available in hard-copy or image form only and are at risk of being lost forever (Brönnimann et al., 2019). Many additional holdings of various libraries, records offices and archives that pre-date the establishment of NMHSs lie undiscovered and uncatalogued. If digitized and made discoverable and accessible, these observations would complement the temporal and spatial coverage of existing records for data sparse regions and times where climate change impact studies are crucial. This would provide long climate data records to support high-quality centennial-scale reanalysis products (Slivinski et al., 2019). Despite huge efforts in data rescue, there are still considerable volumes waiting to be processed (Figure 5). Known data rescue projects and a catalogue of data available for rescue is maintained by WMO, hosted by the Royal Netherlands Meteorological Institute (KNMI), at https://www.idare-portal.org/ and best-practice guidance are available from both WMO and C3S (https://datarescue.climate.copernicus.eu/). 

Data Management 

Access to high quality, well-managed climate data are the cornerstone of climate services. However, standards and recommended practices for sourcing, storing indefinitely, managing, assessing and cataloguing climate data are required, and the infrastructure for their free and unrestricted  exchange. Consistently assessing how well the data are managed is one way to establish or demonstrate the trustworthiness of the data. The High-quality Global Data Management Framework for Climate (HQ-GDMFC, WMO, 2019b)) is a WMO collaborative initiative that provides such an assessment at the global, regional and national levels. The scope of international collaboration within HQ-GDMFC is based on a set of principles: 

1. Promoting adherence to WMO data policies
2. Registering datasets to be shared internationally for use in climate studies, monitoring and applications
3. Facilitating easy access to metadata and documentation underpinning the datasets
4. Promoting preservation and sound, standards-based management of all data that are used, or may potentially be useful for, climate-change monitoring, including backing up in duplicate repositories for the duration of their specified retention periods
5. Assessing and improving the maturity and quality of stewardship practices underpinning the datasets, cataloguing them for easy search, discovery and access, and promoting their use in informing policy-relevant frameworks
6. Promoting acquisition of user feedback on the quality, fitness for purpose and usability of shared datasets.

As part of data centre management, (e.g. WMO technical regulations (WMO, 2019)), operators are responsible for ensuring that a business continuity plan is developed and maintained to mitigate risks associated with disruption of operations to their databases. Such a plan should incorporate provision for routine backup, and procedures for timely restoration of the database and associated infrastructure. WMO Members engage to ensure that climate data is stored indefinitely.

Among the numerous challenges to the implementation of quality climate services at both the global and national level is that much of the existing guidance on climate data management struggles to keep pace with the rapid advances in technologies, community best practices and user requirements. The WMO has developed and baselined the Stewardship Maturity Matrix for Climate Data (SMM-CD) (Peng et al., 2019), to allow data stewards (e.g. in NHMSs) to assess their data management practices in an internationally standardized framework, identifying gaps and other elements of their processes that would benefit most from improvement.

Effective data stewardship requires a combination of sustained stewardship efforts at national, regional and global levels, working synergistically. NMHS either directly manage or otherwise have stewardship over national observations and have the local knowledge to most effectively manage observations made under their auspices. However, due to the global nature of weather and climate, data need to be shared with regional and global repositories to ensure that new products and services derived from these aggregated holdings have the highest possible utility at the national level. 

Data in service to society 

In recent years, major international programmes, such as the WMO Global Framework for Climate Services (GFCS) and the European Union’s Copernicus Climate Change Service (C3S), have been established to coordinate and organize climate data and tools for use by governments, public authorities and private entities around the world. Their common goal is to create a set of operational services and shared practices that put the best available science and tools in the hands of those facing the immediate challenges of adaptation and mitigation in the communities where they live. They operate on the principle that free and unrestricted access to quality-assured data and information about the past, present and future climate is essential to enable climate-resilient and climate-smart societies.  

The implementation of C3S by ECMWF represents a turning point in improving access to observations and to the tools needed to use them effectively in climate services. C3S offers user-driven operational services, including reliable data services and user support via a dedicated Climate Data Store (CDS). The CDS catalogue includes numerous datasets derived from observations addressing the majority of GCOS ECVs, as well as high-quality climate reanalyses generated using ECMWF’s NWP infrastructure. C3S coordinates and supports a wide range of activities to ensure continuous availability and improvement of those datasets, including data rescue, data collection, data management, quality control, data re-processing and reanalysis production.

Way forward 

Observations constitute our primary source of information about how our climate is changing. They provide direct and convincing evidence on the impacts of climate change, are indispensable for the development of seasonal climate predictions and are needed to validate and improve the models used to simulate future climates under different emission scenarios. None of this is possible without shared access to high-quality observations – global, regional, local; past, present and on a sustained basis into the future.

The proposed WMO Unified Data Policy resolution, which calls for free and unrestricted exchange of historical observations, is a potential game-changer for climate services. It would lead to wider availability and greater quality of the science-based information needed for better decision-making in the face of a changing climate.

Climate observations include not only meteorological observations provided by the NMHSs but also ocean and terrestrial observations, covering cryosphere, hydrology and biosphere.  Most terrestrial observations are performed and resourced at a national level. Many observations are exchanged freely at the global level. Hydrological observations are the exception, only a few are exchanged globally(see article 6). The new WMO data policy covers the exchange of all publicly funded Earth system data. “Non-weather” observations, such as the terrestrial and oceanic observations, are for now considered to be recommended data under the policy, but may eventually become core data if and when the requirements to exchange them becomes substantiated and broadly agreed.

Authors

References