WMO Data Exchange – Background, History and Impact

While our everyday experience of weather is dominated by its local impact, weather and climate are truly global phenomena. It is often said that “Weather and Climate know no boundaries”, and an observer will quickly realize that weather systems develop and move across the planet regardless of political boundaries. The implications of this basic fact on how we monitor weather, and how we attempt to understand and predict it, are profound.

The science and practice of meteorology are built on the realization that if we can describe the current state of the atmosphere and underlying surface and know the physical laws which govern their behaviour, we can, in principle, predict future weather and climate in ways that can contribute usefully to human safety and well-being. For nearly 200 years, we have been aware that, if we can observe the present state of the atmosphere over our national territory, we can predict our local weather with some skill for a few hours or maybe a day ahead. For nearly 100 years, we have known that, to predict the future weather for longer than a few days in any country, we must have access to atmospheric data from everywhere else on the globe. As the atmosphere has no geographic boundaries, it is only in its entirety that it can be comprehensively understood and, in our modern times, simulated mathematically. Modern weather or climate prediction is therefore undertaken through international coordination and a global infrastructure – without both of which it would be impossible.

As our understanding of meteorology and Earth System science grew through the eighteenth, nineteenth and twentieth centuries, so too did our awareness of the needs for all countries to access global data and reliable systems for the collection of the observations. The collection of that data started with the invention of the thermometer and the barometer in the seventeenth century and has continuously progressed technologically – leading to today’s vital space-based solutions. The history of WMO data exchange is a remarkable story of scientific vision, technological development and service provision and, most of all, of a unique system of cooperation between institutions, scientific disciplines and national governments for the good of all.

History of Data Exchange

Figure 1 - Twelfth World Meteorological Congress, 1995 (from left to right): A.S. Zaitsev, Assistant Secretary-General, J.W. Zillman, First Vice-President, Zou Jingmemg, President, Prof. G.O.P. Obasi, Secretary-General, and M. Jarraud, Deputy Secretary-General (WMO/Bianco)

The history of international sharing of meteorological data goes back to the early nineteenth century foundations of Humboldtian science (Wulf 2015), the applications-oriented data exchange legacy of the 1853 Brussels Conference (Maury 1855) and the 1873 origins of the International Meteorological Organization (IMO), the predecessor of WMO. IMO built a highly effective international framework for enabling all countries to obtain the observations from other countries and from ships at sea for research and for the provision of weather and climate services to their national communities.

The need to strengthen and expand international data exchange for both research and practical application was central to the replacement of the non-governmental IMO by the intergovernmental WMO. Data exchange was identified as a central purpose of WMO in its 1947 Convention. The new WMO framework was reinforced over its first two decades through the special data collection systems that were put in place for the 1957 International Geophysical Year and then, more ambitiously, through the 1967 launch of the World Weather Watch (Davies, 1990) and the Global Atmospheric Research Programme (GARP).

The 1970s implementation of the World Weather Watch and GARP enabled the National Meteorological Services (NMSs) to greatly strengthen their data collection and exchange, research, modelling and forecasting. This allowed them to support a wide range of public and private meteorological services with extensive national economic and social benefit. However, with the 1980s  trend towards the privatization of government services previously provided as a public good, pressures mounted in several countries to commercialize the public meteorological services of NMSs. This led to competition with the private sector, tensions between previously cooperating NMSs and fees charged for access to data that were previously freely exchanged for research. 

The commercialization issue erupted across the international meteorological community through the late 1980s and early 1990s. Despite the best efforts of the WMO Executive Council, the 1995 World Meteorological Congress faced the prospect of a complete breakdown of international data exchange and a global meteorological data war (WMO, 2019). Delegations were divided between those who believed that, without free exchange of data, international meteorological cooperation would collapse and those who believed that data commercialization was desirable (or inevitable) and that a new international data regime must be found. After long and difficult negotiations, WMO Members reached consensus that the traditional policy and practice of “free and unrestricted international exchange of meteorological and related data and products” was too globally beneficial and too important to be put at risk. The Congress unanimously adopted Resolution 40, affirming the free exchange of “essential” data as a “fundamental principle” of WMO (Figure 1).

The implementation of Resolution 40 proved challenging for WMO and many individual countries, and it was soon realized that it did not fully cover many aspects of data exchange. This included several categories of the “additional” data needed for national weather forecasting as well as hydrological and oceanographic data and the many types of data needed for climate purposes. In due course, hydrological data exchange was addressed through Resolution 25 of the 1999 Congress, oceanographic data by the 2003 Assembly of the Intergovernmental Oceanographic Commission and climatological data through the subsequent WMO Resolution 60. But, while Resolution 40 restored and reinforced the global commitment to free and unrestricted international exchange of “meteorological and related data”, it left the WMO community with a growing awareness of the need for a more robust and unified policy framework for international exchange of all Earth System data. The origins and early history of the WMO system of data exchange and the negotiation and impact of Resolution 40 are summarized in Zillman (2019, 2021) and WMO (2019).

Emergence and expansion of global numerical weather prediction

The basic principles of Numerical Weather Prediction (NWP) were enunciated by Vilhelm Bjerknes (1904), who identified the need to apply dynamical-physical methods to the fundamental tasks of determining the initial state of the atmosphere and the evolution of the atmosphere from one state to another. His work had considerable influence on a remarkable study by Lewis Fry Richardson (1922), who set out in detail a comprehensive series of governing equations and a numerical process for their solution. Richardson’s scheme was “complicated because the atmosphere is complicated”, and well beyond practical application at the time.

The advent of electronic computers in the 1940s made it possible for the first time to solve a much simpler equation numerically (Charney et al., 1950) and in due course to evolve solutions forward in time faster than the actual weather would develop – a prerequisite for the application of NWP in operational forecasting. Initially, the rate of progress was slow. It was not until the 1970s that NWP systems were able to outperform human forecasters consistently and convincingly.

Operational global NWP began on 18 September 1974, in the United States of America (US) (Dey, 1989). It was made possible by the international exchange of data from the ground-based observing systems of the World Weather Watch, and by the availability of data from US satellites: global soundings of temperature from polar orbit and regional winds estimated from tracking clouds viewed from geostationary orbit. It drew on the prior development of global atmospheric modelling and a method of analysing observational data to produce the starting conditions needed by the forecast model. Increased computer power was another enabling factor.

The European Centre for Medium-Range Weather Forecasts (ECMWF) was established in the 1970s in recognition of the potential benefits of a common computational resource and pooled scientific expertise. ECMWF became the second centre providing operational global forecasts on 1 August 1979. The Met Office in the UK and the US Navy followed in 1982. Today, WMO has nine designated Regional Specialized Meteorological Centres(RSMCs) for Global Deterministic NWP as part of its Global Data-Processing and Forecasting System (GDPFS). The GDPFS coordinates the preparation of meteorological analysis and forecast fields and makes them available worldwide. In recent decades, the RSMCs have considerably expanded the number and quality of products made available, enabling all who provide observational data to benefit from the analyses and forecasts that their observations support.

The global forecast products available from multiple sources and for multiple starting times have been important for indicating the uncertainty of forecasts and possible extreme conditions. Extensive probabilistic information from global ensemble forecasting systems have been added to them. These complement the single “deterministic” forecast with a set of generally lower resolution forecasts that are perturbed to account for uncertainties in the forecast model’s starting conditions and physical processes. Ensemble forecasting was first introduced operationally in Europe and the US in December 1992. Eight of the nine RSMCs for Global Deterministic NWP are also designated as RSMCs for Global Ensemble NWP.

The limited-area NWP systems operated by many countries also benefit from global data exchange. Although these systems only require observational data over the domains they cover, specified values at domain boundaries are needed for the duration of the forecast. These boundary values are typically provided by global systems.

Evolution of the global observing system

Observing systems have evolved considerably over the past 75 years or so. Upper-air measurements from the radiosonde network, developed in the 1940s and 1950s, were a crucial addition to the established surface observations from land stations and ships. Observations from aircraft became available in significant numbers in the 1970s, and deployment of substantial numbers of drifting ocean buoys began in 1979. Many of these types of observation have subsequently improved in quality due to better instrumentation and are today available in much greater quantity, due largely to increased automation and the willingness and capacity of WMO Members to transmit them globally.

The first satellite images of weather patterns were obtained in the 1960s, but the key developments in satellite observations for NWP came in the 1970s. Operational measurement of temperature- and humidity-sensitive radiances began in late 1972. An enhancement of radiance measurements from polar orbit and wind estimates from geostationary orbit came later in the decade. Better orbital coverage, from an increasing number of operators, better instruments and more types of measurement have followed decade by decade since then. Today, data from around 90 satellite instruments are processed by the atmospheric component of the ECMWF forecasting system.

Moreover, both in situ and space-based measurements are now being used routinely to determine starting conditions for the oceanic models that are being coupled with atmospheric models for prediction across an increasing range of timescales. Greater sophistication of the representations in forecast models of land surfaces, including hydrological aspects, and of atmospheric composition broaden further the needs for observational data and their international exchange, but also provide opportunities for exchange of a broader set of derived data products.

Figure 2 - Variation over time of measures of the accuracy of ECMWF forecasts at ranges of three, five and seven days ahead. Upper: 731-day running means of anomaly correlations of 500hPa forecasts for the extratropical northern and southern hemispheres for the operational forecasts made from 1 January 1981 to 30 June 2021. Lower: Corresponding values for regions encompassing Europe and Australia/New Zealand, from forecasts made in hindsight twice daily from the ERA5 reanalyses from 1 January 1950 to 30 June 2021. The ERA5 results are shown for two regions where the availability of radiosonde data makes verifying analyses (also taken from ERA5) more reliable than those for the whole hemispheres in the years before satellite data became available.

Uses of observations and the improvement of forecasts

Many observations are used and re-used numerous times. Forecasting centres use them to initialize several types of forecast, to evaluate forecast quality, to calibrate products and to develop and test improvements to forecasting systems. This involves both direct use and the use of analyses or reanalyses based on them.

Figure 2 shows the improvement over time of ECMWF forecasts for three, five and seven days ahead. The upper panel refines and updates a figure first published by Simmons and Hollingsworth (2002). It shows that operational forecasts over the southern hemisphere were substantially poorer on average than those over the northern hemisphere until the early 1980s, that there then followed a period of 20 years during which improvement was greater in the southern hemisphere and that since the 2000s improvement has gone hand-in-hand for the two hemispheres.

The lower panel shows performance from 1950 for forecasts made from ERA5 reanalyses (Hersbach et al., 2020, for more on reanalysis see article 5). Results are presented for Europe and Australia/New Zealand in this case as the availability of radiosonde data makes the verifying analyses (also taken from ERA5) more reliable for these regions than they are for the hemispheres as a whole, especially in the pre-satellite period. The lower panel shows that the ERA5 forecasts over the two regions were largely similar in quality from 1979 onwards. Nevertheless, there was a fairly steady improvement in the ERA5 forecasts from 1979 onward due to the increased availability and quality of observational data. Better use of observational data and better modelling were the main reasons for the operational improvements of the 1980s and 1990s. There was a little more gain in the southern than the northern hemisphere for a spell around 2000, suggesting greater impact of the new satellite-borne instruments introduced at that time.

The very large improvement in ERA5 forecasts for Australia and New Zealand around 1979 was due to the developments of both space-based and in situ observing systems made for GARP’s 1979 Global Weather Experiment, and sustained thereafter. Improvement due to observing system changes in the 1960s and 1970s is also evident for this region. The pre-1979 forecasts for Europe were generally closer in quality to those for later years, but there was improvement in the 1950s, when the expansion of radiosonde coverage included completion of the networks of ocean weather ships, and in the 1970s, when the first operational soundings from space were followed by the observing-system enhancements made as part of GARP.

Forecasting for the tropics poses more difficulties than for the extra-tropics. The approximations used in NWP were developed for mid-latitude weather and some of them are questionable in the tropics. In addition, phenomena that occur on spatial scales smaller than those that can be resolved by the model play a much more significant role in the tropics than they do in temperate latitudes. The observational data coverage, especially for in situ upper air observations, is poor in most areas of the tropics, especially in developing countries. The lack of upper air observations is a serious problem – the relatively few radiosonde observations that are available in the tropics have disproportionally large impacts on NWP skill, indicating that the system is under-nourished with these data. The lack of surface observations severely limits the ability to verify the quality of the actual weather forecast, as distinct from skill of the NWP output.

Nevertheless, there are success stories. Chief among these is the improvement of tropical cyclone forecasts and the efficacy of the resultant actions taken to protect lives and limit material damage. The official forecasts of the US National Hurricane Center, for example, routinely draw on the products made available by five global weather forecasting centres (three of them non-US) and three regional systems. The improvement over the past thirty years in track forecasts for the Atlantic Basin (Figure 3) has been considerable. There have also been distinct, though more modest, improvements over the same period in forecasts of intensity.

Figure 3 - Annual average position error (km) in official US National Hurricane Center forecasts for the Atlantic Basin, from 1990 to 2020 (Adapted from Cangialosi, 2021).

Observations for climate analysis and the implementation of GCOS

Figure 4 - Decadal-averages of global-mean surface temperature estimates from six datasets, expressed as a change over the industrial era. Two datasets (ERA5 and JRA-55) use analysis of synoptic surface air temperature data; the other four use analysis of monthly average station data. See details here.

The observations used for NWP are also used for monitoring, understanding, modelling and predicting climate. In general, climate applications require more comprehensive observations of the Earth System and there is a wide variety of institutional arrangements for making and processing these observations. The integrated Global Climate Observing System (GCOS) was formally established in 1992 as an international, interagency, interdisciplinary framework with the goal of ensuring the availability of comprehensive information on the entire climate system (Houghton et al., 2012). GCOS has identified a set of Essential Climate Variables (ECVs; Bojinski et al., 2014) that are needed for the characterization of the climate system and its changes and whose observation is technically feasible and affordable, mainly relying on coordinated observing systems using proven technology. Thus, they can take advantage where possible of historical datasets. GCOS has also regularly assessed the status of global climate observations and the needs for implementation, reporting to WMO and its other sponsors[1], and to the Parties to the United Nations Framework Convention on Climate Change.

International exchange of data for climate applications is needed both for historical and current observations. Various factors may mean that some of the latest observational data and products are available only with a delay. Such is the case for the monthly climatological data reports (CLIMATs) from observing stations that are important for extending the record of temperature change since the nineteenth century. Nevertheless, the timely meteorological and related data for weather forecasting are also used within a few days at most in the extensions of multi-purpose reanalyses. These reanalyses provide, for example, prompt updates of the temperature data record built from monthly station data (and observations of sea temperatures) for earlier decades (Figure 4). In addition to providing a much more comprehensive set of monthly records of variability and change, the reanalyses complement daily station data in identifying and characterizing the extreme events for which there is a strong and urgent demand for public information, not least concerning the role of climate change. The exchange of data products from reanalysis (see article 5) has become more open over the years since the activity began in the 1990s, providing further benefit to those who provide the observational data on which the products are based.

Design of observing networks

Starting in 1995, WMO established the process for Rolling Review of Requirements (RRR) for observations:

• Requirements for observations are assessed for each of (currently) 14 application areas spanning the full spectrum of WMO activities.
• The capabilities of present and planned surface-based and spaced-based observing systems are also assessed.
• Requirements and capabilities are compared, and current or projected gaps in capabilities are identified.
• Based on this gap analysis, a vision for the future observing system is developed, together with a plan of actions to implement the vision.

The RRR process draws heavily on the experience of both applications experts and technology experts.

From the outset, the RRR process has sought to engage all the application areas covered by WMO programmes, but progress has been more rapid in some areas than in others. Close links to GCOS ensured that the requirements of climate monitoring have always been well represented. However, the main application area driving the RRR process from the beginning has been global NWP.  The NWP community was already well coordinated, and its requirements for observations were fairly well understood. Since the 1990s this community has been effective in articulating its requirements for a rapidly expanding range of satellite data, initially through coordination between the NWP centres of Europe and North America, and the respective space agencies. This evolved into a coordination between all the major global NWP centres and space agencies, under the umbrella of the Global Observation Data Exchange (GODEX) forum with the involvement of the WMO Secretariat.

Coordination on the exchange of surface-based observations has proven to be more difficult. This is partly due to the large number of entities involved – 193 WMO Members, as opposed to a small number of space agencies – and partly due to the absence of a strong and well-organized lobby behind these observing systems. In many parts of the world, especially in developing countries, the observational data coverage has decreased over the last 20 years, even though the requirements for these data remain very strongly supported. In large part triggered by the RRR, the WMO is now taking action through its implementation of the Global Basic Observing network (GBON), in which the surface-based observing network needed to support global NWP and climate reanalysis is designed and defined at the global level. The GBON regulations will include quantitative targets for variables measured and for minimum temporal and spatial resolution, and international exchange will be mandatory.

Much of the drive for improvements in the observing networks and data exchange has come from the global NWP community. However, this has not necessarily meant that the other application areas have been overlooked or poorly served with observations. The output from global NWP systems is directly used to drive many of the other application areas, which thus inherit some of their observational data requirements from NWP.

Historically, separate observing networks were developed to serve different communities and applications, using different standards, formats and communications mechanisms, even though many of the geophysical variables being measured were the same. In principle, it was possible to use the observations made by one community to serve another, but in practice this was often difficult, time-consuming and expensive. In order to eliminate the perceived redundancy between such networks and facilitate a joint use of the assets, WMO developed the concept for a WMO Integrated Global Observing System (WIGOS), which was launched in 2011 and declared operational in 2019. The draft data policy resolution submitted to the Extraordinary Congress in 2021 is in large part driven by the need to facilitate the further development of WIGOS and thus support a truly integrated Earth System approach to monitoring and predicting the environment (see Article 2).

Conclusions

Nearly 60 years of data exchange through the WMO World Weather Watch have shown the immense power and benefits of global collaboration in understanding, predicting and responding to the diverse phenomena of weather and climate. Over this time, weather forecasting has progressed from being a niche area of value mainly to mariners, aviators, farmers and outdoor enthusiasts to becoming accepted as a community necessity and right for nearly all sectors of the economy, used in the everyday lives of nearly all people on the planet. Many of the practices originating in meteorology have found their way into adjacent discipline areas, and many of these are working closely together with the meteorological community. WMO is updating its data policy to respond to these developments, and the driving forces behind the update and its expected impact will be further illustrated in the remainder of this issue.

Authors

References:

Maury, M.F., 1855. The Physical Geography of the Sea and its Meteorology. Harvard University Press, Cambridge, 432pp.
Wulf, A., 2015. The Invention of Nature. John Murray, London, 473pp.
WMO, 2019. Origin, Impact and Aftermath of WMO Resolution 40. WMO-No. 1244, Secretariat of the World Meteorological Organization, Geneva, 23pp. 
Bjerknes, V., 1904: Das Problem der Wettervorhersage, betrachtet vom Standpunkte der Mechanik und der Physik (The problem of weather prediction, considered from the viewpoints of mechanics and physics). Meteorol. Z. 21,1–7. (translated and edited by Volken E. and S. Brönnimann. Meteorol. Z. 18 (2009), 663–667).
Charney, J.G, R. Fjörtoft and J. von Neumann, 1950: Numerical integration of the barotropic vorticity equation. Tellus 2, 4, 237-254.
Richardson, L.F., 1922: Weather prediction by numerical process. Cambridge University Press, 236pp.
Zillman J.W., 2019. Origin, impact and aftermath of WMO Resolution 40. WMO Bulletin, 68, 2, 69-71.
Zillman J.W., 2021. Origin of international meteorological data exchange and WMO Resolution 40. Bulletin of the Australian Meteorological and Oceanographic Society, 34, 3, Special Issue October 2021.