Pioneering the Collection and Exchange of Meteorological Data

by Fred Branski*

Over the past 60 years, WMO and its Members have created and evolved observing and information systems to meet the ever-growing challenges of an increasingly complex society. Through the World Weather Watch (WWW) and a new generation of integrated observation and information systems, WMO Members have been able to provide key services to decision-makers in our society, from the individual, to public agencies, to business enterprises. Observations have grown from the thousands to the billions. Some data centres today process more than 1.7 billion observations per day. It is a testament to WMO and its Members that the systems for which they have been responsible have so significantly contributed to the understanding of our world and its environment.

The WWW legacy

In April 1963, the Fourth World Meteorological Congress approved the concept of the WWW Programme of WMO. Congress recognized the value of an end-to-end approach even though this concept of organization had not yet become mainstream in systems planning at the time. WWW was conceived with three main components, each having a three-tiered structure of national, regional and global focus: the Global Observing System, the Global Data-Processing System and the Global Telecommunication System.

Collecting observations

In the mid-1960s, some 8 000 meteorological stations on land and 4 000 ships were making standard surface observations throughout the world. About one-tenth of the land stations and a few of the ships made upper-air observations; these observations were complemented by 3 000 aircraft and cloud observations from polar-orbiting satellites. WMO Members recognized the value of the observations for enhancing both their own capabilities and the global community as a whole when shared for the common good. With WMO as the organizational umbrella, Members effectively coordinated the Global Observing System (GOS) as a starting point for understanding the weather affecting citizens the world over.

Since the establishment of WWW, the observing network has improved through an increase in both the number and performance of the observing stations. For example, approximately 3 000 additional land stations have been added, and a new network of 1 200 drifting buoys has been established. The space-based components of the observing system have also been further developed, including both geostationary and polar-orbiting meteorological satellites. In addition, many automated observing systems are now part of GOS, including radars, wind profilers and automatic weather stations. Also, significant improvements were made to ship and aircraft observing systems.

Processing data globally

Renamed the Global Data-Processing and Forecast System (GDPFS), the Global Data-Processing System was initially composed of three World Meteorological Centres and several Regional Meteorological Centres (RMCs) responsible for preparing global and regional products to be distributed to National Meteorological Centres (NMCs) through the Global Tele-communication System. From year to year, GDPFS has continuously increased the number and the quality of its output products.

The increasing specialization and sophistication in the provision of meteorological services led to the redesignation of RMCs as Regional Meteorological Specialized Centres (RSMCs) with geographical or activity specialization. These centres today span the full realm of hydrometeorological applications, from regional and global numerical modelling centres, including ensemble prediction systems, to specialized centres supporting aviation, tropical cyclone warning, tsunami warning and long-range climate prediction. GDPFS centres provide a full range of products as well, from observational analysis fields generated through assimilation systems to highly specialized regional warnings and guidance products used to support service delivery by National Hydrological and Meteorological Services (NMHSs).

		Figure 1 — WIGOS will integrate data collection and exchange for diverse sensors and systems on land, in the oceans, in the atmosphere and in space.

Sharing the data

The Global Telecommunication System (GTS) is a coordinated global system of telecommunication infrastructure (facilities and lines) and arrangements for the rapid collection, exchange and distribution of observations and processed information within the framework of WWW. GTS is organized into three levels: the Main Telecommunication Network (MTN), six Regional Meteorological Telecommunication Networks (RMTN) and numerous National Meteorological Telecommunication Networks (NMTN).

MTN is the backbone of GTS, maintaining data exchange at the global level and interconnecting the major Regional Telecommunication Hubs in the six WMO Regions. RMTNs provide for interconnectivity within each region and often include connections to key centres within other regions as well. NMTNs connect the meteorological stations or centres to the NMCs of each Member.

GTS has steadily evolved: it started out as dedicated communication lines connecting one point or centre to another over fixed circuitry. Now, it is an amalgam of technologies, ranging from continued use of fixed-circuit communications to cloud-based communication networks, allowing any centre within the cloud to connect to any other centre. Included today are communication paths over the Internet, as well as satellite-based broadcasts and information collection. Now, centres or observing sites can exchange their data in multiple ways beyond the WMO-defined message-switching methodology, which is still the primary mechanism on GTS. These new methods include file transfers, e-mail and Web-based portals.

Through the GTS data management arrangements, WMO and its Members have coordinated all the communication protocols, data formats, timeliness requirements, distribution requirements, backups and monitoring needed to support a global operational flow of information in real time, 24 hours a day, every day, to support the needs of NMHSs.

Rising demands – the WMO Information System

With development in observing systems technologies, such as radars, satellites and aircraft-based sensors, large volumes of data are being measured. Moreover, improvements in computing technology, such as high-speed and parallel computing facilities and very high-resolution numerical models, are producing vast amounts of products to support better monitoring and prediction of the state of weather, climate, water and other natural resources.

Collection and exchange of massive amounts of data have become challenging for WMO Members, who are also finding the need to access information from a much wider domain. Members are under an ever-increasing demand to provide information and services that support activities beyond the traditional domains of meteorology, hydrology, oceanography and more recently climatology. Even within the traditional domains, Members are now supporting services aimed at decision-makers within areas such as disaster risk reduction, climate adaptation, advanced transportation systems, food security and health.

GTS, although evolving and still meeting the need for operational or time-critical flow of information, was not designed to meet these new demands. It is primarily focused inward to collect and exchange information already within the NMHS community and deliver that information to other Members.

In addition, the rapid uptake of Internet technologies has led WMO Members to use public networks for a large proportion of data and information traditionally communicated over dedicated links. Concurrent with these trends, powerful discovery, access and retrieval functionalities are becoming integral components of twenty-first century data management systems.

The WMO Information System (WIS) helps WMO Members to leverage these developments, serving as a global forum for utilization of and collaboration on meteorological data, products and services collection and exchange between providers and users in support of WMO and related programmes.

WIS enables the collection and sharing of weather-, climate-and water-related data and products. WIS leverages the legacy data exchange arrangements and services developed with GTS and with the Integrated Global Data Dissemination System (IGDDS), a newer space-based system primarily proving satellite products but not limited to them. WIS also embraces new technologies to provide many Internet-like functions, such as data discovery, access and retrieval.

WIS brings a standardized way to manage metadata that is based on and interoperable not only with Internet-based information systems but also with the systems that support many of the world’s library and information repository and retrieval systems. These capabilities are not only opening the domains of information within WMO to the rest of the world but are also allowing WMO Members to more easily interact with non-traditional domains.

WIS provides three fundamental types of services for exchange of information. First, it provides routine collection and dissemination service for time-critical and operation-critical data and products. This service is based on real-time “push” mechanisms, including multicast and broadcast; it is implemented essentially through dedicated telecommunication means providing a guaranteed quality of service.

Second, WIS provides timely delivery service for data and products. This service is based on delayed-mode push mechanisms; it is implemented through a combination of dedicated telecommunication means, especially space-based systems and public data-communication networks such as the Internet.

Third, WIS is making available to WMO a Data Discovery, Access and Retrieval (DAR) service. This service is based on search, request and reply “pull” mechanisms with relevant information management functions and is largely implemented through the Internet. This data exchange service opens WMO information management to the external world, as well as provides the ability to integrate data management across all WMO programmes and requirements.

WIS utilizes international standards to ensure interoperability with other information systems, enabling weather, climate and water information to be discovered and exchanged across any other system based on those standards. Such standards and the use of readily available hardware significantly contribute to the long-term sustainability, scalability and efficiency of WIS.

Managing and exchanging climate data

The following was adapted from CCL Guide to Climatological Practices, Third Edition, online version, under WMO publication.

Data management

Climatological data are most useful if they are edited, quality controlled and stored in a national archive or climate centre and made readily accessible in easy-to-use forms. Although technological innovations are occurring at a rapid pace, many climatological records held by NMHSs are still in non-digital form. These records must be managed along with the increasing quantity of digital records.

A Climate Data Management System (CDMS) is a set of tools and procedures that allows all data relevant to climate studies to be properly stored and managed. The primary goals of database management are to maintain the integrity of the database at all times, and to ensure that the database contains all the data and metadata needed to meet the requirements for which it was established, both now and into the future. Database management systems have revolutionized climate data management by allowing efficient storage, access, conversion and update for many types of data, and by enhancing security of the data.

A major step forward in climate database management occurred with the World Climate Data and Monitoring Programme (WCDMP) Climate Computing project in 1985. This project led to the installation of climate database software on personal computers, thus providing NMHSs, in even the smallest of countries, with the capability to efficiently manage their climate records. The project also provided the foundation for demonstrable improvements in climate services, applications and research.

In the late 1990s, WCDMP initiated a CDMS project to take advantage of the latest technologies to meet the varied and growing data management needs of WMO Members. Aside from advances in database technologies, such as relational databases, query languages and links with geographical information systems, more efficient data capture was made possible with the increase in automatic weather stations, electronic field books, the Internet and other advances in technology.

Data exchange

Exchange of data is essential for climatology. For states that are members of WMO, the obligation to share data and metadata with other Members, and the conditions under which these may be passed to third parties, are covered under WMO Resolution 40 (Cg-XIII) for meteorological data, WMO Resolution 25 (Cg-XIV) for hydrological data, and the Intergovernmental Oceanographic Commission Resolution XXII-6 for oceanographic data. The Resolutions embody the concepts of “essential” and “additional” data, with a specification of a minimum set of data that should be made available in a non-discriminatory manner and at a charge of no more than the cost of reproduction and delivery without charge for the data and products themselves. Members may decide to declare as essential more than the minimum set. The use of agreed-upon international standard formats for data exchange is critical.

A main avenue for dissemination of climate data internationally is through coded messages sent on GTS. In addition, WMO Members are asked to provide data and products that are required to sustain WMO programmes at the global, regional, and national levels and to assist other Members in providing meteorological and climatological services in their countries. Members supplying such additional data and products may place conditions on their re-export. Members of WMO volunteer subsets of their stations to be parts of various networks. Nomination of stations in these networks implies an obligation to share the data internationally.

Climate and related data are also shared through International Council for Science World Data Centres (WDCs). The WDC system works to guarantee access to solar, geophysical, and related environmental data. WMO is actively involved in the provision of data to a number of these WDCs, and there are a number of associated centres operated directly through WMO. Such centres exist for ozone and ultraviolet radiation, greenhouse gases, aerosols, aerosol optical depth, radiation and precipitation chemistry.

Exchange of digital data is simple for many Members because of the range of computer communications systems available. International data exchange agreements allow for the global compilation of publications such as Climatic Normals, World Weather Records, and Monthly Climatic Data for the World. Bilateral or multilateral agreements are also important in creating and exchanging long-term datasets, such as the Global Historical Climate Network, Comprehensive Aerological Reference and Comprehensive Ocean-Atmosphere Data Sets compiled by the United States of America, and the Hadley Centre global observations datasets compiled by the United Kingdom. These datasets are generally provided to research centres.

Bringing integrated management to observations

The WMO Integrated Global Observing System (WIGOS) plans to bring together existing and new WMO observing systems into a robust, coordinated, composite and integrated observing system (Figure 1). In a cost-effective and sustained manner, it will meet the evolving observational requirements of WMO Members for their weather, climate, water and related environmental services and enhance coordination of the WMO observing systems with those of partner organizations for the benefit of society.

This future for WMO observing systems will build on existing sub-systems, both surface-and space-based, in-situ and remote, and will capitalize on existing, new and emerging observing technologies not presently incorporated or fully exploited. WIGOS will thus better enable WMO Members to meet expanding national mandates and contribute to meeting the needs of other environment-related agencies. In doing so, WMO Members will be able to better respond to natural hazards, improve environmental monitoring, and adapt to climate change and human-influenced environmental impacts.

WIGOS, together with WIS, will greatly enhance operational components of WMO Programmes, especially in developing and least developed countries, and they will be robust components of the future Global Framework for Climate Services.

Such an integrated observing system will be a comprehensive “system of systems” interfaced with WMO co-sponsored and other non-WMO observing systems, making major contributions to the Global Earth Observation System of Systems (GEOSS), and will be delivered through enhanced involvement of WMO Members, regions and technical commissions. The space-based component will rely on enhanced collaboration through partnerships such as the Coordination Group for Meteorological Satellites and the Committee on Earth Observation Satellites. Portions of the surface and space-based sub-systems will rely on WMO partner organizations: the Global Terrestrial Observing System, the Global Ocean Observing System, the Global Climate Observing System and others.

Progress in technology will continue to provide a basis for further improvements in the reliability and quality of observations, thus more fully satisfying user needs. Standardization will address best procedures and practices, including quality assurance, data and metadata formats for new and emerging technologies. Further development of integrated surface-based remote sensing systems will make it possible to provide observations of key atmospheric variables and processes relevant to weather, water and climate with high time resolution. Long-term testing at instrument “test-beds” will be used to judge instrument design, performance, reliability, capability and cost-effectiveness for a full integration into WIGOS.

With improvement in seasonal-to-interannual prediction, integration of information from oceans and land will become even more important. The demands of climate modelling require an integrated and comprehensive environmental observing system that can only be provided by WMO and its partners. Development in data assimilation techniques will allow the observations to be fully exploited in numerical models in an integrated manner. Assimilation will provide the means for data to be combined with other data in a cohesive and scientific way.

It is our challenge to meet these needs into the future. It is clear that the foundations of observations and information exchange, WIGOS and WIS, are fundamental to meeting that challenge. In recognizing the challenge and responding to it, WMO will continue to provide the leadership and stewardship to meet relevant future needs of society for the next 60 years and beyond.

Acknowledgements

The author thanks Omar Baddour, Pierre Kerherve, David Thomas and Igor Zahumensky from the WMO Secretariat for their contributions to this article.

_________________

NOAA; President of WMO Commission for Basic Systems