Modernizing Data Exchange for Earth System Monitoring and Prediction

National Meteorological and Hydrological Services (NMHS) have a critical role to play as humanity faces growing risks from weather and climate extremes (IPCC 2021[1]). Climate and weather information and early warning systems enable timely and effective decision-making to protect lives and livelihoods, supporting global efforts to reduce poverty and promote shared prosperity (WMO et al 2015[2]).

The work of every NMHS relies on observations and other data products, shared freely, in real time, in accordance with the principles of the World Weather Watch (WMO 1995[3]). Former WMO President John Zillman described the World Weather Watch as “the most successful fully international system yet devised for sustained global cooperation for the common good in science or in any other field” (Zillman 2018[4]).

However, many countries experience a significant gap between current network performance and the requirements of the global forecasting systems upon which almost all weather and climate services depend (Alliance for Hydromet Development, 2021[5]).

The first article in this Bulletin sets out the history of the collaboration between WMO Members in making and exchanging observations in support of weather forecasting and climate monitoring. This article examines the infrastructure and technology supporting the WMO global data exchange – from its origins more than 50 years ago to the present day. It then looks ahead to how Internet-based technologies can bridge the capacity gap and open new opportunities to make global data more reliable, more accessible and more exploitable in support of building global resilience.

Data exchange in support of Earth system monitoring and prediction

Weather forecasting applications cover a wide range of timescales: from nowcasting and very short-range forecasting, to short- and medium-range forecasting, to monthly, seasonal and longer-range predictions. As the prediction systems supporting these applications become ever more sophisticated, they rely increasingly on observations of all Earth system components to which the atmosphere is linked: the ocean, the cryosphere and the land surface. Moreover, observations are required of a growing number of geophysical variables at ever greater spatial and temporal resolutions.

In recent years, the space agencies around the world have been making key contributions in support of these activities. More and better performing instruments are being deployed on a growing number of satellites, providing information on additional geophysical variables. However, NMHSs are not always able to access the full set of observational data that they could use from these space-based systems; compromises have to be made to reduce the volume of data transmitted to them.

For surface-based observations, many issues need to be tackled in the establishment and maintenance of the observing systems themselves. But even when these issues are resolved, additional problems may arise when attempting to communicate the observations to users in a timely and effective manner. These may relate to a number of issues:

• Data policy (see here) 
• National and international telecommunications infrastructure
• Metadata – the information that accompanies the observations to allow the users to interpret them
• Specific enhancements to support evolving user requirements, such as the transition to radiosonde data at high vertical resolution
• Exploding data volumes, particularly associated with surface-based remote sensing, for example, weather radars.

Additionally, while the data exchange in the early days of WMO (see Article 1) mainly focused on the exchange of observations, there has developed over the years an increasing need to exchange other types of meteorological data and products. These additional data types are now driving some of the requirements for improved communications technology. In terms of data volumes, the main challenges arise from the requirement to exchange the output of Numerical Weather Prediction (NWP) or, more generally, Earth system models, and the increasing resolution of these models and thus the data volumes generated by them.

In the remainder of this article, we describe (i) the development of the existing WMO data exchange networks, (ii) what is happening already to address the outstanding problems of data exchange, (iii) what is planned, and (iv) how the whole WMO community will be involved in improving the exchange of the data on which all rely.

A brief history of the GTS, the WIS and their shortcomings

Figure 1 - The GTS as defined by WMO in 1969, and “almost current” even today.

In 1971 the Sixth World Meteorological Congress approved the Manual on the Global Telecommunication System (GTS), thereby starting the operational life of the system. The Manual described the GTS as "The coordinated global system of telecommunication facilities and arrangements for the rapid collection, exchange and distribution of observations and processed information within the framework of the World Weather Watch" (WMO-No. 49).

During the past fifty years, the GTS has maintained a continuous real-time exchange of essential data, providing observations to the Global Data Processing and Forecasting System Centres and disseminating processed information to NMHSs. Despite some evolution of the technologies used for data exchange, the GTS has kept its basic technical foundations unchanged. The emergence of increasingly rapid, high bandwidth global connectivity through the Internet now offers new opportunities for the future evolution of the GTS.

One example of the GTS architecture is the so-called "store and forward" mechanism: a message received by a Centre is stored and forwarded to the "next" Centre in the complex topology shown in figure 1. This mechanism, which predates the Internet, is based on the use of private networks to ensure high availability of the connections between NMHSs. Today, however, migrating to the Internet could provide a similar level of resilience at lower costs.

Another example is the use of identifiers, called “GTS headings” to route data through the complex network depicted in figure 1. These headings, based on groups of six letters, are statically assigned to bulletins, and "routing tables" are maintained in each transmission centre to direct the messages along the planned route through the network. Whilst this mechanism has worked successfully for the past fifty years, the static nature of routing tables and the relatively simple syntax of the GTS identifiers are not scalable to the current explosion in both volume and variety of data. Now, with a growing need for diversified data, the routing mechanism is one of the most severe structural limitations of the GTS. A fundamental system redesign is needed to overcome this.

A further limitation of the GTS is in the complexity of the topology, which requires a level of coordination between WMO Members that is sometimes difficult to reach, for a variety of technical and political reasons. The possibility of a drastic simplification of the data exchange topology could not have been anticipated in the earlier years of the GTS. Today, with the Web as a backbone of global data and information exchange, there is a clear way forward that may help WMO resolve many of the fundamental issues with the architecture of the GTS.

A significant move to improve the system and address these issues was initiated by the Fifteenth World Meteorological Congress in 2007, driven by the need to provide data access to entities not directly connected to the GTS. This led to the development of the WMO Information System (WIS), which was intended to complement the GTS. WIS provides a searchable catalogue and global cache to enable additional discovery, access and retrieval services through Web portals, maintained by 15 designated Global Information System Centres (GISCs), each of them operated by a WMO Member.

WIS also defined new roles for WMO Centres worldwide, recognizing the need to improve the coordination between Members and to facilitate data exchange beyond the World Weather Watch. However, WIS still uses the GTS as its underlying operational service for data exchange with only minor improvements, thereby inheriting most of its intrinsic limitations.

Finding and accessing data through WIS and GTS

Teams of technicians specialized in GTS operations ensure that data are continuously acquired and delivered to support the operational activities of the NMHSs. However, finding and accessing data from the GTS requires specialized knowledge available only within a limited community of GTS experts from mostly well-resourced NMHSs. This means that the NMHSs of less developed WMO Members are often not well-equipped to access and use this valuable stream of real-time data, and that other institutions and the general public are entirely excluded.

The implementation of WIS, commencing in 2007, meant that users worldwide could now, in principle, search and access data freely or request permission from the data owners. However, despite enabling the publication of many datasets from GTS and other sources, WIS has never totally fulfilled its original purpose of providing easy access to WMO data. WIS users have encountered various problems:

• The complex portal interface does not provide a seamless experience to users
• Searches return too many results
• Searches return varied data types and products, making it difficult for users to perform more granular searches
• Broken links make the data inaccessible
• Specialized WMO data formats, with few available processing tools, make the use of the retrieved data problematic.

The 15 GISCs provide a variety of web interfaces. However, the GISC portals are less valuable than originally intended because they present too many barriers to non-expert users. The WIS catalogue currently consists of over 100 000 records published by several hundred entities, not all following consistent description standards. The complexity of the information for each record presents difficulties in maintaining a consistent and effective catalogue with meaningful, high-quality metadata. This often makes searches ineffective: finding data without the help of GTS experts can be an impossible task using the current WIS catalogue model. Thus, ultimately the WIS catalogue is targeting the wrong audience – the original intent of exposing the GTS to non-experts through search portals has not succeeded. Additionally, the translation from GTS language and WMO-specific data structures and formats is currently missing. Without this translation, the data will not reach the intended wider audience.

The growing variety and volume of data used by NMHSs make the current WIS data discovery and access methodologies an unsuitable solution for Earth system monitoring and prediction. A clean break with the past and a significant leap forward in technologies and architecture is therefore urgently needed for the future evolution of WIS. A new approach is essential to make the data accessible to all NMHSs, especially those of less developed countries, to the external organizations fostering research and supporting the evolution of WMO programmes, and to the growing community of other potential users worldwide.

WIS 2.0

WIS 2.0 is now being designed and implemented to address the issues of the current WIS and GTS implementations discussed above:  to meet the demand for data volume, variety, and velocity. By doing so, WIS 2.0 will make authoritative weather, water and climate data more relevant than ever before for everyone.

WIS 2.0 will provide low-barrier infrastructure, data and services, resulting in easy and approachable data sharing for all of the WMO community and beyond. However, getting there will not happen by accident; WIS 2.0 is grounded by three foundational pillars:

• Simpler data exchange
• Open standards
• Cloud-based infrastructure.

Figure 2 - Conceptual view of WIS 2.0

Simpler data exchange

WIS 2.0 prioritizes public telecommunication networks, in contrast to the use of private networks for GTS links. The use of the Internet will enable the best choice for a local connection, using technology that is commonly available and well understood.

WIS 2.0 will thus both rely on and actively support the implementation of United Nations Sustainable Development Goal (SDG) 9, which includes the target to provide affordable, universal Internet access to Least Developed Countries (LDCs).

Figure 3. Open standard message protocols in WIS 2.0.

The backbone to modern and ubiquitous information sharing is the World Wide Web. Adopting Web technologies as the core of WIS 2.0 will lay the foundation for improved discovery, access and use of weather, climate and water data. The Web also provides a truly collaborative platform for a more participatory approach, where users are no longer just observers.

Data exchange using the Web also facilitates easy access mechanisms. NMHSs can publish their data as directories of flat files, as well as via Web Service APIs[6], in order to allow for dynamic discovery, access and visualization, enabling users to download exactly what they are looking for. Browsers and search engines allow Web users to discover data without the need for specialized software. The Web also allows additional platforms to access data, for example, desktop Geographic Information Systems (GIS), mobile applications, forecaster workstations, etc.

Providing data on the Web does not automatically mean that all data are freely available to all without restrictions on use. Access controls and security developed for applications such as online banking and e-Commerce may be implemented to limit access to data and services where needed. Web technologies also allow for authentication and authorization where necessary, practices which allow the provider to retain control of who can access published resources, and to request users to accept a license specifying the terms and conditions for use of the data as a condition for providing access to them.

WIS 2.0 will not push data around the network like GTS is doing today. Real-time data exchange will be implemented with “publish-subscribe” open standards using a simple group messaging system, analogous to a “WhatsApp for weather”. Data providers will be able to publish their data via Web services, and users can ask to subscribe to those data streams they are interested in. As new data become available, the subscribed users will receive them immediately, the same way that users receive a message from a WhatsApp group they belong to.

Leveraging open standards

WIS 2.0 will leverage existing industry standards, which are open and publicly available. In today’s standards development ecosystem, standards bodies work closely together to minimize overlap and build on one another’s areas of expertise. The World Wide Web Consortium provides the framework of Web standards, which are leveraged by the Open Geospatial Consortium and other key standards bodies. WMO’s use of open standards allows for a “build by exception” philosophy. It will leverage open standards that have industry adoption and wider, stable and robust implementations, thus extending the reach of WMO data sharing and lowering the barrier to access by Members.

Open standards also provide organizations with access to a wide range of off-the-shelf software (open source and proprietary). This lowers the cost of software development and maintenance, and it helps lower the barriers to implementation and use. Organizations will be able to choose from existing tools that enable them to access and use their selected data quickly and efficiently.

 

Cloud-based infrastructure

Satellites, radars and numerical models are generating more data than ever before. Storage, management and processing of this data requires expensive infrastructure. Moreover, the data volumes are becoming so large that it is increasingly impractical to download all the data for local processing by the user. A better approach is to move the processing closer to the data using cloud technology. Cloud platforms' provisioning of infrastructure and software-as-a-service enables processing adjacent to the data in environments that can be easily replicated and re-used.

While WIS 2.0 will not enforce the use of the cloud, it will encourage WIS centres to adopt cloud technologies where appropriate to meet their users’ needs. So whilst the use of cloud services will not be mandated by WMO technical regulations, WIS 2.0 will encourage a gradual adoption of cloud technologies where they provide the most effective solution.

Cloud-based infrastructure provides a turn-key solution to hosting data and services in a flexible manner. This means that a system implemented by a specific country can be packaged and deployed easily in other countries with similar needs. The use of cloud technologies will allow WIS 2.0 to deploy infrastructure and systems efficiently with minimum effort for the NMHSs by shipping ready-made services and allowing the implementation of consistent data processing and exchange techniques.

It should be made clear that hosting data and/or services on the cloud does not affect data ownership. Even in a cloud environment, organizations still retain ownership of their data, software, configuration and change management, exactly as if they were hosting their own infrastructure. As a result, data authority and provenance stay with the organization, and the cloud is simply a technical means to publish the data.

Cloud services are highly effective tools to deliver infrastructure and software. However, the need to fund these services on an ongoing basis poses a challenge for some Members and does not align well with typical business models used by the international development agencies. However, there are opportunities to obtain seed funding and technical support and training from cloud service companies. Indeed, one such opportunity is being explored through the WIS 2.0 demonstration project "Malawi Automatic Weather Stations Data Exchange". The cloud services provided without charge to WMO by Amazon will allow the development of a WIS 2.0 data exchange system that would potentially be deployable in other countries. The continuous funding needs may be covered by the Systematic Observations Financing Facility (SOFF) initiative[7]. This represents an opportunity to make the exchange of observational data constant and reliable in regions where the lack of data is a long-standing issue affecting NWP quality and the performance of early warning systems.

 

A new approach for implementing WIS 2.0

WIS 2.0 will use the lessons learned in the development and implementation of WIS, including its limited success in meeting the needs of the wider WMO community. A more collaborative implementation approach is planned, helping to lower barriers and increase system participation by WMO Members and partner organizations. As with many modern data initiatives, WIS 2.0 embraces a co-development approach, working with organizations to participate in WIS while iteratively evolving the core system. There will be a particular focus on the needs of LDCs and on ensuring that nobody is left behind.

The WIS 2.0 Principles[8] are central to its success. They comprise a set of technical and working practices intended to modernize access to promote discoverability and accessibility of data and information resources while improving the efficiency of physical data exchange.

Two critical elements of co-development include the core WIS 2.0 engagement function and the WIS 2.0 portfolio of demonstration projects. WIS engagement is undertaken with WMO Regional Associations, with various WMO Programmes and with external partners such as the Intergovernmental Oceanographic Commission (IOC) of UNESCO and the private sector. This permits advance and ongoing identification of user and contributor needs and opportunities, barriers to participation, and any factors that need to be considered by the teams developing the WIS 2.0 architecture and technical components. Meanwhile, the WIS 2.0 demonstration projects will explore, demonstrate and evolve elements of WIS 2.0 through focused initiatives that adhere to the WIS 2.0 Principles.

 

Co-developing WIS 2.0 through demonstration projects

Demonstration projects have been selected based on their alignment with the WIS 2.0 Principles, their role in evolving and validating WIS 2.0 concepts, solutions and implementation approach, their demonstration of benefits that WIS 2.0 will bring to the WMO community, and the cooperation by several WMO Members participating in the project. These projects encompass several elements, including:

• Data discovery – activities include investigations into lightweight data description (metadata), cataloguing and search, and implementation of a modernized catalogue – covering the GISC Beijing area of responsibility – which can be indexed and searched by commercial search engines
• Data exchange – activities include investigations into lightweight data exchange protocols as a modern web-based alternative to GTS data exchange, and the establishment of data exchange arrangements using industry formats including NetCDF (Network Common Data Form)
• Earth system domains – activities include data exchange between specific applications related to Earth system domains, and demonstrations of light-weight approaches to lower barriers for participating centers
• Supporting LDCs and small-island developing states (SIDS) and territories – activities include modernization of the Malawi Automatic Weather Stations data exchange to support forecasting requirements, and implementation of interconnections between GISC Casablanca and centres in its area of responsibility and leveraging the Internet for data exchange.

 

WIS 2.0 and support for LDCs

Although the network of WIS Centres is well established and fully operational, it is recognized that there are regions where data availability is still very sparse. Therefore, WIS 2.0 will place a specific focus on improving data availability by supporting LDCs in the challenge of exchanging and making use of data. A combination of lightweight standards and protocols, cloud technologies and the public Internet will enable LDCs to leverage existing capabilities where they exist, and to manage complexity by lowering technological barriers and optimizing data exchange to account for infrastructure limitations.

Again, the Malawi Automatic Weather Stations Data Exchange demonstration project provides an example of this work. This project seeks to modernize the regional data exchange to address long-standing gaps in the observational data coverage. The work entails a mix of cellular infrastructure upgrades, process optimization and IT systems upgrades to leverage the cloud for reliability and sustainability, and to enable the flow of data via WIS 2.0.

 

Supporting Members through the transition to WIS 2.0

As described above, the fundamental component of WIS is the GTS, the private dedicated network and the technology stack used for real-time global data exchange. It is recognized that, although a modernized architecture and lighter-weight standards and protocols will facilitate easier participation in WIS 2.0, the shift away from the GTS, as envisioned for the project, will require dedicated support to ensure that Members are able to migrate smoothly. Members will be supported in the transition to WIS 2.0 through a combination of training, outreach and development of communities of practice to enable them to focus on challenges in specific aspects of the transition. A change management strategy will be implemented for the project, with the migration away from the GTS managed prior to its eventual shutdown. It will be critical for Members to fully participate in this work to enable to complete the migration as efficiently as possible, and the WIS 2.0 team is available to help on request.

 

WIS 2.0 enabling the unified data policy implementation

WIS 2.0 represents the next step in data exchange infrastructure for WMO. It will provide the technological means to implement the new WMO Unified Data Policy. It will allow the data owner better control over how the data is shared and used by allowing either open or restricted access, as required. WIS 2.0 presents a critical opportunity to overcome long-standing challenges with the GTS, enabling NMHSs, the wider WMO community and many other users worldwide to access weather, climate and related Earth system data more easily than ever before. The needs are urgent, the vision is clear and compelling, and migration work is now underway. WIS 2.0 will play a vital role in bridging the capacity gap and building global resilience in the face of increasing weather and climate risks.

Footnotes