Communicating Forecast Uncertainty for Service Providers

by Jon Gill*

Introduction

Uncertainty is an inherent ingredient in the hydrometeorological forecasting process. Forecasters are familiar with the question of uncertainty and predictability and must deal with it every time a forecast is prepared. Sometimes, the available computer models or other guidance are consistent in their predictions and the forecaster is confident of the outcome. At other times, the models may differ greatly or the weather parameter may be intrinsically difficult to forecast. Nevertheless, a forecast must be made, even when confidence is low.

Uncertainty in the forecast can also arise from how the forecaster utilizes the available information. Even if the model predictions are highly accurate, they must still be interpreted and translated by the forecaster into actual weather. This interpretation must then be rendered into a forecast, which, in turn, is received and interpreted by the user. Uncertainty can occur at each of these stages of the “information chain”.

Communicating the uncertainty of the forecast is vital to users. It allows them to make better decisions that are attuned to the reliability of the forecast. It also helps to manage the expectations of users for accurate forecasts.

This article addresses the issue of communicating forecast uncertainty. Although it includes a discussion of the sources of uncertainty and touches on the related science (e.g. probabilistic forecasting, the use of numerical weather prediction (NWP) ensembles), this is not their focus. Rather, the emphasis is on how service providers, including National Meteorological and Hydrological Services (NMHSs), can utilize forecast uncertainty information, including the best ways to communicate this information for the benefit of users.

Strategies for communicating forecast uncertainty are being developed by many forecast service providers. As these strategies are developed, it is important to be aware of some of the possible pitfalls. For example, meteorologists, as scientists, are quite comfortable with uncertainty and the language of probabilities. This is not usually the case for the general public and so there is a significant risk of misunderstanding.

The conventional text-based forecast offers little opportunity for expressing uncertainty. There is limited space in the forecast, it is not easy for recipients to absorb every word that is there, and it can take the forecaster a long time to get the words “just right”. Not only that: the verbal language of uncertainty can often be rather subjective, so that what the forecaster intends may not match what the recipient understands. One possible solution is to devise a simple numerical scale for confidence and attach it to all forecasts. This idea is not new! In an article published in Monthly Weather Review in 1906, W.E. Cooke suggested a five-point scale for describing uncertainty:

5 We may rely upon this with almost absolute certainty;

4 We may rely upon this with tolerable certainty, but may be wrong about once in 10 times;

3 Very doubtful. More likely right than wrong, but probably wrong about four times out of ten;

2 Just possible, but not likely. If showers are indicated, for example, they will not be heavy even if they occur at all;

1 The barest possibility. Not at all likely.

A forecast might read: south-west district—fine weather throughout (5) except in the extreme south-west, where a few light coastal showers are possible (2). Warm inland (4), with a cool change expected on the west coast (3).

Another way to express uncertainty is to include in the forecast the next most likely scenario, as well as the expected one. This allows users to make back-up plans. Although many users only want a single forecast upon which to base their decisions, some users with more specialized needs can derive value from knowing what the alternatives might be. This is especially true of emergency managers who need to know alternative and worst-case scenarios so they can plan their resources with all contingencies covered.

Using probabilities is a common way of expressing uncertainty and is a widespread practice. It is important that the probabilities are based on objective scientific techniques, so that they are reliable, trustworthy and well-calibrated to the true probability distribution of the phenomena in question. The definitions of the probabilities must also be clearly defined and communicated, so that users understand what they mean.

The focus of this article is on ways to describe and communicate forecast uncertainty, highlighting the key issues that service providers will need to recognize and address.

Why communicate forecast uncertainty?

There are several reasons why communicating forecast uncertainty is useful, both for users of the forecast and also for the providers. Each of these reasons is described in the following sections.

Knowledge of forecast uncertainty assists decision-making

The central reason to communicate forecast uncertainty is to assist people to make more effective decisions. This is especially so when the user of the forecast has options available to them and wants to weigh up contingencies. Such situations are very common and range in scope from simple day-to-day decisions, such as about what clothes to wear, to major emergency responses, such as evacuation planning. The following examples describe how uncertainty information can improve the quality and effectiveness of a decision:

• A farmer wishes to fertilize a crop
  For this to be successful, a small amount of rain is desirable to help the soil absorb the fertilizer. The farmer has established a rule which says that, if the probability of rainfall is less than 80 per cent, then the risk of wasting the fertilizer is too high; he waits until the chances improve. The farmer needs a high degree of confidence before deciding to apply fertilizer;
• A government food agency is assessing food security for the coming year.
  The seasonal climate forecast suggests that there is a slightly greater-than-normal chance of below-average rains over the growing season. Accordingly, the food agency initiates a food stockpiling programme. The consequences of inadequate rain are so great that the food agency responds, even though the uncertainty of the prediction is relatively high.
• An emergency services agency is deciding whether to evacuate a community ahead of an approaching tropical cyclone.
  The forecast states that there is a 10 per cent chance of destructive winds being experienced. Even though this is numerically low, it is high enough—relative to the potential consequences—for the agency to commence evacuations.

In each of these three cases, users have tuned their responses to differing levels of forecast uncertainty, according to their own particular needs. This is why information on forecast uncertainty is such a useful part of the service: it allows people to react to the forecast in a way that is appropriate to their situation. Without this information, for example, if the forecast is simply “rain” or “no rain”, the user is unable to tune responses reliably.

Communicating uncertainty helps manage user expectations

Meteorologists are routinely faced with uncertainty when making a forecast. They can find this to be stressful if users of the forecast have an expectation that the forecast is always right. Forecasters also know that some situations are more predictable than others: if they are able to communicate this to users, then a more effective relationship can be established, one in which they have a realistic understanding of the accuracy and reliability of the service.

Communicating uncertainty retains user confidence

Retaining the confidence of users is critical. Users who understand that forecasts can have a degree of uncertainty and are able to attune their decision-making to uncertainty information provided by the service provider are much more likely to retain confidence in the service. Surveys show that uncertainty information does not undermine people’s confidence in the product: on the contrary, it reassures people that they are being dealt with honestly and gives them confidence that the service is being provided objectively and scientifically.

Forecast uncertainty reflects the state of the science

It is important that meteorological services are based on good science. Uncertainty is inherent in the predictions from numerical prediction models and it is appropriate that this uncertainty is factored into the forecast and warning services that are provided. Little credit is given to the profession and the credibility of the service provider is undermined if the accuracy of the service is overstated.

Sources of forecast uncertainty

In order to communicate uncertainty effectively, it is important to understand where it comes from. Some uncertainty accumulates within the forecast process chain as result of the inherently chaotic behaviour of the atmosphere, limitations in our ability to measure and model the state of the atmosphere and in our efforts to interpret the observational and model data.

Further uncertainty arises when forecasters endeavour to turn their scientific understanding of the situation into plain language. Terminology and phraseology are often unable to perfectly encapsulate the expected forecast scenario. The format and length of the forecast may also be restrictive. As a result, uncertainty may arise because the forecaster is unable to describe the full story of what will happen.

Finally, uncertainty can occur when the forecast is received and interpreted by the user who does not always have the same understanding of the terminology or the intent of the forecast.

Strategies to deal with these uncertainties, in terms of communication, will vary. For example, in the case of scientific uncertainty, the use of probabilities can be an effective way to communicate uncertainty levels; in the case of uncertainty due to forecast interpretation, the use of clear language and well-defined terminology would be an important element of effective communication.

How to communicate forecast uncertainty

Human perceptions of uncertainty information

The prime motivation for communicating forecast uncertainty information is to assist better decision-making on the part of those receiving the information. For these recipients to respond, however, they must first interpret and understand the information.

How people perceive and respond to language and information of this kind has been investigated by behavioural scientists. Much can be learned from these studies.

For example, it has been shown that the way people interpret and describe uncertainty information can be influenced by the significance or magnitude of the event (Patt and Schrag, 2003). Such studies suggest, for example, that if light rain and heavy rain are both objectively forecast to have a 10 per cent chance, people describe the heavy rain event subjectively as being more likely.

People often expect this exaggeration behaviour in others and so will “decode” what they are told. Thus, when receiving a forecast that describes a high-impact event as a medium likelihood, users will often assign a lower threat level due to a belief that the provider of the forecast was exaggerating.

It is important to bear in mind this tendency by users to “exaggerate” and “decode” the information they receive. An effective strategy is to use objective numerical measures of uncertainty (e.g. probabilities), together with plain language that is clearly defined. An example of this approach is the uncertainty scale used by the Intergovernmental Panel on Climate Change (IPCC), which clearly defines the language and the corresponding probability thresholds (see table below).

IPCC likelihood scale

Terminology	Likelihood of the occurrence/outcome
Virtually certain	> 99% probability
Very likely	> 90% probability
Likely	> 66% probability
About as likely as not	33% - 66% probability
Unlikely	< 33% probability
Very unlikely	< 10% probability
Exceptionally unlikely	< 1% probability

User sophistication

It is important to bear in mind that different users will have different requirements for uncertainty information, as well as different levels of understanding. For some, particularly those involved in emergency response, detailed quantitative estimates of uncertainty are required. Specific response plans will be in place that describe certain actions to be taken according to defined thresholds. For example, a community evacuation plan may be activated if the probability of cyclone-force winds being experienced increases beyond 20 per cent.

Sophisticated users of uncertainty information are aware of the underpinning reasons for uncertainty and, when providing this information, service providers can use technical language and speak in some detail. The use of relatively complex graphics is also possible.

For less sophisticated users, service providers need to be quite careful about the use of complex information. Such users are less likely to understand the sources of uncertainty and will prefer simple messages and graphics.

Over time and with sufficient experience and user education, it is possible to improve the level of user understanding and sophistication. Gigerenzer et al. (2005) showed that, in New York, where the public has lengthy experience of probability rainfall forecasts, a majority of users correctly understood a forecast for 30 per cent probability of rain to mean that there is a 3-in-10 chance of rain somewhere in the city. On the other hand, in four European cities, where probability forecasting is not used, the majority of users incorrectly interpreted the forecast to mean rain would fall 30 per cent of the time or over 30 per cent of the area.

Use of colour

Colour is a powerful tool for conveying information and meaning. Like any tool, it needs to be used carefully. It is a common practice to use colour in the graphical presentation of probability (or other uncertainty) information. Great care should be taken that the colours that are chosen send the right message.

Figure 1 is an example of a probabilistic seasonal rainfall forecast issued by the Australian Bureau of Meteorology. Note that probability values below 50 per cent are denoted by warm colours.

Figure 1 — Seasonal rainfall forecast (Australian Bureau of Meteorology)		Figure 2 — Seasonal rainfall forecast (Australian Bureau of Meteorology)

By using colour this way, users would often interpret the message inappropriately. Numerically, 49 per cent is not very different from 51 per cent, yet users would interpret the forecast by its colours, concluding that the yellow areas will be dry and the light green areas will be wet.

Recognizing this problem, a new colour palette was devised that has been more effective in communicating the correct message. In the example above (Figure 2), all values between 40 per cent and 60 per cent are depicted in white or grey. The same level of information is still provided, but the “emotive” colours have been shifted so that they now apply only to the high/low probability values.

Examples of uncertainty information

This section provides some examples of effective methods for conveying uncertainty information. The examples make use of the principles and ideas expressed above. Service providers are encouraged to consider these examples when developing or enhancing the delivery of uncertainty information to users.

Words

The language of uncertainty can be either complex or simple. When presenting a weather briefing or preparing a forecast for the general public, forecasters may make use of phrases such as “chance of”, “one or two” or “possible”. Sometimes, non-specific descriptors may be applied, such as “later”, “developing” or “in the area”. These descriptors are deliberately vague because the forecaster is uncertain about the precise time or location of the phenomenon being forecast.

Often, the uncertainty associated with a forecast is due to the presence of an unpredictable weather pattern. A narrative description of the situation, including possible alternative scenarios, can be an effective way of conveying uncertainty to more sophisticated users. Radio and television are ideal ways to communicate this information.

Although language is essential for communicating uncertainty, its verbal form can introduce confusion in the mind of the user. What, for example, is the difference between “chance of” and “possible”? Does “chance of” mean the same for one forecaster as another? While it is useful to use such words and phrases so that users do not have an expectation of certainty, it is important to try and apply some consistency. Using clear definitions and procedures will help in this respect: for example, a rule could be instigated which says that a forecast of “possible showers” would only be used when the probability is above a defined threshold of 30 per cent.

Graphs

Simple graphs can be a useful way to present uncertainty information in quantitative terms. Figure 3 is an example of how a seasonal rainfall probability forecast could be presented as a pie chart.

Figure 3 — Example of a rainfall probability pie chart
Figure 4 — Meteogram of forecast temperature from an ensemble prediction scheme (ECMWF)
Figure 5 — Icons showing forecast probability
Figure 6 — Greater Horn of Africa Consensus Climate Outlook (IGAD Climate Prediction and Applications Centre)

One of the attractive features of this format is that it shows all possibilities at once. Users are therefore made aware, not only of the most likely outcome, but also of alternatives.

Another effective way of showing uncertainty, particularly uncertainty that increases with lead time, is the use of time series that include “error bars”. Figure 4 is an example of a time series forecast of temperature that shows the uncertainty at each time step.

Icons

It can be difficult to utilize a pictorial icon for communicating uncertainty. Where icons are used for this purpose, it is common practice simply to superimpose the uncertainty information in numerical terms (e.g. as a probability) on the icon, for example (Figure 5).

Charts and maps

Uncertainty information lends itself well to spatial depiction. A chart or map presentation is often an effective way to present both the forecast and the uncertainty associated with it.

The Greater Horn of Africa Consensus Climate Outlook is a good example (Figure 6). Zones of equal probability range are colour-coded (with grey for the neutral forecasts) and show at a glance the spatial distribution of rainfall likelihood.

For each region on the map, a seasonal forecast is also provided in the form of a box containing three numbers. These numbers (from top to bottom) are the percentage probability of above-, near-and below-normal rainfall. The beauty of showing all three numbers together is that all scenarios are described. In other words, it is made clear to users that, although one particular outcome might be the most favoured, the alternatives are also possible.

This format is frequently used for short-and medium-term probability forecasts as well. Such forecasts are most commonly produced by ensemble prediction schemes and can be presented in a number of complementary ways. For example, rainfall probability charts can be presented according to defined rainfall thresholds (e.g. the probability of rainfall in excess of 5 mm).

Another example of effective graphical presentation of uncertainty is the depiction of the forecast tropical cyclone track as a cone, as issued by the Cuban National Forecast Centre and the US National Hurricane Center. This ensures that members of the general public do not put too much emphasis on a single path and therefore assume they are safe if the path is not shown passing directly over them. Also, this depiction reinforces the fact that, due to its size, a hurricane can affect a very large area and is not confined to a point or narrow swath. The explanatory note at the top of the graphic is very important: “Assuming AVERAGE FORECAST ERROR, the EYE should track in the white cone in the next 72 hours”.

Scales of uncertainty

Worded categories

When describing uncertainty, it is often useful to use predefined categories that have specific meaning. This assists users to understand the precise level of uncertainty that the forecaster has in mind. Such an approach is demonstrated by the IPCC likelihood scale above.

Numerical categories

Uncertainty ratings can also be assigned using a numerical scale. This does not necessarily add more information than using worded categories, but it is simple and makes the forecast easier to read. As long as users know how the numbers are defined, then it can be a quick and efficient way to convey uncertainty information.

This approach has been adopted by the Swiss Federal Office of Meteorology and Climatology, which includes, as part of its forecasts, a “reliability” measure on a scale from 1 to 10 (Figure 7).

	Figure 7— Four-day forecast, including measure of reliability (out of 10) (Télévision Suisse and Swiss Federal Office of Meteorology and Climatology)

Probabilities

Perhaps the most common way to express uncertainty information is to use probabilities. This is a concept with which users are familiar. Like any quantitative measure, probabilities should be defined carefully and their meaning clearly explained.

When defining a forecast probability, the first decision is to choose what quantity the probability will refer to. It may be the occurrence of some phenomenon at a particular location and time, e.g. the probability of a thunderstorm. It may be a category, e.g. the probability of 10-50 mm of rain. A common choice is the anomaly category, e.g. the probability of above-average rainfall. The choice will be dictated by the phenomenon under consideration and the service requirement.

One of the particular challenges for users of probability information is having a reference point for the information. This is particularly important to assist with interpretation and response. It can be a good idea to accompany the probabilistic prediction by comparison with the normal. For example, a prediction such as “60 per cent chance of a storm this afternoon” is enhanced if a message such as the following is attached: “This is about twice the normal chance for this time of year.”

Although probabilities are a commonly accepted means to convey uncertainty information, they do come with particular communication difficulties. For a start, many users simply wish to know whether the forecast event will happen or not. These users are not interested in probabilistic predictions and will often view such predictions as an attempt to avoid responsibility and to “hedge bets”. This is where effective user education is required, so that there is an appropriate understanding of why meteorology is not an exact science.

The consequence of this is that, in the absence of a categorical yes/no forecast, a user may turn to the probabilistic forecast and translate it into a categorical one. For example, a seasonal prediction for an increased chance of above-average summertime temperatures may be interpreted as a statement that it will be a hot summer. There are countless examples where the media have oversimplified probabilistic outlooks in this way, in order to generate a catchy headline.

A second challenge is to understand what the probability of occurrence actually refers to. Is it at a point? Over a spatial area? Over time? Every effort needs to be made to ensure that the terminology is clearly defined and understood, not just by the users, but also by the forecasters who issue the forecast.

A third challenge is the so-called 50-50 problem. This arises when there is no strong influence on the atmosphere favouring one particular scenario over its alternative. For example, in countries affected by El Niño, there is no strong influence away from the average during neutral phases and seasonal forecasts may say something like: “50-50 chance of neutral rainfall conditions”. Although this statement has meaning from a strict climatological perspective, it leaves users with a strong sense that the service provider is “sitting on the fence”. At times like this, it may be better to present the forecast in another way, perhaps by presenting probabilities of rainfall (in this example), according to defined threshold amounts.

Different media, different methods

The choice of method and format for communicating uncertainty information will depend greatly on the media being utilized. What works well in one channel may not be effective in another.

For face-to-face weather briefings, or radio interviews, or wherever the forecast can be provided verbally, the use of plain language and narrative can be effective. In these settings, the forecaster has time to explain the situation, can discuss alternative scenarios, explain why and how the NWP models are different and give an overall and comprehensive view of the situation. The use of non-verbal communication skills, such as speech intonation or body language, can also be effective ways to give the listener a sense of the forecast confidence.

Where the forecast is presented in a more prescriptive way, such as in writing, then the forecaster should ensure that their description of uncertainty is confined to predefined or well-understood terms. If phrases such as “a chance of” are used, there should be some underlying definition that specifies to what this chance is numerically equivalent. Numerical measures of uncertainty may also be used.

Graphical depictions of forecast uncertainty are a useful presentation style and are especially suitable for Web-based display. They can be accompanied by explanatory information to help users interpret what can be rather complex information. For television, the options are more restricted because of the limited broadcast time available. A simple “confidence index” may be the best approach here.

Conclusion

Uncertainty is an inherent ingredient of forecasting; communicating it effectively is of great benefit. It helps users make better decisions and it helps service providers manage the expectations of users for accurate forecasts.

This article has addressed the issue of communicating forecast uncertainty. The emphasis has been on how service providers can incorporate uncertainty information in their meteorological forecast services, including the best ways to communicate this information for the benefit of users. Examples have been given of effective presentation methods and some of the pitfalls have been highlighted. Service providers are encouraged to use this information as a guide to how best to communicate forecast uncertainty and make it a routine and effective part of their service.

References

Cooke, W.E., 1906: Forecasts and verifications in Western Australia. Monthly Weather Review, 34, 23-24.

Patt, A. and D. Schrag, 2003: Using specific language to describe risk and probability. Climatic Change, 61, 17-30.

Gigerenzer, G., R. Hertwig, E. van den Broek, B. Fasolo and K.V. Katsikopoulos, 2005: A 30% Chance of Rain Tomorrow: How Does the Public Understand Probabilistic Weather Forecasts? Risk Analysis 25 (3), 623-629.

---

* Bureau of Meteorology, Australia