Tuesday, January 27, 2015

Let the recriminations commence!

It's becoming apparent that the forecasts for a 'historic' snowstorm for New York city have proven to be incorrect.  At least one meteorologist has already apologized for not getting this correct.  So the recriminations will commence in the media, I'm sure.  Blogs will be written, Bill Belichick's nasty mischaracterization of weather forecasts will be resurrected, politicians will voice their displeasure, letters to the editor will be written, and so on.  Mea culpas and excuses will be offered.

All of this could have been avoided, of course, and I don't mean by the obvious decision to issue a different forecast.  Let me explain briefly from where forecasts come these days:  to an ever-increasing extent, forecasts are more or less coming from numerical weather prediction [NWP] computer-based models.  Human forecasters who have to make forecast decisions are presented with a number of different model forecasts, some produced by different computer models, and some produced by starting the same model with slightly different input starting conditions.  To some extent, the variability within the ensemble is a measure of the uncertainty of the forecast, but even the uncertainty is, well, uncertain!  This collection of different model forecasts is referred to as an 'ensemble' of forecasts, and at times, there can be considerable variation among the different forecast model results, as was the case with this event.  It was quite likely to snow heavily somewhere at some time - the problem was to pinpoint when and where.  A 'historic' snowfall in New York city would have devastating consequences! 

In reality, what is inevitably true is that all the model forecasts will be wrong, to a greater or lesser degree.  No human- or computer-based weather forecast has ever been perfectly correct, and none ever will be.  Uncertainty is inevitable, so the best any forecaster can do is to say "[some weather event, like a blizzard or a tornado] is likely to happen, and my confidence in that occurrence is [some way to express the uncertainty of that forecast, such as a probability]."  Numerous studies have shown that forecasters are actually pretty good at estimating their uncertainty; this is seen by their forecast reliability.  Reliability of a probabilistic forecast means that, given a particular forecast probability, as that forecast probability increases, the frequency of occurrence of the forecast event also increases.  Forecasts are perfectly reliable when the probability forecast is X percent and the occurrence frequency is also X percent.  In general, it's not a good idea to present forecast event probabilities as either zero or 100 percent, although it's possible in certain situations - the probability of having a blizzard in the ensuing 24 hours when it's presently 100 deg F in July is pretty close to zero.

If the forecast probability is neither zero nor 100 percent, then a particular forecast is neither wholly right or wholly wrong.  If the event occurs although the forecast probability was low (say, 1 percent), then this is the one case out of 100 you would expect to find.  If the event fails to occur despite a forecast probability of 99 percent, again, it's the expected one case out of 100.  In the case of the snowstorm in the northeast, the problem was to know just where the heavy snow would be.  Clearly, when the event occurs, you want the forecast probabilities to be high and when it fails to occur, you want the forecast probability to be low.  Of course, there at times when it doesn't turn out that way, as I've described.

Getting back to the 'dilemma' of choosing from an ensemble of forecasts, the most likely event is the average of all the ensemble members.  But even if that average actually turns out to be the best forecast, it will not be completely accurate.  And there are times when the forecast that would have been the best is one of the ensemble members - the problem is to know with confidence which would be the correct choice, and the challenge is that science simply can't predict that with accuracy.  And in some situations, none of the ensemble members is very close to the real evolution of the weather, unfortunately - i.e., none of the model solutions were accurate.  There is no scientific way to choose the "model of the day" and efforts to do so are simply a waste time!

Everyone wants the forecasts to be absolutely correct, all the time - but like the Rolling Stones say, "You can't always get what you want!"  We need to re-negotiate our contract with forecast users who expect us to provide a level of certainty we simply are [and always will be] incapable of providing.  Users deserve to know our uncertainty in the forecast products we send out.  Not doing so is scientifically dishonest - forecasters know their uncertainty, so not sharing it is to withhold important information!

13 comments:

Rob Dale said...

Any thoughts on "how" to tell the users that uncertainty? Snow ranges were used but those didn't work out well.

Chuck Doswell said...

Rob,

Yes, I have thoughts about that, but not specific ideas on how to communicate uncertaity. I'm a meteorologist, not a psychologist or communications expert. Ideas are cheap. What we need is a process involving social scientists of various sorts to help us find out the most effective ways to communicate uncertainty. If we change the format/content of our forecast products, we should conduct a massive campaign to make users aware of the changes BEFORE we implement them and help them make the transition. This must include a way to get across the very idea that our forecasts are inherently uncertain, since we historically have kowtowed to what users want, not what they need and what we can provide.

Jim Duke said...

The NYCity snowstorm, like many other weather occurrences, has a dimension beyond occur/not occur - impact. Some weather phenomena (tornadoes, violent winds, ice storms, etc.) threaten a level of human and economic disruption that transcends simple occur / non occur. Hence, the forecaster in the present instance can find shelter in the notion that the threshold for tipping into severe warning mode with dramatic and perilous calls to action is weighted lower because of the potential for disaster.

===== Roger Edwards ===== said...

The key is "what we can provide". Without capability to fulfill them, the user's needs and wants each are irrelevant. Similarly to something I stated in an e-mail to which Chuck was party:

We as an enterprise need to grow a pair and tell the "public" and everybody else who demands impossibly deterministic forecasts:

"No. You're not getting this from us because it's not possible. Anybody who tries to sell you a forecast like this is trying to rip you off. The science isn't there yet to give you this rigid, specific answer. Nobody is going to be that good in your lifetime or mine. Here's the best we actually can do...(forecast + expression of uncertainty)"

Alas, very few manager/bureaucrats (in government, and private-sector managers too) have the stones to be that honest with the constituency. We all pay for that in lost credibility and undeserved bad reputation.

Jason Goehring said...

I'm a meteorologist and a farmer. By adding a "percent chance of uncertainty" to the forecasts, they loose their usefulness.
For example, if somebody asks "is 3 days from now good for a picnic?" If I say "there is a 40% chance of rain, partly cloudy to overcast, and 5 to 20 mph wind"- does that help that person decide to plan a picnic? If I say "it will be a mostly sunny, light wind and will not rain until 9 pm"- then that helps that person decide.
Yes, I have a greater chance of being wrong, but that person has useful information. They understand that weather forecasts are uncertain.
If the forecast is 20% chance of rain. To a meteorologist, that means most likely it will not rain but there is uncertainty. To an average person, it means its going to rain.
Forecasts need to balance that uncertainty percent with usefulness. In my opinion, public forecasts are becoming more percentage and less useful.
Either way, we are going to be wrong with our forecasts in the minds of the public, mostly because they only remember parts not the whole thing. Just how much do you want your audience know what to be prepared for.

John Hart said...

Chuck,

Do you know of any research that has shown that a forecast with little spread among the ensemble members is likely to be more accurate than a forecast with great spread? I know it seems to be a logical conclusion, but only if the ensemble members span the space effectively. That is sometimes not the case - especially as the forecast hour extends beyond 24-36 hours (which is the period that decision-makers want to start hearing about upcoming threats with confidence). There are times that we simply do not know the uncertainty inherent to our forecasts.

Chuck Doswell said...

Jim Duke,

What you seem to be saying is that it's better to err on the side of caution when it comes to devastating events. Whereas that may be true for decision-makers using a weather forecast as input, I would like the forecasts to not be biased one way or the other. Any bias undermines the credibility of the product.

Chuck Doswell said...

Jason Goehring,

I agree that we have failed to explain the meaning of probabilistic forecasts in terms the non-meteorologist can understand. Unfortunately, your perspective concerning probabilities is flawed. Many studies have shown that using words such as "likely" have different meanings to different people. Probability as we are using it is a measure of our confidence in the forecast - I'm sure you realize that we cannot make forecasts with absolute 100 percent confidence very often. Do you want to know quantitatively how much confidence we have in the forecast, or do you want to guess that confidence level on your own? In order to use forecast probability as part of your decision-making as a farmer, you have to account for many factors in a decision that we know nothing about. Do you really want us to make your decision for you by pretending to be able to do something that you and I both know we can't deliver?

Chuck Doswell said...

John,

What I have seen regarding the ensembles is this: the dispersion in the single-model ensembles is not sufficient to support the idea that the spread is directly interpretable as uncertainty. Multi-model ensembles are better at getting the necessary disperson, but even they fail sometimes.

No doubt the ensembles remain limited in their capability to describe the uncertainties, but they are clearly an improvement over relying on a single run of a single, deterministic model. Yes, the uncertainty is uncertain, especially at longer ranges. Thus, your confidence should be correspondingly lowered at those ranges.

Jason Goehring said...

Chuck,
I agree that a lot of people want the forecast to make the decision for them. I also am not a fan of the words "likely, probably...". What bothers me is when a 10% or even 20% chance is thrown into the forecast as a "cover my butt" qualifier. It reduces the credibility of the forecast for real threats.
In my forecasting class in college, we squared the error, so I figured out it was better to be a little off all the time than far off some of the time. I had the best score, but looking back now, my forecasts were all junk. Didn't provide useful information.
I think forecasts have become more of that "little off all the time" mentality.
I think we need to convey the importance of the major events better and not stress as much about the little ones.

Chuck Doswell said...

Jason Goehring,

You seem to be saying all probability forecasts are a CYA exercise. That's so wildly incorrect, I hardly know where to begin to explain. Just how do probability forecasts reduce credibility?

Weather forecasters are the butt of many jokes precisely because the public realizes we can be wrong and are so more often than we like. It seems to me that an honest admission that we can't be exactly right all the time would increase our credibility, not decrease it!

I don't want forecasts that make decisions for our users, who need to make those decisions for themselves using information in addition to the weather forecast that we know nothing about. All we do by pretending we can make perfect deterministic, binary (yes/no) forecasts is set ourselves up for humiliation.

And the idea of treating routine forecasts with less care than big events is muddled thinking. We need to do our best job forecasting every day, not just days with potentially big events. Your comment suggests that our forecasts are only important on days with disaster looming, but that's just not true.

Jason Goehring said...

I think you are misunderstanding my comments. I'm not saying all probability forecasts decrease credibility. Putting up a 10% CYA probability often, when the forecaster is pretty much sure nothing will happen, reduces credibility. By saying 10% chance of rain for the next 3 days, a lot of the public just remembers "rain"; then it doesn't rain (what the forecaster really meant). So when there is a 80% chance of rain, the public says "they said it was going to rain those 3 days and it didn't" so they dismiss that chances. The biggest problem is in the public's understanding of probabilistic forecasts. Maybe instead of 10% chance of rain, 90% chance of sunny skies.
I'm also not saying that we need to treat the routine days with less than our best effort.

The only other thing I'm going to add is this; some other industries that issue forecasts don't add probabilities. (Example) a few weeks ago I was going to sell some of my stock of soybeans, but then I read and heard 2 different market advisors (forecasters) say that they think the price will go up. That was their opinion, I knew they were not certain and there was risk. So I waited. Now I still have my soybeans, but its worth $3K less than it was 3 weeks ago. These market forecasts are just like weather forecasts, yet like you said weather forecasters are the butt of jokes and the market advisors are not humiliated when wrong. I know next time they will be right and I will make an extra $6K.

Chuck Doswell said...

Jason Goehring,

If I'm misunderstanding your comments, it's because their verbal expression isn't conveying your intended meaning clearly. To this point, you've not managed to advance a coherent, logical argument that justifies your stubborn insistence that providing uncertainty information in weather forecasts is not the right thing to do. Rather, you simply keep repeating your flawed understanding of probabilistic forecasting

Therefore, I conclude that this exchange is of no further value here, as we're simply talking past each other.