Wednesday, April 27, 2016

A "busted" tornado forecast, in retrospect


26 April 2016 (coincidentally, the 25th anniversary of a major tornado outbreak in the Plains) is a classic illustration of the challenges associated with tornado forecasting.   The connection between the synoptic-scale weather systems and the occurrence of a major tornado outbreak ("outbreak" means different things to different people – there's no formal definition) is complicated and depends heavily on details at smaller scales.   One can get the synoptic-scale forecast mostly right but the development of tornadic supercells can be quite sensitive to the detailed structure and evolution at scales ranging from the size of a single storm to features on scales thousands of km across.  In meteorology, getting all of those details exactly right in the forecast is something that more or less never happens.  We can forecast tornado outbreaks in advance with varying levels of confidence, but they're never a sure thing.  Sometimes the details conspire to ruin the forecast.  What looks portentious, even a few hours in advance, can unravel quickly, such that the event doesn't unfold as forecast.

This case reflects certain facts about how severe storm forecasts work at the Storm Prediction Center.  The "culture" of the office contributed to the way the forecasts evolved.  If the situation looks like a possible outbreak, there‘s pressure from a variety of sources to give advance notice of upcoming tornado outbreak potential.  Once a forecast is issued, subsequent forecasts tend to maintain a relatively high level, even when new information (or a new forecaster) might suggest a downgrade of the forecast.  There's a reason for that:  users are uncomfortable with vacillation of the threat level, and if the threat is downgraded, and then even newer information means a return to enhanced threat, the indecision can come across as incompetence.  In other words, it can be unwise to back off the threat level.  Moreover, there's an asymmetric penalty for missed forecasts:  a false alarm for an event that never occurs can't result in human casualties and destruction, whereas an unforecasted event that kills people can be cause for investigations and possible disciplinary action.  This makes overforecasting almost inevitable.

In this case, there were some indications from the forecast models that the probability of a major tornado outbreak was decreasing as the fateful day approached, but the outlooks continued to raise concerns that a tornado outbreak could occur.  I don't necessarily see that as an error; it's realistic given the current state of our science.  An interesting facet to the case is that in the morning outlook on the day of the event, the forecast tornado probability was still only 10%.  The outlook was not upgraded to "High Risk".  I believe this is a plausibly accurate reflection of forecaster uncertainty.  However, the media were continuing the drumbeat of concern for a major event - the issue of the media is not going to be dealt with here.  Technically, a severe weather outlook is not focused only on tornadoes, and the nontornadic aspects of the forecast worked out pretty well.  Therefore, my comments here are restricted only to the forecast of a significant tornado outbreak with multiple, long-track, strong to violent tornadoes (EF2-EF5)

In my view, and this is purely a personal opinion, the biggest "mistake" from the SPC was issuing a PDS ("Particularly dangerous situation") watch in the early afternoon.  This was not warranted by the information of which I was aware (I was out storm chasing).  Whatever explanation might be offered in justification of this decision is in direct contradiction to the observed events.  I'm sure if offered a "do-over", the choice would be not to make it a PDS watch.

Make no bones about it.  Tornado forecasting isn't an easy job and perfection is out of the question.  I mean no disrespect to any forecaster involved in this event but we have to accept that the outcome is generating some backlash that's quite understandable.  Uncertainty is inevitable and probability is the language of uncertainty; by whatever verbiage we use to express it, we meteorologists need to communicate our uncertainty to our users such they accept the real capabilities of meteorological science as applied to the task of forecasting tornadoes.  By all means, we need to find out how to communicate with our users so that they understand our message, and know how to respond in the appropriate way to our weather forecasts.  We simply can't provide a 100% level of confidence in the forecast information we provide.  Our users must learn that they bear some responsibility for their own self-interests.   Weather hazards can present people with life-and-death situations, so in their own best interests, they need to pay attention and learn how to make the best use of what the science allows us to provide.

6 comments:

Brian Curran said...

Chuck,

I don't think the forecast was a bust per se, given the number of QLCS-like features over central and northeastern Oklahoma after dark (one such vortex hit southwest Tulsa last night). The 00Z sounding from Norman, however, revealed (mostly) meridional flow. The kiss of death for anything but QLCS vortices, which were easy to see, even on RadarScope. Anyway, once the initial outlook happened six days' prior, it's hard to stop a train of hype with that kind of momentum. The weather enterprise failed yesterday. Forecasters did not.

Jim Caruso said...

Chuck, thanks for the explanation of the reasons behind continuity in successive forecasts. I have seen continuity explicitly referenced in AFDs as a reason for not changing the forecast despite new, conflicting information. I have never seen SPC specifically mention continuity in their Convective Outlooks but I assumed it was no less a factor in their forecasts. I face the same need to avoid vacillating in the financial forecasts I issue in my own profession as a chief financial offer, so I certainly understand the issue. In any event, I think SPC's Convective Outlooks that day were measured and certainly did NOT paint a picture of a tornado outbreak, and of course the MDT risk was more about wind and hail than tornados. But I agree with you about the PDS watch and I thought it strange that SPC did not also issue a PWO given the elevated level of the other products...

Joel Genung said...

This is a very fair and accurate assessment of "What happened?" even if it can be termed as a "busted" forecast. I also agree on the bias introduced once the media is thrown into the frenzy. IMHO, they are sometimes their own worst enemy. In the end, my assumption is the forecasters will add this to their libraries of pattern recognition and benefit from the experience. And in the end, I still can sleep secure knowing that in the vast majority of instances, the SPC calls it almost perfectly every time.

Steve Holmes said...

Chuck, I get what you say about changing the forecast with the risk it could change yet again. That's a damning indictment of us the public if we mistake changing circumstances in the atmosphere for indecision among those monitoring it. Would thinking the SPC wishy-washy, though, have been any worse than thinking it wrong?

There was an amazing disconnect between what the weather sites proclaimed and the talk on Stormtrack.com. On Stormtrack, sentiment soured on the event about two days out Inconsistent model runs. Not enough moisture. Concern about the timing of the elements coming together. The word "bust" was used 24 hours in advance. That leads me to believe a lot of forecasters saw the same thing, but for whatever reason, kept quiet about it.

Is there a "herd mentality" in meteorology as there is in news, or a reluctance of broadcast mets to take a different line from the NWS/SPC (something Mike Smith said in his book on the Joplin warnings)? With all the hype leading up to Tuesday, it would have taken a major set of stones for a met to go against the flow. Are you aware of any forecasters who publicly tried to tone down the concern?

Chuck Doswell said...

Brian:
The conditions were not favorable for an outbreak of long-track violent tornadoes. The relatively low tornado day-1 tornado probabilities in a Moderate risk outlook were a realistic assessment of the threat. Earlier expectations for a "particularly dangerous situation" were overblown, and the PDS watch was not a very good call.

Steve:
No, I don't know of any forecasters who made a point of toning down the perceived threat. There may be some, but I'm unaware of them.

I don't think there's any particularly strong "herd mentality" among forecasters, but the SPC is widely (and deservedly) recognized as the experts on severe weather forecasting - but they're not perfect, of course. No one is. The office culture works against major changes to the forecast of a colleague within the SPC. The SPC has a difficult job and they do remarkably well, but that doesn't mean there's no room for post-mortem review and improvements.

Steve Holmes said...

It's interesting, Chuck, that the SPC dropped the May 8 S KS-N OK outlook from Moderate at 1730z on Day 2 to Enhanced on Day 1 0600z. I wonder if there's any carryover from April 26. These folks are scientists, but they're also human and are aware of the "bust" talk from April 26.