The National Hurricane Center accurately predicted the path of Hurricane Irma, which struck southwest Florida on Sept. 8. This was not an easy storm to forecast, though, as computer models disagreed with one another on important details right up until landfall.
But in general, the European model, which is run by the European Center for Medium-range Weather Forecasting (ECMWF), performed far better than the premiere U.S. model, known as the Global Forecast System (GFS).
Hurricane Irma is one more in a long line of storms to shine a spotlight on problems with the GFS, particularly at intermediate to longer timescales. The issue gained prominence after Hurricane Sandy struck New Jersey in October 2012, which the European model hinted at at least a week in advance. The GFS model, however, didn’t catch on to the storm’s unusual track until about 5 days in advance.
Given weather’s sizable influence on the economy, the growing gap between the reliability of non-U.S. weather models and their international competitors could put billions of people at risk.
Critics of the GFS say it needs to be improved with greater computer processing power. In addition, they say, the model needs to process weather information in more advanced ways, with greater resolution in both the horizontal and vertical scale, since the weather on the surface depends heavily on what is going on in the mid-to-upper atmosphere.
The National Weather Service is aware that it’s running behind in the so-called weather modeling wars, and the agency is touting recent improvements as well as planned changes set to take place in the next few years.
Category 5 Hurricane Irma seen from space before striking the northern Leeward Islands.
Michael Farrar, who heads the Environmental Modeling Center (EMC), which is the lead office within the National Oceanic and Atmospheric Administration (NOAA) that develops and operates computer models, said “it’s no secret” that the GFS has been behind the competition.
“While it’s continued to improve remarkably over time… it’s consistently behind the European model,” Farrar said in an interview.
He pointed out that the GFS accurately predicted the movement of Hurricane Harvey, which devastated the Houston area with the most extreme rainstorm in U.S. history, but that the forecasts for Irma were less reliable.
Still, he cautioned that the European model, too, missed the eventual track the storm took, which was just inland in southern Florida.
“Even the European model has got its issues,” Farrar said.
No computer model or human forecaster got Hurricane Irma exactly right. Forecasters relied heavily on ensemble forecasting to gauge the storm’s threat to the U.S. well in advance. Such forecasting involves running a computer model numerous times while varying the initial conditions slightly each time.
Evolution in Hurricane Irma ensemble track forecasts from the European model.
Ensembles can help reveal how uncertain a forecast is, since if a storm’s track is extremely sensitive to small changes in initial conditions, that is usually an indication of greater uncertainty. Ensemble forecasting can also help compensate for model weaknesses or biases.
The weather model wars are not an issue only for meteorologists and weather geeks to be concerned with. With high impact events, like a blizzard or a hurricane like Irma, lives are on the line, as well as big time dollars.
Hurricane Irma track forecasts from the GFS model.
Hurricane Irma track forecasts from the European model.
“The average person watching their local or national news hear terms such as American and Euro model and many learn that the Euro is better. CNN showed side-by-side comparisons between the GFS and ECMWF,” said Ryan Maue, a private sector meteorologist and a proponent of a major new effort to try to boost U.S. computer modeling capabilities.
Overall, during the life of Hurricane Irma, the European model had better run-to-run consistency than the GFS did, which, along with its track record on this and other storms, caused forecasters to give it added weight while making their projections for Hurricane Irma.
The GFS model did, however, perform better than the European model when it came to short-range forecasts near the time of landfall, when it correctly suggested that the storm would take a more inland track, rather than moving over or just west of Tampa Bay.
Forecast skill score comparisons, maintained by Brian Tang at the University of Albany, show that the European model was far superior to the GFS model during the long trek that Hurricane Irma took from off the coast of Africa, through the northern Leeward Islands, the Caribbean, Bahamas, Cuba, and then into the mainland U.S.
Here’s how to read this chart: The GFS model is represented by the acronym, AVNO, while the ECMWF is the European model. All the others are models from other countries and groups, such as the CMC, or Canadian model, and the UKM, from the UK Met Office. Also, the acronym, “OFCL,” represents the official Hurricane Center human forecast.
Annotated version of model verification scores for weather models’ forecasts for Hurricane Irma.
Image: Brian Tang/u albany/mashable
Forecast models are typically evaluated by their forecast skill at different timescales, including three, four, and five days. You can see from this chart that, for example, 120 hours in advance, the official forecast was off by around 300 kilometers. The European model had an average error of just 175 kilometers at that timescale, whereas the average 120-hour error of the GFS model at 120 hours was about 475 kilometers.
The models were more tightly clustered at 12- and 24-hour timescales, but with the European model still coming out on top.
Warnings that were overruled
The European Center is planning to build a next-generation supercomputing center in Italy and continues to outpace the U.S. in terms of forecast accuracy and computing power. Lessening the impact of this disparity, though, is the fact that American forecasters do have access to the European models’ simulations, through a cooperative agreement.
The U.S. does not, however, have unfettered access to all the ECMWF’s products, as there are limits on what the Weather Service can do in terms of publicly disseminating model projections. Weather forecasting is vital to national security, given its role in warfighting and protecting critical infrastructure in the U.S. This makes it impossible for the Weather Service to simply give up the race, and use other countries’ models exclusively.
For now, forecasters are stuck with a temperamental model that can fail to catch on to upcoming threats until days after the European model has sounded the alarm. This was the case with Hurricane Sandy in 2012, when the European model showed the storm would hit the Mid-Atlantic more than a week in advance, and the GFS eventually came around to the same conclusion.
The GFS model went through an upgrade earlier this year, but it may not have improved things when it comes to hurricane forecasting. The upgrade improved the model’s horizontal resolution, but not its vertical resolution. In general, the greater the resolution, the more details a model can pick up, and the more accurate a projection will be. According to a NOAA budget document, the new core, known as FV3, will enable NOAA “to simulate individual clouds and storms at resolutions as fine as one to three kilometers.”
As Mashable reported in March, the Hurricane Center warned leadership at the National Weather Service that the upgrade might actually reduce the accuracy of track forecasts for Atlantic tropical storms and hurricanes by 9 to 10 percent.
One slide in a presentation to senior management summarized the Hurricane Center’s reservations. It bluntly stated: “The loss of short- to medium-range [tropical cyclone] track and intensity forecast skill for the Atlantic basin in the proposed 2017 GFS is unacceptable to the National Hurricane Center.”
Farrar, however, said that the EMC reviewed the NHC’s objections, and conducted more simulations. These failed to show the 10 percent decline in the accuracy of hurricane track forecasts.
“We re-factored a couple of things, we reran the test cases, that difference went away, the hurricane center withdrew its objection,” Farrar said.
What we need to do
When it comes to improving the GFS model, NOAA has further improvements planned. For example, the primary and backup computer system is set to go from 2.8 petaflops to 4.2 petaflops of computing power.
In addition, the agency is upgrading how the model ingests and processes weather data, which could yield big improvements.
This will “close the gap pretty quickly with the Europeans,” Farrar said.
Still even with these improvements, the fact is that NOAA is outmanned and underfunded when compared to the Europeans and other modeling groups. The U.S. has no centralized, well-funded center solely devoted to weather modeling on a specific timescale, whereas that’s all the ECMWF does.
“The staff that I can afford to devote to my global model is far less than what the European center does,” Farrar said. “I’ve got a lot less resources if you’re gonna compare apples to apples.”
For example, the ECMWF employed 300 people in fiscal year 2015, and spent about $68 million British pounds. The U.S. modeling center, which maintains and upgrades at least 20 models of the ocean and atmosphere, has 175 employees and contractors, according to Susan Buchanan, a Weather Service spokeswoman.
Instead, through the EMC, NOAA operates about 20 computer models, including some that focus on the oceans, some on weather, and a plume model to track the spread volcanic ash and hazardous air pollutants.
In Farrar’s words, “Our mission space is much more complex, and so our computing resources are spread over our” many missions. He said the U.S. government spreads out computer modeling groups at various agencies and cabinet departments, which can make it difficult to make significant improvements in one area.
“As a nation we’re spending a lot more than the Europeans but we’re not spending together,” he said, noting that a new partnership aims to fix this problem.
Farrar says the NWS would like more resources, but is trying to do more with less in the meantime. “Obviously any scientific organization can do a lot more with more,” he said, adding: “It’s unlikely that we’re gonna get more. We should do more with what we’ve got.”
Farrar’s office is now working with research laboratories within NOAA, NASA, the Navy, as well as NASA to improve their modeling.
President Donald Trump’s Fiscal Year 2018 budget request actually sought a cut in weather and water modeling, though this has been met with a skeptical reception on Capitol Hill.
“There is no reason why [the] U.S. cannot replicate the ECMWF or UKMET success in weather forecasting immediately, whether that’s done at the government level or with private industry,” Maue said in a Twitter message.
“Regardless, as we enter into an even more hyper-digitized world of automation, weather forecasts are going to be even more critical to maintain our national competitiveness and economic power,” he said.
Meanwhile, private companies are increasingly jumping into the fray, seeing business opportunities in weather prediction. Panasonic and IBM have developed their own models, and IBM’s even has a weather-related name: “Deep Thunder.”
That’s certainly a lot catchier than “GFS.”