When you’re choosing an AI tool to incorporate into your everyday workflow, there are plenty of tools (like Arena.ai and LiveBench.ai) to help guide your decision. But where should you turn to compare the world’s most accurate weather forecasts?
Answering such a seemingly simple question is a lot harder than it might look at first glance. For some context, the weather forecast models that agencies such as the National Oceanic and Atmospheric Administration (NOAA) or European Centre for Medium-Range Weather Forecasts (ECMWF) develop and run have substantially improved over the past few decades. To observe such a trend, meteorologists typically distill “forecast skill” into a few very niche metrics: “What was the average 3-day track error for hurricane forecasts?” “What was the average error of 2-meter temperature forecast over the northern hemisphere?” “How ‘extra wiggly’ were big storm patterns in the mid-latitudes at day 5 in the forecast?”
When AI weather prediction systems (AIWP) emerged onto the scene a bit over 5 years ago, answering this question took on a new level of importance. AIWP has promised state-of-the-art forecast skill, increasingly rivaling or beating that of NOAA’s and ECMWF’s top models…at least according to their developers. Now, with so many forecast models to choose from, how do you pick the right one for your particular use case?
A team at Google DeepMind led by Stephan Rasp has made great efforts to provide answers to this question through the WeatherBench project. WeatherBench provides a standard protocol and set of summary statistics to produce “scorecards” that allow us to compare the relative strengths and weaknesses between different AI- and physics-based (NWP) weather models. While invaluable, approaches like WeatherBench are frozen in time and do not necessarily tell you how well a given weather model might be performing for today’s forecast – especially when extreme weather events are expected.
Brightband has focused on building tools to help answer this specific facet of the “accuracy” question. With initiatives like Extreme WeatherBench (EWB) we’ve made it much easier to evaluate case studies of impactful weather events. Today, we’re launching another tool – Operational WeatherBench (OWB). OWB adopts the now-standard approach established by the original WeatherBench and applies it to a suite of live, operational forecasts. We compile real-time skill scores for a suite of AI- and physics-based weather forecast models, from public data sources and from AI models that Brightband runs operationally. This currently includes models from NOAA, ECMWF, Google, Microsoft, and NVIDIA.
Similar to tools like LiveBench.ai, OWB includes a “leaderboard” which summarizes a real-world, head-to-head competition between different weather forecast models. The leaderboard evolves over time based on actual recent forecast performance, so users can see at-a-glance how well their preferred model has fared against others.
For technical users, OWB includes a timeseries view of all the WeatherBench statistics. This provides enormous context for anyone using these tools on a daily basis; forecast skill is correlated over time, as challenging weather patterns or degraded weather observations persist for a few days to a week. By focusing on real-time forecasts, OWB offers a neutral playground; it completely side-steps issues relating to yearly model training/validation splits, and focuses entirely on recent weather events to avoid subjectively choosing case studies where one model performed particularly well.
Our goal with OWB is to provide a unique, rich source of information for anyone who uses AIWP or NWP forecasts to track the weather, and to help them understand which models may be the most helpful at any given time. We will continue to expand the suite of included forecast models as new, open-source or open-weight models become available to the community, and we’re planning on bringing a similar “operational” context to our Extreme WeatherBench project, too.
If you’re interested in learning more about OWB or would like to contribute your own AIWP or NWP forecasts, please don’t hesitate to reach out to us at hello@brightband.com or on X @brightbandtech.