Tag: schevenhoven

Supermodeling: improving predictions with an ensemble of interacting models

Schevenhoven , F., Keenlyside, N., Counillon, F., Carrassi, A., Chapman, W.E., Devilliers, M., Gupta, A., Koseki, S., Selten, F., Shen, M.L., Wang, S. 2023: Supermodeling: improving predictions with an ensemble of interacting models. BAMS. https://doi.org/10.1175/BAMS-D-22-0070.1

Summary: The modeling of weather and climate has been a success story. The skill of forecasts continues to improve and model biases continue to decrease. Combining the output of multiple models has further improved forecast skill and reduced biases. But are we exploiting the full capacity of state-of-the-art models in making forecasts and projections? Supermodeling is a recent step forward in the multimodel ensemble approach. Instead of combining model output after the simulations are completed, in a supermodel individual models exchange state information as they run, influencing each other’s behavior. By learning the optimal parameters that determine how models influence each other based on past observations, model errors are reduced at an early stage before they propagate into larger scales and affect other regions and variables. The models synchronize on a common solution that through learning remains closer to the observed evolution. Effectively a new dynamical system has been created, a supermodel, that optimally combines the strengths of the constituent models. The supermodel approach has the potential to rapidly improve current state-of-the-art weather forecasts and climate predictions. In this paper we introduce supermodeling, demonstrate its potential in examples of various complexity, and discuss learning strategies. We conclude with a discussion of remaining challenges for a successful application of supermodeling in the context of state-of-the-art models. The supermodeling approach is not limited to the modeling of weather and climate, but can be applied to improve the prediction capabilities of any complex system, for which a set of different models exists.

Link to publication. You are most welcome to contact us or the corresponding author(s) directly, if you have questions.

Training a supermodel with noisy and sparse observations: a case study with CPT and the synch rule on SPEEDO – v.1

Schevenhoven, F., Carrassi, A. 2022: Training a supermodel with noisy and sparse observations: a case study with CPT and the synch rule on SPEEDO – v.1. Geosci. Model Dev. https://doi.org/10.5194/gmd-15-3831-2022

Summary: As an alternative to using the standard multi-model ensemble (MME) approach to combine the output of different models to improve prediction skill, models can also be combined dynamically to form a so-called supermodel. The supermodel approach enables a quicker correction of the model errors. In this study we connect different versions of SPEEDO, a global atmosphere-ocean-land model of intermediate complexity, into a supermodel. We focus on a weighted supermodel, in which the supermodel state is a weighted superposition of different imperfect model states. The estimation, “the training”, of the optimal weights of this combination is a critical aspect in the construction of a supermodel. In our previous works two algorithms were developed: (i) cross pollination in time (CPT)-based technique and (ii) a synchronization-based learning rule (synch rule). Those algorithms have so far been applied under the assumption of complete and noise-free observations. Here we go beyond and consider the more realistic case of noisy data that do not cover the full system’s state and are not taken at each model’s computational time step. We revise the training methods to cope with this observational scenario, while still being able to estimate accurate weights. In the synch rule an additional term is introduced to maintain physical balances, while in CPT nudging terms are added to let the models stay closer to the observations during training. Furthermore, we propose a novel formulation of the CPT method allowing the weights to be negative. This makes it possible for CPT to deal with cases in which the individual model biases have the same sign, a situation that hampers constructing a skillfully weighted supermodel based on positive weights. With these developments, both CPT and the synch rule have been made suitable to train a supermodel consisting of state of the art weather and climate models.

Link to publication. You are most welcome to contact us or the corresponding author(s) directly, if you have questions.

Training of supermodels in the context of weather and climate forecasting

Schevenhoven, Francine (2021-02-08). Training of supermodels in the context of weather and climate forecasting (PhD thesis, University of Bergen, Bergen, Norway). https://bora.uib.no/bora-xmlui/handle/11250/2727454 .

Summary: Given a set of imperfect weather or climate models, predictions can be improved by combining the models dynamically into a so called `supermodel’. The models are optimally combined to compensate their individual errors. This is different from the standard multi-model ensemble approach (MME), where the model output is statistically combined after the simulations. Instead, the supermodel can create a trajectory closer to observations than any of the imperfect models. By intervening during the forecast, errors can be reduced at an early stage and the ensemble can exhibit different dynamical behavior than any of the individual models. In this way, common errors between the models can be removed and new, physically correct behavior can appear.
In our simplified context of models sharing the same evolution function and phase space, we can define either a connected or a weighted supermodel. A connected supermodel uses nudging to bring the models closer together, while in a weighted supermodel all model states are replaced at regular time intervals (i.e., restarted) by the weighted average of the individual model states. To obtain optimal connection coefficients or weights, we need to train the supermodel on the basis of historical observations. A standard training approach such as minimization of a cost function requires many model simulations, which is computationally very expensive. This thesis has focused on developing two new methods to efficiently train supermodels. The first method is based on an idea called cross pollination in time, where models exchange states during the training. The second method is a synchronization-based learning rule, originally developed for parameter estimation.
The techniques are developed on low-order systems, such as Lorenz63, and later applied to different versions of the intermediate-complexity global coupled atmosphere-ocean-land model SPEEDO. Here the observations are from the same models, but with different parameters. The applicability of the method to real observations is tested using sensitivity to noisy and incomplete data. The characteristics the individual models should have in order to be combined together into a supermodel are identified, as well as which physical variables should be connected in a supermodel, and which ones should not. Both training methods result in supermodels that outperform both the individual models and the MME, for short term predictions as well as long term simulations. Furthermore, we show that the novel use of negative weights can improve predictions in cases where model errors do not cancel (for instance, all models are too warm with respect to the truth). A crucial advantage of the proposed training schemes is that in the present context relatively short training periods suffice to find good solutions. Although the validity of our conclusions in the context of real observations and model scenarios has yet to be proved, our results are very encouraging. In principle, the methods are suitable to train supermodels constructed using state-of-the art weather and climate models.

Link to publication. You are most welcome to contact us or the corresponding author(s) directly, if you have questions.

Improving weather and climate predictions by training of supermodels.

Schevenhoven, F., F. Selten, A. Carrassi, Keenlyside, N. 2019: Improving weather and climate predictions by training of supermodels. Earth Syst. Dynam., 10, 789–807. https://doi.org/10.5194/esd-10-789-2019

Summary: Recent studies demonstrate that weather and climate predictions potentially improve by dynamically combining different models into a so-called “supermodel”. Here, we focus on the weighted supermodel – the supermodel’s time derivative is a weighted superposition of the time derivatives of the imperfect models, referred to as weighted supermodeling. A crucial step is to train the weights of the supermodel on the basis of historical observations. Here, we apply two different training methods to a supermodel of up to four different versions of the global atmosphere–ocean–land model SPEEDO. The standard version is regarded as truth. The first training method is based on an idea called cross pollination in time (CPT), where models exchange states during the training. The second method is a synchronization-based learning rule, originally developed for parameter estimation. We demonstrate that both training methods yield climate simulations and weather predictions of superior quality as compared to the individual model versions. Supermodel predictions also outperform predictions based on the commonly used multi-model ensemble (MME) mean. Furthermore, we find evidence that negative weights can improve predictions in cases where model errors do not cancel (for instance, all models are warm with respect to the truth). In principle, the proposed training schemes are applicable to state-of-the-art models and historical observations. A prime advantage of the proposed training schemes is that in the present context relatively short training periods suffice to find good solutions. Additional work needs to be done to assess the limitations due to incomplete and noisy data, to combine models that are structurally different (different resolution and state representation, for instance) and to evaluate cases for which the truth falls outside of the model class.

Link to publication. You are most welcome to contact us or the corresponding author(s) directly, if you have questions.