Michaela K. Mooney
During periods of high geomagnetic activity, particles precipitating into the atmosphere can cause auroral emission producing amazing displays across the sky. However, auroral emission can also affect long-range radio communications and the accompanying geomagnetic storms can induce strong currents in oil pipelines and electricity transmission lines at ground level. These effects impact industry sectors such as aviation, energy and defence. Forecasting the location and probability of aurora is therefore of interest to many end users. In addition, forecasting when the aurora may be visible can also be a key tool in promoting public awareness and engagement with space weather.
The OVATION-Prime 2013 auroral precipitation model (Newell et al., 2014) is currently in operation at the UK Met Office and delivers a 30-minute forecast of the probability of observing the aurora. By applying terrestrial weather forecast verification techniques, we evaluate the performance of this operational implementation of OVATION against auroral emission boundaries determined from global far-ultraviolet (FUV) observations captured by the IMAGE satellite between 2000-2002.
Our analysis shows that, as a deterministic model, OVATION-Prime 2013 performs well at predicting the location of the auroral oval under nominal space weather conditions but the probabilities of aurora occurring forecast by the model generally tend to be underpredicted.
The results of this study will assist space weather forecasters at the Met Office with interpreting the output of the auroral forecast model in daily operations and improve published space weather reports.
The energetic particles in our upper atmosphere associated with the aurora can disrupt radar signals and radio communication, posing a significant risk to the aviation and defence sectors. The OVATION-Prime 2013 model (OP-2013; Newell et al., 2014) is the current state-of-the-art aurora forecast model that is used in daily space weather forecasts at the Met Office Space Weather Operations Centre, however, the operational version of OP-2013 has not yet been fully evaluated. In this study, we use weather forecast verification techniques to evaluate the performance of OP-2013 against satellite observations of the aurora. Our key results show that OP-2013 performs well at predicting the location of the auroral oval, with a ROC score of 0.83. Comparing the forecast probabilities of aurora occurring against the observed occurrence of aurora, OP-2013 tends to under-predict the probability of aurora occurring.
Slide 2
OP-2013 outputs a map showing the predicted location and probability of aurora occurring 30 minutes ahead of time, in geographic coordinates. OP-2013 is run every 30 minutes using upstream solar wind parameters measured at L1.
The IMAGE satellite took observations of the aurora from space in far ultraviolet (FUV) wavelengths. From space looking down on the Earth, the auroral forms an oval shape (the “auroral oval”) around the poles. Longden et al. (2010) used the IMAGE satellite data to identify the poleward and equatorward boundaries of the auroral oval. We use these boundaries to define where the aurora occurred to compare with the corresponding auroral forecasts from OP-2013.
Slide 3
The two verification analysis techniques we apply are relative operating characteristic (ROC) curve analysis and reliability analysis.
ROC curves help us to evaluate how well the model discriminates between regions of aurora and no aurora, i.e. how well the model predicts the location of the auroral oval. ROC curves are obtained by plotting the hit rate against the false alarm rate. A skilful model will have a ROC curve that tends towards the top left corner. The ROC score is calculated from the area under the ROC curve, with a maximum ROC score of 1 for very skilful models. The ROC curve for OP-2013 tends towards the top left corner and has a ROC score of 0.83 which tells us that OP-2013 performs well at predicting the location of the auroral oval.
Reliability diagrams evaluate the probabilities of aurora occurring forecast by the model against the observed occurrence of the aurora. Our results show that for forecast probabilities of aurora occurring less than 0.8, the probabilities are under-forecast by up to a factor of 6.5. For probabilities of aurora forecast greater than or equal to 0.90, the probabilities are over-forecast by a up to approximately 20%. The results of the reliability analysis can be used to correct the probabilities forecast by the model.
Slide 4
In summary,
★ Our results show that the OP-2013 model performs well at predicting the location of the aurora with a high ROC score of 0.83. Further analysis has shown that this breaks down slightly under higher levels of geomagnetic activity, when the aurora are more active.
★ Our evaluation of the forecast probabilities of aurora occurring show that the OP-2013 model tends to under-predict the probability of auroral occurrence, often significantly.
★ The results of this study will be provided to Met Office Space Weather Forecasters to improve their interpretation of the OP-2013 auroral forecasts.
For more details on this study, please see our A&G article: How well do we forecast the aurora? Mooney et al., (2019). An alternative link to the open access version of the article can be found here.