\n",
"
Tip: You can use these storm phase descriptions for contextualizing your findings:\n",
"\n",
"* Climatology / quiet periods: Dst is generally horizontal and nearly 0 nano-Tesla.\n",
"* Sudden Impulse: Dst rises from near 0 to positive values rapidly over a few hours.\n",
"* Storm Sudden Commencement and Main Phase: Dst drops sharply and remains significantly negative for up to several days.\n",
"* Storm Peak: Dst reaches its minimum (most negative) value.\n",
"* Recovery Phase: Dst recovers from large negative values back to climatology, near 0 nano-Tesla."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9kEgSgWZbeCN"
},
"source": [
"
\n",
"Exercise: Degraded Observations\n",
"\n",
"Degrade the instrument measurements and run the model to see how the performance is impacted. Start simple by adding Gaussian noise (mean 0), to the least important and the most important input parameters (aka features) and evaluating a specific event.\n",
"
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"Congragulations on engaging with the learning objectives of this
Chapter 2 LSTM focused notebook--the benchmark from the NOAA MagNet competition. There is one additional
Chapter 1 notebook in the MagNet LSTM series, on model development in case you didn't start there.\n",
"\n",
"There is an additional NCAI notebook in preparation for this MagNet series:\n",
"A higher performing ensemble Convolutional Neural Netowork (CNN) from the NOAA Geomagnetism team based on the 2nd place entry from the MagNet competition. \n",
"As mentioned in an earlier section, this notebook's precursor is the [TAI4ES Space Weather CNN Notebook](https://github.com/ai2es/tai4es-trustathon-2022/tree/main/space)\n",
"\n",
"Additionally, a web search will provide other
Dst modeling notebooks and publications using ML techniques."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Examples in the community\n",
"\n",
"For a comprehensive treatment of the need to build robust predictions of the
Dst space weather storm indicator (e.g. for magnetic navigation applications), see Nair et al., 2023 and references therein:\n",
"* Nair et al., 2023 (
in press) (TODO: Update with public URL as soon as available),\n",
"\n",
"For a summary, see:\n",
"* https://www.drivendata.org/competitions/73/noaa-magnetic-forecasting/\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data statement\n",
"The competition discussed above used
public data for development and the public leaderboard. A
private dataset was kept internal during the competition for use in scoring by the organizers. Since the competition has passed, both datasets are publicly accessible from NOAA.\n",
"\n",
"All data used in this notebook are publicly available here:\n",
"* https://ngdc.noaa.gov/geomag/data/geomag/magnet/public.zip\n",
"* https://ngdc.noaa.gov/geomag/data/geomag/magnet/private.zip"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## References\n",
"\n",
"* Nair, M., Redmon, R.J., Young, L.Y., Chulliat, A., Trotta, B., Chung, C., Lipstein, G., Slavitt, I. (2023),\"MagNet - a data-science competition to predict Disturbance Storm-time index (Dst) from solar wind data\", Space Weather,
In Press.\n",
"* [CIRES GeoMag MagNet repository](https://github.com/liyo6397/MagNet/), TODO: update URL to new CIRES repo.\n",
"* [Trustworthy Artificial Intelligence for Environmental Science 2022 Summer School](https://www2.cisl.ucar.edu/events/tai4es-2022-summer-school), TAI4ES, accessed July 2022.\n",
"* [TAI4ES Space Weather Notebooks (LSTM, CNN)](https://github.com/ai2es/tai4es-trustathon-2022/tree/main/space), GitHub, accessed July 2022.\n",
"* [MagNet: Model the Geomagnetic Field](https://ngdc.noaa.gov/geomag/mag-net-challenge.html), NOAA, accessed March 2022.\n",
"* Chung, C. (2020), \"HOW TO PREDICT DISTURBANCES IN THE GEOMAGENTIC FIELD WITH LSTMS - BENCHMARK\", Blogpost, Accessed March 2022, Available Online: https://drivendata.co/blog/model-geomagnetic-field-benchmark/.\n",
"* DrivenData (2020), \"MagNet: Model the Geomagnetic Field\", Web Resource, Accessed March 2022, Available Online: https://www.drivendata.org/competitions/73/noaa-magnetic-forecasting/.\n",
"* [Interpretable Machine Learning by Christop Molnar](https://christophm.github.io/interpretable-ml-book/shap.html)\n",
"* Redmon, R. J., Seaton, D. B., Steenburgh, R., He, J., & Rodriguez, J. V. (2018). September 2017's geoeffective space weather and impacts to Caribbean radio communications during hurricane response. Space Weather, 16, 1190–1201. https://doi.org/10.1029/2018SW001897"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Metadata\n",
" * Language / package(s):\n",
" * Language: Python, \n",
" * Packages: Keras Tensor Flow, Matplotlib, Numpy, Pandas, Scikit-learn\n",
" * Scientific domain:\n",
" * Space Weather, Geomagnetic modeling\n",
" * Application keywords\n",
" * Magnetic Navigation\n",
" * Geophysical keywords\n",
" * Disturbance Storm Index (
Dst), Solar Wind\n",
" * AI keywords\n",
" * Long Short-Term Memory (LSTM)\n",
" * Explainable AI (XAI), Permutation Feature Importance"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## License\n",
"\n",
"### Software and Content Description License\n",
"Software code created by U.S. Government employees is not subject to copyright in the United States (17 U.S.C. §105). The United States/Department of Commerce reserve all rights to seek and obtain copyright protection in countries other than the United States for Software authored in its entirety by the Department of Commerce. To this end, the Department of Commerce hereby grants to Recipient a royalty-free, nonexclusive license to use, copy, and create derivative works of the Software outside of the United States."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Disclaimer\n",
"\n",
"> This Jupyter notebook is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA Jupyter notebooks are provided on an 'as is' basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this Jupyter notebook will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government."
]
}
],
"metadata": {
"accelerator": "GPU",
"colab": {
"collapsed_sections": [
"v003LoxwbeB6",
"kRzCSuRwbeB7",
"YIeZxbJXbeB7",
"CdZDisojbeB8",
"kJQsAZGlbeB8",
"GCrRyAmibeB9",
"5WEhRi6RMX0o",
"AVRI8V_AI3NS",
"ZDtowUzbMXNx",
"Vk0h7YPGMxUw",
"Is66dqN0M3VP",
"5HoW8LxqM8eS",
"J261z_FvOK_p",
"HqKPO3qXOWDb",
"_islsePpOgRI",
"DHXdMwzzTHaF",
"wkwxYtOqT8WT",
"nl-bgU9WHXQO",
"yLbZKYfgEjoT",
"dWs2MeMe0ZFm",
"acm3GxIUbeCL",
"jn5uRW1kDq2p",
"9kEgSgWZbeCN"
],
"gpuType": "V100",
"machine_shape": "hm",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.16"
}
},
"nbformat": 4,
"nbformat_minor": 4
}