2022 High Performance Computing Practical Lecture 10.2 Distributed Deep Learning Part2 👨‍💻

Practical Lecture 10.2 - Distributed Deep Learning 👨‍💻 - Part Two

Invited Lecture given by PhD Student Rocco Sedona

Advanced Scientific Computing
16 university lectures with additional practical lectures for hands-on exercises in context
The University of Iceland, School of Engineering and Natural Sciences
Faculty of Industrial Engineering, Mechanical Engineering and Computer Science

Spring 2022

Course Outline
1. High Performance Computing
2. Parallel Programming with MPI
3. Parallelization Fundamentals
4. Advanced MPI Techniques
5. Parallel Algorithms & Data Structures
6. Parallel Programming with OpenMP
7. Hybrid Programming & Patterns
8. Debugging & Profiling & Performance Analysis
9. Accelerators & Graphical Processing Units
10. Parallel & Scalable Machine & Deep Learning
11. HPC in Health & Neurosciences
12. Computational Fluid Dynamics & Finite Elements
13. Systems Biology & Bioinformatics
14. Molecular Systems & Material Sciences
15. Terrestrial Systems & Climate
16. Epilogue

Lecture Outline

Part One: Introduction
Key concepts from Lecture 10.1
Other Important Concepts
Distributed Training: Theory

Part Two: Frameworks
Horovod
DeepSpeed
Others
A Remote Sensing Application