2020-10-02 - Albert Reuther - Enabling Interactive, On-Demand HPC for Rapid Prototyping and ML

Abstract: Fordecades, the use of HPC systems was limited to those in the physical scienceswho had mastered their domain in conjunction with a deep understanding of HPCarchitectures and algorithms. During these same decades, consumer computingdevice advances produced laptops, tablets, and smartphones that allow millionsto interactively develop and share code projects using high productivitylanguages and environments. The HPC community faces many challenges associatedwith guiding researchers from disciplines that routinely utilize highproductivity interactive tools to effectively use HPC systems, since it is fruitlessto expect them to give up the interactive, on-demand nature of their workflows.

For over adecade, MIT Lincoln Laboratory has been supporting interactive, on-demand HPCby seamlessly integrating familiar high productivity tools to provide userswith an increased number of design turns, rapid prototyping capability, andfaster time to insight. In this talk, we discuss the lessons learned whilesupporting interactive, on-demand high performance computing from theperspectives of the users and the team supporting the users and the system. Atits core, it involves an expansion of what the HPC ecosystem paradigm entailsincluding expansions in system architecture, scheduler policies, metrics ofsuccess, and supported software development environments and tools. We concludewith how our team supports users and the systems in this paradigm expansion.