An Uber Journey in Distributed Deep Learning

In this video from the Stanford HPC Conference, Alex Sergeev from Uber presents: An Uber Journey in Distributed Deep Learning.

"Uber engineering covers a wide development terrain from autonomous vehicles and trip forecasting to fraud detection and more. Inspired by the works of Baidu, Facebook, et al., Uber has developed “Horovod”, an open source distributed framework for TensorFlow. Inducing a ‘good harvest’, as asserted by ingenious branding influenced by the traditional Russian folk dance of linking hands and dancing in a circle during a planting season, Horovod makes it easy to scale training of deep learning models from a single GPU to hundreds – on one or across multiple servers.
Uber explores the journey in distributed deep learning in two parts. The HPC problem, tools to perform distributed computation; and the ML problem, tweaking model code and hyperparameters.
The talk will touch upon practical aspects, the mechanisms of deep learning training, mechanics of Horovod and performance expectations on standard models such as Inception V3 and ResNet-101; to achieve practical results, using HPC techniques and technologies such as allreduce, RDMA and MPI, training convolutional networks and LSTMs on your favorite supercomputer in hours, versus days/weeks, with the same final accuracy."

and