AI/ML Data Center Design - Part 1

Petr Lapukhov joins Jeff Doyle and Jeff Tantsura to discuss the finer points of AI/ML Data Center design.

[LibertypopUK]

All three legends !!! Such a pleasure to listen to

[Diego-npsr]

thank you guys!! that was very precious content

[ChrisWhyte]

Really enjoyed this one! Greatly appreciate the insight.

[samyogdhital]

Introduction (00:00:00 - 00:02:12)

• The hosts, Jeff and Stu, introduce the show "Between Two Nerds, " which discusses various aspects of the networking industry. They also introduce their guest, Peter, who has experience in web-scale infrastructure, networking, automation, and software operations at companies like Microsoft, Facebook, and Nvidia. (00:00:00 - 00:02:12)

Topic: AI Data Center Design (00:02:12 - 00:03:04)

• The main topic of the episode is AI data center design and its unique requirements. They plan to discuss the drivers for AI data center design, emphasizing the importance of understanding AI/ML workflows. (00:02:12 - 00:03:04)

AIML Workflows and Network Design (00:03:04 - 00:04:00)

• The discussion highlights that while network design principles remain the same, the consequences of incorrect design are much greater in AI data centers due to the massive data sets involved in machine learning training and inference. (00:03:04 - 00:03:50)
• The speed of change in AI is rapid, with cluster sizes growing from 4K GPUs to hundreds of thousands, making network design more complex and critical. (00:03:50 - 00:04:00)

GPU Dominance in AI (00:04:00 - 00:07:01)

• Nvidia's dominance in the GPU market is acknowledged, and the discussion shifts to why GPUs are fundamental for AI processing clusters. (00:04:00 - 00:06:01)
• It's explained that GPUs, initially designed for graphics, have an architecture that is massively parallel, making them suitable for machine learning tasks that also require massive parallelism. (00:06:01 - 00:07:01)
• The development of APIs by Nvidia allowed researchers to run matrix multiplications on GPUs, leading to their dominance in AI training. (00:07:01 - 00:09:35)

GPU Training and Networking (00:09:35 - 00:11:09)

• The conversation details how the need to parallelize training across multiple GPUs led to the necessity for high-performance networking. (00:09:35 - 00:10:00)
• Initially, CPUs were also used for training, but GPUs became dominant due to their superior performance with massive parallelism. (00:10:00 - 00:11:09)

Google TPUs vs. Nvidia GPUs (00:13:42 - 00:15:19)

• Google's use of Tensor Processing Units (TPUs) is discussed, noting that they are optimized for matrix multiplications but are less flexible than GPUs. (00:13:42 - 00:14:10)
• It is noted that while TPUs have their place, GPUs are considered more efficient and programmable. The flexibility of GPUs allows them to adapt to rapidly changing workloads. (00:14:10 - 00:15:19)

Open Source Contributions (00:17:40 - 00:18:07)

• The open-source nature of Nvidia’s software infrastructure, especially CUDA, is highlighted as a key factor in its growth. The open-source model allowed for worldwide contributions, which helped build its system around GPUs. (00:17:40 - 00:18:07)

Model Size and GPU Scaling (00:22:22 - 00:25:36)

• The discussion transitions to how model size drives the network size and number of GPUs needed for training. As models grow larger, more GPUs are required. This is not just due to computational needs but also to fit the model in memory. (00:22:22 - 00:23:05)
• Training times are also a key driver. Larger and more scalable hardware allows faster training, which is crucial for time to market and better services. (00:23:05 - 00:24:15)
• It’s noted that the time to train grows as a power law, whereas adding GPUs provides linear improvements. There are practical limitations to the number of GPUs that can be placed in a data center due to space and power. (00:24:15 - 00:25:36)

Training and Inference (00:30:24 - 00:32:32)

• The presenters describe the AI workflow as a linear process, where data is used to train models, which are then used for inference. They also highlight that this is an iterative process, as feedback from inference can further improve the model. (00:30:24 - 00:31:05)
• They discuss how training is a compute-intensive process that requires a high-performance network, while inference needs low latency and a fast response time for end-users. (00:31:05 - 00:32:32)

Importance of Low Latency for Inference (00:32:32 - 00:34:44)

• Low latency becomes more critical for machine-to-machine interactions. Human attention spans need quick responses, so if models don't provide output quickly enough, consumers may move on. (00:32:32 - 00:33:10)
• It is also noted that latency is becoming an important constraint for machine-to-machine inference as more and more applications call other applications to do inferencing. Optimizing for latency becomes more important. (00:33:10 - 00:34:44)

AI Network Characteristics (00:37:36 - 00:40:00)

• AI networks are optimized for performance, not for cost savings. It is more important to ensure the maximum performance from your network than to save money when building it. (00:37:36 - 00:38:30)
• Job completion time is the primary metric. The network must not hinder job completion. (00:38:30 - 00:39:30)
• AI networks are not under-subscribed, and the goal is to maintain network utilization above 90%. (00:39:30 - 00:40:00)

Power Consumption and Network Layers (00:40:00 - 00:42:22)

• Power is a major constraint. While trying to build larger data centers for larger clusters, it is crucial to consider the amount of power available. (00:40:00 - 00:41:15)
• AI clusters have multiple types of networks, including IP networks, scale-up networks (like Nvidia's NVLink), and platform-specific libraries. (00:41:15 - 00:42:22)

Historical Perspective and GPU Direct (00:43:53 - 00:47:58)

• The evolution from Hadoop to today's AI clusters is discussed. In the early days, TCP/IP was often used, but this became inefficient with the massive data flows of GPU-based AI. (00:43:53 - 00:45:45)
• With RDMA over Converged Ethernet (RoCE) and GPU direct, the efficiency of data transfer has significantly improved. GPU direct provides zero-copy data transfers directly from GPU memory to the network, bypassing the CPU. (00:45:45 - 00:47:58)

NVLink Technology (00:50:15 - 00:52:30)

• NVLink is highlighted as a key technology for connecting GPUs within a server, offering much higher bandwidth than scale-out networking options. Current and next-generation plans show increasing numbers of GPUs that can be connected via NVlink, which has also moved from internal to the server to being part of the rack. (00:50:15 - 00:52:30)

RDMA and GPU Direct Explained (00:52:30 - 00:53:30)

• A brief explanation of Remote Direct Memory Access (RDMA), highlighting that it enables direct access to memory over a network, bypassing the CPU and the kernel. (00:52:30 - 00:53:30)
• GPU direct allows the NIC to read memory directly from the GPU and write it on the wire. (00:53:30 - 00:54:55)

Parallelization Techniques and NCCL (00:55:57 - 00:57:30)

• The most common parallelization techniques include data, model, and tensor parallelism. Communication libraries such as NVIDIA Collective Communications Library (NCCL) play a crucial role in enabling communication between GPUs for training. (00:55:57 - 00:57:30)

Data Locality and Server Design (00:57:30 - 01:00:00)

• If a single GPU has enough memory to hold the entire dataset, there is no need for external networking. Data locality in single servers is discussed, as well as server designs that help optimize training. (00:57:30 - 01:00:00)

NVIDIA Communication Library and Data Parallelism (01:00:00 - 01:02:28)

• Discussion about how NVIDIA Collective Communications Library (NCCL) was designed to be optimized for GPUs. The importance of data parallelism, which involves slicing the data into sub-slices, and model parallelism is discussed. They note that the data can be stretched across the cluster, achieving large computing capabilities. (01:00:00 - 01:02:28)

[RommelsAsparagus]

Great intro. A couple of questions, has nVidia thought about full immersion cooling for their GPU clusters instead of just cold water on the backplane? Also, do you think inference data centers will become a larger market than training data centers? Would you distribute inference in something like a CDN to optimize latency? Thanks.