How EFA works and why we don't use infiniband in the cloud.

AWS’s compute infrastructure is very much not like a ‘normal’ supercomputer (whatever that is). We don’t start with a blank page every few years and design the next big system. It’s more like a city where we have to build on what’s there already, renovate occasionally, and push for bigger and better and faster whilst keeping the lights on at all times.

That leads to a bunch of design decisions that drive our engineers in a very different direction and our Elastic Fabric Adapter is an example of just that. Brian Barrett (one of our Principal Engineers in the HPC team) joins us this week to talk about the genesis of EFA, how it works, and why it convinced us that we could do without specialist fabrics like Infiniband and still deliver the same (or better) application scaling performance that our HPC customers were pushing us for.