Live-coding an asynchronous Rust crate for short EC2 jobs (part 3)

This video covers converting the previously entirely synchronous tsunami crate we've built thus far into one that uses futures/tokio to multiplex the setup of many machines onto a single thread. In the first part of the video, we replace all the innards of the library with asynchronous parts (including getting rid of rayon) while maintaining a (mostly) synchronous external API. In the second part, we start rewriting the library so that it presents an asynchronous interface to users, which would allow tsunami to be used in a larger asynchronous system.

Note that my internet connection cut out at around 2:56:42, so there's a somewhat jarring jump there.

Some useful chunking points:

- First: recap + discussion of futures-based APIs.
- Setting up infrastructure for using async rusoto: 11:22
- Multiplexing machine setup (and rm rayon!): 26:22
- Using async_ssh for multiplexed setup: 42:07
- Fighting the borrowchecker: 1:44:01
- Fought borrowchk and won. Cleaning up the futures API: 2:26:33
- Retrying operations with futures: 2:35:44
- Making the whole thing asynchronous: 2:50:09
- Cancelling requests on a timeout: 3:30:10

From this point forward, there's *a lot* of fighting with the compiler, and basically iterating through lots and lots of similar errors. You *may* not find this particularly interesting, and may at least want to watch at 2x speed. You've been warned. Brief ending at 5:07:01.

- Threading variables through *everywhere*: 3:54:01
- Making all the future error types match up: 4:09:54
- Oh no... borrowck + long futures... 4:36:05

[VictorGavrish]

The resulting code looks really messy. You might understand what's going on, having written it, but it looks completely opaque from the outside. Is there any way to get around that? (I believe async/await will fix some of this, but that's a long way off afaik.) I feel like once you've found the incantation that makes the typechecker and borrowchecker happy, you won't want to touch it ever again...

Edit: What I'm thinking is, there should be a way to trade execution speed for clarity here. I don't believe this code should be CPU-limited in any way (the majority of the time we'll be waiting for AWS to start up instances in any case), so resource usage should not be a concern. So just allocate most things in Arc<RwLock<_>> to get around all borrowchecker errors at once maybe?

[glebabroskin]

Thanks for interesting video! Small vim tip: instead of doing "mechanical conversion" that means writing core.run(...) every time you could record a macro and complete every edit with just two clicks

[xoomayose]

This is pretty cool :)

Have you considered to use CloudFormation to create/destroy all resources?