Rust at speed — building a fast concurrent database

This is a guest lecture I gave at Two Sigma in November 2018 where I discussed the experience of using Rust for building larger, high-performance systems. In it, I cover what makes Rust an attractive option for such projects; Noria, the high-performance research database prototype I've built using Rust; an interesting concurrent data-structure we use in Noria; and how I've found Rust to work in that context.

[bdafeesh]

How you speak is perfectly paced and you hardly stutter/'um'. I'm always trying to improve my speaking abilities and you offer a great example to follow. Keep it up man

[distrologic]

21:36 This is actually one of the best explanations of the rust ownership system I have heard yet.

[plorio]

I used that exact same atomic counter technique to decide when to the other copy of data is safe to write to at work. Felt quite clever when I got it to work. Really cool to see the same thing in the wild.

[aemmelpear]

I absolutely love how you can tell, how much you are excited about the Rust programming language. Always with a big beaming smile when talking about the nice features Rust has.

[nickmaxwellambient]

Jon, just found your channel today. It's rare to find someone who both covers really in-depth topics and does so in an entertaining way. Fabulous job you are doing here.

[yashashav_dk]

This talk is worth it's weight in gold. Thank you kind, Jon!

[dsincl]

Really enjoyed this talk. The epiphany for me when hitting the borrow checker wall was when I realized that it was helping me not penalizing me for the code I've written. This small mental shift made everything all the easier and suddenly Rust became a pure joy to work with. I have to admit I didn't read through the whole Rust book from start to finish before hitting the wall. I usually never do that when learning a new language but you are absolutely right. Reading the book from start to finish will save you a lot of pain and frustration with the language.

[simonray]

I pretty sure he love this language. Full of energy with a great talk

[JohnHAdams-vopk]

Brilliant explanation of ownership. Simple and to the point. Better than any book I've read. Thank you for this, now it's pretty solid in my head.

[yurikhomyakov]

Thank you for an amazing talk. As soon as Rust 1.0 came out I knew something interesting was cooking!

[scriptozavr]

Enjoyed the talk, Jon is indeed a phenomenal speaker!

[rajsahu]

Jon is inspiring in so many ways. The work (which he explains in the presentation) and the way he described every topic! Just awesome. I probably never watched any hour long talk in one go until this one! <3

[samferrer]

Wonderful lecture ... and the switchable rw-lanes is a must takeaway... thank you!

[ankurleonardo]

Every second in this video was worth watching. Awesome DS glimpse.

[sgdfdsgs]

Your talk was fantastic! Your passion and delivery were both amazing, and it really got me excited about Rust. Thank you for sharing your knowledge and enthusiasm with us.

This talk is amazing! You explain things in a very approachable way!

[corv]

Very interesting talk. I'm curious why you couldn't just store an atomic counter of how many readers are active per map instead of doing the whole epoch and ignore it if it's even thing.

[BenjaminCronce]

Super-linear is real, up to a point. Good memory access patterns can cause different threads to share cached data, such that 2 threads is more than 2x faster than 1 and 4 threads is more than 2x faster than 2. But in my limited experience, I never noticed past 4-6 threads, but at least beyond that was essentially perfectly linear.

[raysonlogin]

The 1 map for readers and another one for writer mechanism is like RCU - whenever the writer is ready then the pointer to the data structure is published such that the readers get the updated version.

Brilliant talk, thanks a lot! I really like how you've covered the ownership model, along with how you've implemented the lock free evmap datastructure and why that is safe. Well done.