Distributed SQLite with Litestream and LiteFS

Ben and Aaron discuss replication and backups in SQLite, Litestream and LiteFS, and future mad scientist projects Ben is working on.

Follow Ben:

Follow me:

Chapters:
00:00 - Introduction
02:27 - Creating BoltDB for Shopify
04:45 - Litestream Overview and Technical Details
05:50 - Running Databases Next to Applications
06:48 - Backing Up SQL Databases with Litestream
07:45 - Preventing Data Loss with Continuous Backup
09:03 - WAL Mode and Checkpointing in SQLite
12:12 - How Litestream Works and Design Goals
14:05 - Shadow WAL and Network Latency Handling
18:00 - Data Compression and File Recovery
19:54 - Handling Transactions and Shadow WALs in Practice
22:05 - Litestream Recovery Process and Best Practices
24:02 - Introducing LightFS for Distributed SQLite
28:40 - Using LightFS as a Custom Filesystem
31:25 - Failover and Read Replicas with LightFS
34:15 - Right Forwarding and Application Consistency
37:45 - Vision for Litestream and LightFS
39:00 - LightFS Cloud: A Paid Backup Service
41:10 - SQLite Performance and Simplicity
43:15 - The Growing Popularity of SQLite
45:20 - Use Cases for SQLite in Modern Applications
49:00 - Future Possibilities: Remote SQLite over SSH
52:05 - The SQLite Community and Innovation
54:00 - Where to Follow Ben Johnson

[robrobob]

What a cool guy! So chilled. Thanks for hanging out Ben ;-) This is such a prime example of some of the amazing things people are doing with SQLite.

[KentCDodds-vids]

Ben is the bomb and I loved this chat. I've been using LiteFS for years and it's fantastic. I even made the transition from postgres to SQLite and it was not too bad! Prisma made it less bad than it could've been.

[zuma]

This is such an awesome video. I've been interested in LiteFS, Litestream, and SQlite in general for a while now. Ben Johnson is such a knowledgeable person, and Aaron is also clearly more than well versed enough to make the most out of his knowledge. Thanks.

[Kane]

Prime was dead on when he said Aaron is the nicest man on the internet.

[MundoTecChannel]

first DHH, then Ben Jonson. next: aaron francis with the creator of life

[mahmoudmousa]

Aaron I'm really becoming a fan dude the way you ask genius questions and have no issues clarifying things <3

[sinaahmadpour]

I love technical talks like this. I wish there were more. Great conversation. Thank you Aaron 🤩🙏🏻

[dandogamer]

Currentlly 23 mins and ive learnt so much. I never knew Ben was the guy who wrote all my favourite fly articles ❤

[nimmneun]

15min in ... loving it, great guest 🎉 you're on a roll man 😊

[pookiepats]

Litestream is the reverse of SQLite cloud ☁️ this is pretty awesome

Honestly Litestream seems more appropriate for most use cases unless you just NEEDED multi tenancy from a wholistic dataset perspective

[markripley]

What a great discussion! One of very few I’ve watched where I actually wished I could have been on the call with you guys. More like this please Aaron 🙏

[Cyber_Lanka]

Aaron is unarguably the most underrated tech youtuber. He deserves much more views

[RobertWinkky]

Great content Aaron! You are killing it

[vladogir]

Amazing interview! Please more of those gold nuggets!

[JOHNSMITH-verq]

I feel like we need Dr hipp himself up in here

[Cool_Goose]

Aaron is making me feel old by not knowing what Flex is :D

[razvancomsa]

Is this techology like binlog but on steroids ? It would be amazing to have something like this in mysql :( I wanted to play around with the binlog from what I understand it's possible to do some sort of point in time recovery but it's orders of magnitude harder than in sqlite which is just a file ?

[daviyen]

Some of us spent years trying to forget that adobe flex existed.

[magmaticly]

I just started watching this, and I don't know which man is Aaron and which is Ben.

[Dontcaredidntask-qm]

Are all these modifications to SQLite slowly reinventing SQL DBs? At this point, why would you not just spin up a SQL DB?