Nimble, A New Columnar File Format - Yoav Helfman, Meta

In this talk, we will introduce Nimble, a new columnar file format developed by Meta, formerly known as Alpha. We will discuss the reasons for designing a new file format and provide a glimpse into its inner workings.

[MrNoipe]

The q&a portion was unfortunately cut off.

[DannoHung]

Seems like bunch of micro-optimizations aimed at some fairly narrow use cases then extrapolated to mitigate any other issues for really tuned in users. The fundamental problem with the explosion of encodings and sub-encodings and compression optimization is that you have to be an expert in both the data you’re writing and the format to have a hope of getting close to using it effectively.

And yeah, presumably that’s why fellas Yotam have jobs, but it’d be nice if these things also came with sensible tuning guides.

[ArisFilms]

working on a similar concept for a video container

[jonny__b]

"columnar" data, isn't this just "planar" data?