The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)

The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads. As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is 'many small files', and will discuss the open-source Delta Lake format in relation to this and Parquet in general. This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.

About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.

Connect with us:

[prabhumaganur]

The best representation of Parquet file structure!! Simply Awesome!!

[robinjamwal]

Great talk, Great Teach, Excellent Tutor! One of the best presentation I have ever viewed and listened.

[manishsingh]

This content explained most of the thing and It is really amazing .

[SunilBuge]

Great overview to address performance issues with storage layer design 👍

[lhok]

Best Parquet File presentation I watch

[YinghuaShen-kwys]

Great, this makes me know more about Parquet. Thanks for the pre!

[kehaarable]

Awesome video - not too much extraneous or labored points. Thank you!

[mallikarjunyadav]

Awesome video with great content and explanation. Very very useful.

[AM-izgk]

Impressive presentation well structured explanations.

[flaviofukabori]

Amazing. All concepts really well explained.

[AmitParopkari]

Finally understood what parquet format, thanks.
So I have one small doubt, does it mean that footer metadata is nothing but schema details, like underlying table details? Like way to mention table name, column names? etc.
I'll also dig from my side, but just meanwhile ....

[tadastadux]

@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?

[higiniofuentes]

Seems the time and i/o needed before use the data in doing the sort first is not considered?

[salookie]

interesting how Parquet (columnar analytical focused) data can be optimized using dictionary-based compression and partitioning

[chrisjfox]

I haven't watched this yet but for the sake of prioritizing when I do, how well does this topic apply to platforms and systems other than Spark?

[rum]

anyone who says parquet is columnar format is having just bookish knowledge

[spacedustpi]

Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.

[jeremygiaco]

How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.

[thevijayraj]

Bucketing explanation was not great. Rest was fantabulous.