filmov
tv
5 Reasons Parquet Files Are Better Than CSV for Data Analyses | PyData Global 2021
Показать описание
5 Reasons Parquet Files Are Better Than CSV for Data Analyses
Speaker: Matthew Powers
Summary
Parquet files are well supported by most languages / libraries, are easier to work with, and typically more performant than CSV files. This talk summarizes the main benefits of Parquet files and shows how they’re faster with benchmarking analyses. You’ll also learn how to convert CSV files to Parquet.
Description
5 reasons Parquet files are better than CSV:
schema - examine how the schema is embedded in the file metadata leveraging PyArrow
file sizes - compare file sizes when identical data is written to CSV and Parquet
columnar file format - examine performance benefits from leveraging column pruning to skip data
predicate pushdown filtering - understand how to query row group metadata with PyArrow and how to skip entire row groups based on column metadata
immutable - why immutable file formats are better
How to convert CSV files to Parquet with Pandas, Dask, and PySpark. Will show how to convert a single file or multiple files in parallel.
When to use CSV files and when to avoid them.
Matthew Powers's Bio
Powers is a tech evangelist at Coiled.
He used Spark / PySpark for 6 years and is now help devs understand when Dask is a better fit.
He's written two books, has a popular blog, and regularly contributes to open source codebases.
In a past life, he passed all three CFA exams and worked in finance.
PyData Global 2021
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Speaker: Matthew Powers
Summary
Parquet files are well supported by most languages / libraries, are easier to work with, and typically more performant than CSV files. This talk summarizes the main benefits of Parquet files and shows how they’re faster with benchmarking analyses. You’ll also learn how to convert CSV files to Parquet.
Description
5 reasons Parquet files are better than CSV:
schema - examine how the schema is embedded in the file metadata leveraging PyArrow
file sizes - compare file sizes when identical data is written to CSV and Parquet
columnar file format - examine performance benefits from leveraging column pruning to skip data
predicate pushdown filtering - understand how to query row group metadata with PyArrow and how to skip entire row groups based on column metadata
immutable - why immutable file formats are better
How to convert CSV files to Parquet with Pandas, Dask, and PySpark. Will show how to convert a single file or multiple files in parallel.
When to use CSV files and when to avoid them.
Matthew Powers's Bio
Powers is a tech evangelist at Coiled.
He used Spark / PySpark for 6 years and is now help devs understand when Dask is a better fit.
He's written two books, has a popular blog, and regularly contributes to open source codebases.
In a past life, he passed all three CFA exams and worked in finance.
PyData Global 2021
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
0:29:29
5 Reasons Parquet Files Are Better Than CSV for Data Analyses | PyData Global 2021
0:01:01
✅ Why I use Parquet File as Data Scientist? #datascience #dataanalytics
0:11:28
Parquet File Format - Explained to a 5 Year Old!
0:08:02
What is Apache Parquet file?
0:05:16
An introduction to Apache Parquet
0:02:18
What is Parquet? Simply Explained
0:00:44
Advantages of PARQUET FILE FORMAT in Apache Spark | Data Engineer Interview Questions #interview
0:11:33
What Why and How of Parquet Files
0:00:50
Why PARQUET file formats? #Shorts
0:06:27
What is Parquet File Format?
0:00:54
The Future of Data Storage: Parquet Files and Object Stores
0:07:32
Apache Parquet Explained in 5 minutes
0:40:46
The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)
0:14:40
Row Format vs Column Format | Why Parquet is better than Avro | Why Columnar formats are preferred
0:01:00
The key reasons why Parquet files, particularly NanoParquet, are better than CSV #gis
0:00:56
What is a delta parquet file? #deltaparquetfile #apachedeltaio #data #coding
0:24:38
Apache Parquet: Parquet file internals and inspecting Parquet file structure
0:04:53
What is Apache Parquet files
0:00:57
Cut your file size in half with Parquet
0:00:22
How to Parquet File in Pandas Python
0:12:54
This INCREDIBLE trick will speed up your data processes.
0:07:30
Parquet a Columnar file Storage format using pandas | By Viswateja
0:09:08
Feeding a Tensorflow Model from a Parquet File - Florian Dejax
0:04:21
Reading Parquet Files in Python
Комментарии