Turbo charge PySpark df with PyArrow for pandas DataFrame and Parquet files - the code

On a single node machine (eg laptop w/ multiple CPU cores):
Activate all CPU cores/threads with PySpark and apply PyArrow to read pandas df and parquet files.

Caution:
LAZY Evaluation w/ Spark: execution will not start until an action is triggered (concerning all my speed tests - smile).

Spark and data lakes.
Parquet files and Spark.
Load Spark dataframe from parquet file.
Apply PyArrow acceleration to transform Pandas df.
Speed test of Spark (8 CPU threads) compared to Pandas operations.
File format for data lakes: parquet.
SQL tables: beautiful interface.

Accelerate your pandas df performance multiple times.

#code_your_own_AI
#code_in_real_time
#datascience
#computerscience
#spark
#pandasdataframe
#dataframe
#pyspark
#cpu
#databricks
#apachespark