Reading Parquet Files in Python

This video is a step by step guide on how to read parquet files in python. Leveraging the pandas library, we can read in data into python without needing pyspark or hadoop cluster. This walkthrough discusses how to install the prerequisites you will need in python as well.

#python

[industryrule-]

Bang on. Thanks for even including the error portion for installing pyarrow. Helpful.

[jozzalex]

You save my life!! Thx for the tutorial!!!

[lockwood]

any tips on the error "binary file expected, got text file" ? Second dataset in a row with this error.

[itsevennow]

Very helpful tutorial. Newbie question - I am able to load my parquet file in the notebook. It has 130 columns. But it shows only 20 columns. How can I see all the columns? even if it is for at least 1 or 2 rows is fine.

[mihaelacostea]

What to do when you have a parquet file somewhere on someone else's cloud, is it possible to feed it to Pandas without saving it locally? I am not seing a way to save it locally. it's a coding challenge that simply gives you the link to the cloud location of the parquet data.

[KhalilYasser]

Thanks a lot. I encountered that Jupyter kernel is dead and when restaring the kernel and trying again, I got the same problem. I even tried to put the code in .py file and run it from the terninal but I got nothing printed `print(df.head())`

[gauravanand]

How to read a list of parquet files and read it as a single dataframe?

[hiyoungsun]

Thanks a lot! This video helps a lot! Could you also let us know how to convert the parquet file to .csv file in Python?

[MetallicSiren]

@DataEng Uncomplicated I'm getting a NameError for parquet_file, but it has been defined as shown in the video. Please help, thanks

[akshatahabbu]

Is there a way to load parquet file into oracle DB directly using python scripts?

[neelrama]

I get the error OSError: Passed non-file path:. Have you had this ?

[diy__diy]

FileNotFoundError: [Errno 2] No such file or directory .... my directory path and file name are correct

[enzopablofranciscocarratal]

I get the error: name 'pd' is not defined any advice?

[nishaddhamne]

Can you let me know commands to edit parquet metadata information

[shubhammural]

how to read parquet file from azure blob storage?

[zmtnx]

man why can't they just use a zip file...