Daniel Chen: Cleaning and Tidying Data in Pandas | PyData DC 2018

PyData DC 2018

Most of your time is going to involve processing/cleaning/munging data. How do you know your data is clean? Sometimes you know what you need beforehand, but other times you don't. We'll cover the basics of looking at your data and getting started with the Pandas Python library, and then focus on how to "tidy" and reshape data. We'll finish with applying customized processing functions on our data.
===

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

0:00 Introduction
0:18 Setup: Github Repo, Jupyter Setup
7:43 Dataset / Dataframe At A Glance
13:12 Filtering, Slicing a Dataset / Dataframe
13:25 Extract a Single Column: df['col_name']
14:12 Dataframe vs Series
14:41 Extract N Columns: df[['col1_name', 'col2_name']]
20:38 Extract Multiple Rows and Columns
22:00 Extract Rows using Boolean Subsetting
23:24 Extract Rows using Multiple Boolean Subsetting
24:55 Cleaning a Dataset / Dataframe
25:38 General Issues according to a "Tidy Data" Research Paper
29:45 Issue 1: Column Headers are Values and not Variables Names
30:19 Load Pew Dataset
36:59 Load Billboard Dataset
42:00 Issue 2: Multiple Variables are Stored in 1 Column
43:06 Load Ebola Dataset
47:14 Split Column using String Manipulation through Accessors
53:13 Add Column to Dataframe
56:10 Issue 3: Variables Stored in Rows And Columns
56:25 Load Weather Dataset
1:1:00 Transform Rows into Columns
1:7:42 Issue 4: Multiple Types of Observational Unit in Same Table (i.e De-nomalized Table)
1:9:43 Extract Type Observational Unit in new Dataframe, Drop Duplicates
1:11:30 Create "key" for extracted observational unit dataframe
1:13:22 Merge / Join dataframe on common columns
1:16:25 Randomly Sample a dataframe
1:17:15 Note on Memory Consumption between all 3 dataframes
01:18:25 Summary from "Tidy Data" Research Paper
01:20:06 Q&A
01:21:21 Q&A 1: Simulating R's Chaining in Python
01:24:49 Q&A 2: Best Practices on Braquet Notation vs Chaining

[mohitupadhayay]

Time stamps :

32: 57 : Melt function for Transposing columns and Rows.
42:17 : Handling multiple variables
48 :11 : Split Function
56: 20 : Variables stored in both Rows-Columns.
1:01:10 : Pandas Pivot table
1:06:11 : Reset_index() function
1:10:23 : Drop duplicates function

[ediray]

This is the single most helpful python pandas tutorial that I've ever seen. Thank you Daniel Chen. Great job!

[tabaicanking]

One of the best data cleanih and tidying tutorial on YouTube

[sergioluis]

Simple and easy to follow. Completely new Python student and this was such a great introduction, eager to continue learning! Thanks a lot Daniel

[blessingogbeh]

I'm just 10 mins in and I already love you. Thank you for this presentation

[yassaryelurkar]

Great video. Underrated, isn't recommended by youtube. Btw thanks for the video.

[kennethstephani]

This is a terrific video, Daniel. Thank you very much!!

[matosleni]

Excellent lecture! Thanks for letting it public!

[marcmarc]

Those timestamps were well made and very helpful - thank you!

[gagandhaliwal]

Well explained and easy to understand . I appreciate your help so much. Many thanks

[emprze]

Great video, that gets to the point and explains complex concepts so elegantly.

[vinuvish]

Thanks for the video. I started with pandas and got confused with the brackets, it is clear now.

[MouradBENKADOUR]

Excellent Tutorial about pandas and Data Cleansing and Data preparation

[iliasdimadis]

Great presentation! Thank you very much for this helpfull and understandable-easy to follow Tutorial.

[lucasmunkombwe]

Great tutorial for beginners like me. Thank you

[michaelhaag]

Very nice talk, I finally got it. Seems like a nice dude also

[oyinkanchekwas]

Hello, please how do we get the datasets you used in the video? Could you share a link we could download from? Thanks.

[asankacool]

Awesome explanation and super content @Daniel !

[kaluemeka]

Very excellent and explanatory in detail

[ajiboroibrahim]

Thanks for the timestamps but
@ 16:26 it's df.loc not df.iloc