Cleansing the CSV data and processing in Pyspark| Scenario based question| Spark Interview Questions

preview_player
Показать описание
Hi Friends,
Sample code is checked into GitHub:

In this video, I have explained the procedure for reading a csv file and processing it using PySpark.
The CSV has multiple lines present for a single Id and has uneven columns ( different number of columns for each row).
Please subscribe to my channel for more interesting learnings.
  1. SparklingFuture
  2. #multiline csv
  3. #split
  4. #explode
  5. #grouping and updating multi column data in a single row
  6. #multiple rows to single row
Рекомендации по теме
Cleaning Data in 0:21:04

Cleaning Data in Excel | Excel Tutorials for Beginners

How to Clean 0:12:39

How to Clean CSV File in Excel

How to clean 0:00:46

How to clean up messy CSV data in 10 seconds! #excel #exceltutorial #exceltips #exceltricks

Clean Excel Data 0:05:52

Clean Excel Data With Python Pandas - Removing Unwanted Characters

Cleansing the CSV 0:11:37

Cleansing the CSV data and processing in Pyspark| Scenario based question| Spark Interview Questions

Opening .CSV Files 0:01:39

Opening .CSV Files with Excel - Quick Tip on Delimited Text Files

How To Import 0:20:38

How To Import & Clean Messy Accounting Data in Excel | Use Power Query to Import SAP Data

Opening CSV Files 0:11:12

Opening CSV Files in Microsoft Excel - Data Cleansing - Business Intelligence with Data Mining

Human Resource Analysis 1:16:12

Human Resource Analysis using Power BI | Live Bootcamp

How to Do 0:23:41

How to Do Data Cleaning (step-by-step tutorial on real-life dataset)

Data Cleaning using 0:48:49

Data Cleaning using Python pandas in Jupyter Notebook - How to clean CSV data in Jupyter Notebook?

How to Clean 0:14:18

How to Clean Up CSV Data in Python for Beginners

Filter Massive CSV 0:06:03

Filter Massive CSV using PowerQuery in Excel

Clean up messy 0:00:48

Clean up messy CSV data in Excel #shorts #excel #csv #data #cleanup

Import CSV Files 0:12:23

Import CSV Files As Pandas DataFrame With Few Data Cleaning Options

How to fix 0:00:21

How to fix messy CSV data in 10 seconds! #excel #exceltips #exceltricks

Data Cleaning in 0:42:16

Data Cleaning in SQL | Google Data Analytics Certificate

How to clean 0:11:49

How to clean and format a csv file?

Data Cleaning and 0:10:24

Data Cleaning and Preprocessing Without Coding | Clean Your CSV Data Using API | RapidApi

Python Pandas Tutorial 0:31:54

Python Pandas Tutorial (Part 9): Cleaning Data - Casting Datatypes and Handling Missing Values

Python CSV files 0:07:59

Python CSV files - with PANDAS

ChatGPT Code Interpreter 0:03:50

ChatGPT Code Interpreter Demo: Cleaning CSV Data & Plotting Graphs

Effortlessly Clean NaN 0:14:12

Effortlessly Clean NaN Values from CSV Files with NumPy | Read CSV file

Data cleaning in 0:14:48

Data cleaning in SPSS

Комментарии
Автор

Your tutorials are simply special Sravana!!

sudippandit
Автор

Superb, everyone can easily understand 👍 👏

sravankumar
Автор

amazing vide. Now i know where i am wrong. thx for the video.

deathseal
Автор

Please do more videos scenario based on pyspark .current project using pyspark we r doing transformations in ADB , adf only FOR data movement only.

sravankumar
Автор

@sparklingFuture
why cant we use pivot and filter data on top of it it will be single liner right?

shahids
Автор

Your videos are awesome with more advance approach but pls upgrade your audio system. Its request.. 🙏

akashbalmiki
Автор

can you please this scenario how to Load CSV file in to JSON with Nested Hierarchy using pyspark in ADB like custid, custname, itemname, quanity this csv when we convert to nested json custid, custname, purchases { itemname : book, quantity : 2} like one customer buy multiple items

sravankumar
Автор

hello...can you please confirm when you first extracted data from CSV where did you mention the column names. how did the column names generate in the show command

rajanib
Автор

How to Merge Spark DataFrame - Complex type if we have two json files json 1 schema and json2 schema is differenr how can we merge using pyspark. can you please explain this scenario.

sravankumar