Apache Spark 2 - Spark SQL – Basic Transformations such as filtering, aggregations, joins etc

As part of this session we will see basic transformations we can perform on top of Data Frames such as filtering, aggregations, joins etc using SQL. We will build end to end application by taking a simple problem statement.

itversiry LMS course(CCA 175 Spark and Hadoop Developer – Python – 93 Days Lab):

* Spark SQL – Overview
* Problem Statement – Get daily product revenue
* Relationship with Hive
* Projecting data using select
* Filtering data using where
* Joining Data Sets
* Grouping data and performing aggregations
* Sorting data
* Development Life Cycle

On our YouTube channel we conduct live sessions regularly. Please do subscribe to get notifications for our live sessions by clicking here.

For quick itversity updates, subscribe to our newsletter or follow us on social platforms.

#Python #PySpark #Spark2 #itversity #Spark #DataEngineering

Join this channel to get access to perks:

[raghavagrawal]

how to do a full setup of Hadoop and integrate it with hive and spark.

[SpiritOfIndiaaa]

Thanks a lot sir . nice tutorial, sir if I have to load data from oracle database incrementally depends on the date . where and how should be my filter logic . I am ingesting that data in hdfs parquet files as I need to further process it ...how to handle those use-case where should be filtering should be done ....for better performance