Windowing Functions in Spark SQL Part 1 | Lead and Lag Functions | Windowing Functions Tutorial

Windowing Functions in Spark SQL Part 1 | Lead and Lag Functions | Windowing Functions Tutorial
Hello and welcome back to Hadoop tutorials powered by Acadgild. In this tutorial, you will be able to learn the windowing functions in Spark SQL. Let’s start understanding the internals of lag and lead functions in this lecture.

These are some inbuilt functions which operate on a set of rows and return a single value for each row from the underlying query. This can be very useful to simplify complex queries and break them down into logical components.

To give you a brief idea about these windowing functions in spark, we will be using stock market data.

Let's now understand the input(Dataset) and expected the output of the lag function.

Show the document consisting of sample input and output and explain it

1st column in this dataset is the date, 2nd column is the ticker which is nothing but the stock name, 3rd column is the open value of a stock, 4th column is the closing value of the stock, and the last column is the volume of the stocks on that particular day.

in the sample output, we perform the query to retrieve ticker, date, closing price of a stock on a particular day and the derived column is the closing value of the same stock on the previous day

Watch the complete video followed by execution of the same.

Please subscribe and stay tuned for more such videos.
#sparksql, #windowing, #Hadoop, #bigdata

For more updates on courses and tips follow us on:

[SpiritOfIndiaaa]

thanks, what if it should be done in a streaming app, where previous data is in hdfs path ?

[megharaina]

What if u want the previous value of the derived column ? How do y achieve that lag(derived col, 1)