Spark User-defined functions

Spark lets us define custom SQL functions and use it in our spark SQL. Those custom SQL functions created by user is known as user-defined functions. For example, lower() is a SQL function that we normally use in our SQL queries to lowercase certain column values. If a new such function is created by a user, it is known as user-defined function.

Let’s create a “square” udf that we can use it to square a column value. To create the UDF, I am creating a scala method called “squared”. It takes an integer as a parameter. What this function does is simply returns the squared value of the input.

Now, in the spark SQL let’s write a query. We have a dataframe with column name as “value”. It contains list of integers. Here I am applying the “square” user-defined function on the “value” column. This should return the squared value of this column.

Let’s display the results, using show command. This returns the squared value results as expected.