Date Functions in Spark SQL with Scala | Spark coding interview questions

Показать описание

Hi Friends,
Today we will discuss about all the date functions available in Spark SQL with Scala sample code.
Sample Input file can be downloaded from the below link:

Please subscribe to my channel and provide your comments.

Рекомендации по теме

Комментарии

nice video.. could u please share how to read the same data with dateType

ramanareddy

how to assign date to the variable eg val a = 22-03-22

poonkodigopal

@Sravana lakshmi pisupati, thanks for great video.

this is one of the interview question(either python or scala)

I have csv file having below date columns and each column has different date format as below .

hiredate = dd-mmm-yyyy
report_date=dd/mm/yy
end_date = mmm/yyyy

input data

empid, ename, hiredate, report_date, end_date
1, sreenu, 17-oct-2020, 18/10/20, Jun-2021

if date is MMM-YYYY then display last day of the month else convert YYYY-MM-DD format.

output df

empid, ename, hiredate, report_date, end_date
1, sreenu, 2020-10-17, 2020-10-18, 2021-06-30
-

I want create a method/function to check date column date pattern and apply above logic to convert date as per logic.
in future if any new date column added with above format it should convert as per logic .

can you please help me on this. if possible create video on this.

kottakaburlu

Hii sravana need your help to get a column as age from the column existing as date of birth(dob)in spark sql

bhavanichilukamari

Hi, Can you please share the dataset for this example

nareshkumar

Date Functions in Spark SQL with Scala | Spark coding interview questions

Date Functions in Spark SQL with Scala | Spark coding interview questions

50. Date functions in PySpark | current_date(), to_date(), date_format() functions #pspark #spark

Data Engineering Spark SQL - Spark SQL Functions - Date Manipulation Functions

PySpark Examples - How to handle Date and Time in spark - Spark SQL

Spark SQL Date and TimeStamp - Spark Functions | Apache Spark Tutorial | Using PySpark

08 Spark SQL - Functions - Manipulating Dates

Spark SQL - Pre-defined Functions - Date Manipulation Functions

36. Date Functions In PySpark | Current_date() | Date_format() | To_date()

Big Data Mock Interview

51. datediff(), months_between(), add_months(), date_add(), month(), year() functions in PySpark

PySpark - How to Convert Date to TIMESTAMP Apache Spark

Apache Spark | Spark Scenario on Date and Time Functions | Using PySpark

SparkSQL Date function in Databricks #SparkSQL #databricks

Spark SQL - Pre-defined Functions - Data Type Conversion

Pyspark Interview Questions -- Pyspark SQL Date and Timestamp Functions

52. Timestamp Functions in PySpark | Azure Databricks #spark #pyspark #azuresynapse #databricks

PySpark - Date formatting, convert string to Timestamp Date Object - Part 5.2

Pyspark Scenarios 16: Convert pyspark string to date format issue dd-mm-yy old format #pyspark

Spark SQL - Pre-defined Functions - Using CASE and WHEN

Working with Unix Timestamp (EPOCH) in Spark SQL with Scala

Comprehensive View on Date-time APIs of Apache Spark 3.0

Spark SQL - Pre-defined Functions - Overview of Functions

Data cleansing importance in Pyspark | Multiple date format, clean special characters in header

Apache Spark Python - Processing Column Data - Using Date and Time Trunc Functions