AWS Glue Studio: Datetime To Date | Custom transform python

Walkthrough with aws glue studio on how to create a datetime field from a date field through a custom transform node with python. This is useful for creating partitions in your data. AWS Athena is then leveraged to query the results in s3.

00:00 Introduction
00:53 Datetime conversion walkthrough in python
04:54 Athena query comparison

#awsgluestudio

[joegenshlea]

I have a date coming in as ISO 8601 timestamp meaning it's like this "2023-01-15'T'10:34:56" and I'd like to convert this to a datetime or timestamp format in the datafram e(or dynamic frame). Can you give me any pointers?

[BeABetterDev]

Cool feature. Are the jobs run on Lambda to transform the data?

[raghuvarprajapati]

Hi, I want to convert a string column where values are basically date in string for exp- "05-10-2021", "10/02/2009", "2023-04-02", "1710044061" etc, so basically different date format in string. And I want to convert this to a unified date column like yyyy-mm-dd format for all format type, how to achieve this using custom transform?

[sheetalm]

Hi, I want to cast a string column to map/array because I'll be using explodeTransform later on that column. Is creating a new column a 'must' after casting its datatype?

Nice tutorial !!, have a video with Spark SQL transformations ?

[ibrahimshaik]

Can we call a custom method inside "custom transformation node" on same column instead of creating new column?

[DineshGowda]

Since we are converting it spark dataframe, does this work on huge data set?

[krishnasanagavarapu]

Can we do the same for transforming to year/month sub folders only ?