Apache Spark Split Single Column to Multiple Column | Spark Real Time Use Case | Spark with Scala

#databricks #bigdata #apachespark

Apache Spark Split Single Column to Multiple Column | Spark Real Time Use Case | Spark with Scala

In this video, we will discuss on how to split single column in spark dataframe to multiple column using Spark. We will also discuss on the best approach to follow and finally understand why not to use split() or reason to avoid the split() function for this Scenario.

Blog link for code Snippets:
---------provided soon------------

Blog link to learn more on Spark:

Linkedin profile:

FB page:

[saleman]

Thanks.. need more of your helpful scenarios .

[Gokulpriya]

Hello Azar, please share the video link on how to implement the same in Pyspark

[michaelghaly]

I tried from_csv in python but the df_is not solved


option = {'sep': ', '}

df_split = df_is.select('*', pyspark.sql.functions.from_csv(df_is['name'], "firstname STRING, middlename STRING, lastname STRING",

df_split.printSchema()

df_multiple = df_split.select('*', 'value_parsed.*').drop('value_parsed')

df_is.show()
df_multiple.show()

[riyazalimohammad]

Hello Azar! Hope you are doing well.

I'm trying to implement the solution in PySpark. I encountered couple of Challenges, I couldn't implement the list comprehension solution as I cannot simply call the column name without invoking the dataframe.
secondly, While using from_csv, I couldn't pass the schema defined, I had to write a sql like schema in quotes.
Could you please give us an entire solution in PySpark?

[mohitpande]

Hi can we split by comma to multiple columns at once Like what if there are many column as name1, an we want tp split by comma for all the columns at once

[sravankumar]

How can we write pyspark , can you please share that code

[areeshkumar]

can you provide sample data for all problems

[manzoor]

Hi Azhar... Can we connect through call or whatsapp if possible...