Converting a single column to multiple rows | Apache PySpark | Realtime Scenario

Hi Friends,

In this video, I have explained the Pyspark code for splitting a single column delimited value to multiple rows and columns.


Please subscribe to my channel for more interesting learnings.

[gurumoorthysivakolunthu]

This is ultimate,
Best intermediate / high level of problem statement... Very Informative... Your explanation is very easy to understand...
Have few doubts, Mam...
1. What does -- " . *? " -- Mean.. ?
2. In f.split -- "col" -- is the exploded column...?
3. Why *map and not f.map?
4. In *map statement -- what does -- f.col("value ") -- Mean...?
5. In *map -- how lambda i -- is number -- which is passed as index (i)...
...
Apologies for many questions, Mam... This is because you have covered many concepts in this video...
...
Thank you, Mam... Thanks a lot for your ultimate effort in knowledge sharing...
...

[tusharhatwar]

Thanks Mam for video.

Waiting for more videos

[SriRamTrading]

Its long method, we can obly use split+ explode to solve this if any one wants the answer will post here

[udaybhanu]

Hi sravana I am learning Pyspark please give some ideas... I want full course...

[manzoor]

That python script will be run on AWS cli.

[sureshtm]

Hi Sravana, your videos are very informative, I have one question
val list1= List(List(1, 2), List(2, 3), List(4))
O/P is (1, 1)(2, 2), (4, 0)
it will be very helpful for me if you can send me how to produce this output

[indra]

do you provide online training/job support?

[manzoor]

Hi sravana.. very nice step by step explanation you give.. which definately helps alot... I have one requirement can you please me out in that... My requirement is i need to write a python script using boto3 which will download the files from AWS s3 prod to my local...and upload the same to AWS s3 dev from my local... And del the same so that iteratively it can do the process.

[pusarlaaishwarya]

hi sravana do you teach with scala also?

[chsreenivasrao]

very informative. Can you please cover real time scenarios related to Py spark query performance if possible

[NaveenKumar-kbfm]

Thank you very much for the clear explanation, do you take personal online classes to teach scala spark .

[rajashekhar]

Hello Mam do teach online classes as well?

[chaitue]

Hello madam, can you please share what if the file is flat text file and there is no delimiter. And of course no headers as well.
Ex. 44: 15PMFBUKAT GN00File review: Last correspondence in file

[sravankumar]

Hi sravana I have one doubt, in our current project they implemented scd type 1 and 2 using pyspark . I am not able to understand, we have configuration scripts we just mentioned in the insert scripts scd type 1 or 2 . Backend automatically it is running. Can you please explain scd type 1 and scd type 2 . Can you please reply this message