Should Data Engineers learn Scala?

Apache Spark has brought the programming language, Scala, into the forefront. Should data engineers learn this language?

It's not 100% clear-cut and really depends on your career goals. In this video, I dive deeper into some of those tradeoffs.

[sandeepsingavarapu]

Thanks Zach. It will be great if you can do a video on an End to End data engineering project and share some project ideas(at all levels like Basic, Intermediate, Advanced) to build and practice.

[mgtproductions]

Don't assume that using the Spark Scala API is going to have the 'side effect' of turning you into a competent Scala developer - some of the worst candidates I have interviewed have 'learned' Scala this way.

[letechnicaljames]

Good to know and see you create another video. Scala can be helpful for learning PySpark as well as to prep for working for a big tech company. :)

[mgtproductions]

Scala day rates/salaries are typically higher than Python day rates/salaries....that's a compelling reason right there.

[hi_im_julian]

Thanks Zach. Your points really helped me consider how to prioritise some of the learning I need to do for 2022. Content aside, I kind of dig how raw and unedited your video was. Actually your linkedin post brought me here, and I saw you got into working out to help battle your depression. For me, going to the gym is as much mental strength building as it is physical strength building. Maybe consider some other self-care things to add to your routine like skincare or nutrition (if you already don't). Keep it up

[vishnuvardhanreddysyamala]

Thanks for the valuable information Zach. Looking forward to hear more about Pyspark.

[deeplearningexplained]

Man that was a rollercoaster haha! At first I was like "that's it I'm finally bumping Scala in my learning list", then I just deleted that task thanks Zach 👍

[adriananslow]

I liked your succinct summary. Good video, thanks.

[benjamincabalona]

Really great points. I would add FP as a reason. Being exposed to FP (because when you learn scala it’s inevitable that you hear about FP) changes the way you think about programming. Even if you don’t fully adopt FP, IMO knowing abt it helps your career.

[DiegoSanchez-czib]

Thanks for the information, I was starting to learn scala and I have many doubts about learn this programming language but in this video you clear my doubts, greetings from Colombia

[saulcruz]

looking forward to learning more about Scala Zach, keep us posted.

[domtaylor]

I know Scala from a SWE perspective, and like it, so not like I'm biased towards Python (which I actually dislike) but I didn't get your point about testing. I write some PySpark ETL these days and it's perfectly testable.

[clumsyjerk]

Great video Zach, looking forward to similar content from you.

[Fahadkhan-fjsx]

Great Video ! Bring in the casual tone, like you are conversing with a viewer. Assume us as your students ;) . I am sure, its going to be hit channel !

[ingscc]

Nice video. Good tips. Nevertheless, i think any DE should have at least notions of scala, it can makes the difference when getting a job against other candidates. Also, at least in Spain, i'd say scala is way more than 10% jobs in spark, maybe 50%, maybe it depends on your environment.

P.S. Suscribed and looking forward for the next video ;)

[shashankemani]

Such valuable and insightful information. Thanks!

[matthewgin]

very informative thank you. Am I wasting my time learning Scala? Well what else would I be learning? Regexes, sed and awk commands. More python. I think it'd be great if other tools were using Scala as well and it'd make it much more reasonable to learn it. I also found aws glue has better integration with pyspark over spark native.

[rohitpandeyji]

Very informative video,
Could you please share some unit test strategy and design framework for pyspark ETL project

[natebradshaw]

The software developer in me now wants to learn scala, but the career developer in me is saying PySpark.. maybe I'll explore both

[alok]

@Zach, Thanks for such a succinct yet well-expalined video. I think, Scala is also more performant than python and it also teaches Functional programming. Scala is a jewel when it comes to parallel programming because its base is singleton class and immutable data types. What's your thought on this ?