Mastering Spark Unit Testing (Ted Malaska)

Traveling to different companies and building out a number of Spark solutions, I have found that there is a lack of knowledge around how to unit test Spark applications. In this talk we will address that by walking through examples for unit testing, Spark Core, Spark MlLib, Spark GraphX, Spark SQL, and Spark Streaming. We will build and run the unit tests in real time and show additional how to debug Spark as easier as any other Java process. The end goal is to encourage more developers to build unit tests along side their Spark applications to increase velocity of development, increase stability and production quality.

[taylorperkins]

Thank you so much! I have inherited a scala spark streaming code base and have been in the process of learning all that it does. This code base does not have any actual unit testing, and this talk has given me the confidence and encouragement to be able to write unit tests to make it stronger and to better understand what it does. Figuring out how to test spark outside of a production environment has been a big challenge. This has definitely helped a lot

[singhharmeet]

This is probably the best explanation of RDD, DF and streams and how to test them. Thanks for sharing

[recordable]

Has more insight than some paid courses. It was really helpful

[outofbody]

Great video. Can we get something similar but for spark structured streaming?

[sadanandametikala]

can spark unit test run on cluster also ?if yes how to get the test failures report ??

[sukumaar]

Thanks you for informative video, please provide link of this side.