Advancing Spark - Databricks Runtime 9

The newest major runtime release has already leapt out of preview and is now available for use! There's a whole load of new features, from optimised image streaming in Autoloader, new SQL scalar functions and some handy data profiling utilities!

In today's video, Simon takes this opportunity to dive head-first into the new features, walk through the release notes and test out some of the new functionality

[alexischicoine]

I've been using photon for a project it's working well. The DBU increase makes the savings on Databricks a bit small, but you save on the vms and it's also allowed me to do more with the limited number of cores I currently have access to in our spot quota.

[physll]

Your question about the implementation of SQL functions is an interesting one. You can only express these functions in SQL. Therefore, all the planner has to do is expand out your query to incorporate the full set of SQL transformations in your function. That means you get to keep all the nice things about the existing spark functions (like full stage code generation). What will they think of next, eh?