Advancing Spark - Databricks SQL Variables & Dynamic WHERE

Whenever someone comes from a heavy SQL background, they find some of their favourite toys missing from the Spark SQL world - but there have been huge investments from Databricks to catch up here over the past few months. We thought it was time to talk SQL Variables, Parameterised Tables and even the WHERE (*) functionality!

So if you've been sitting in the pyspark wastelands dreaming of the good old days of SQL Variables, this is the video you've been waiting for! In this video, Simon runs through how Spark SQL parameters work, some of the neat tricks you can do using them, and has finally thought of a use case for "any column filtering"!

[mkrichey]

Awesome presentation as always Simon! Great features which will be very useful!

[Juan-PaulHynek]

Brilliantly put together, thanks Simon!

[selimberntsen]

Great video, will really come in handy! Can these features only be used on SQL pools or also on Spark clusters?

[johnnywinter]

oomph... throwing shade at the SQL editor 😵

[piniki]

Is this working in databrick SQL mode?

[alexischicoine]

Where would you use a variable instead of a widget?

[alexischicoine]

Nice name no servers here. Can’t wait to try the serverless for jobs.

[StartDataLate]

Could you point out what exactly the "where 'text' in (T.*) "is called? at 10:55 timeframe. I am running issue with searching in a table that has struct type. and it is saying that the data type mismatch. ... and this search does not work