Databricks VS Code Extension v2: Setup and Feature Demo

Databricks Visual Studio Code Extension v2, the next major release, is now generally available. In this video I walk through the initial setup and the main ways you will run code and deploy resources using this extension. I also provide some key tips to make sure you don't get stuck along the way.

* All thoughts and opinions are my own *

References:

More from Dustin:

CHAPTERS
0:00 Intro
4:43 Install
8:50 Run Python file options
11:18 Setup Virtual Environment
12:15 Using Databricks Connect
15:14 Run Databricks notebook
16:50 Fix bundle sync
19:27 Run IPython notebook
20:50 Databricks Asset Bundle functions
24:28 Summary + Tips
26:40 Outro

[nelsonndegwa]

Nice! Any chance you could demo working with R notebooks on VS code in Databricks?

[ExplainedbyAI-qn]

At 2:35 you mention getting into some new databricks features like 'jump to code/definition' in a different video. Could you add a link to that video?

The option to see where code is defined, especially 'intellisense-like-behaviour' is something I miss a lot, most of all when using the magic %run command to import functions from different notebooks.

[voxdiary]

it would be really nice to be able to do display(df) in .py files and then have a specialized window at the terminal location that displays results. is there a work around?

[ManishJindalmanisism]

hey Dustin, , when the .py already ran using cluster in our workspace, what was requirement of databricks connect? In either case the compute used is of my databricks cluster and not the local laptop

[gardnmi]

Still needs work.

Issues I found so far:
1. The authentication can still clobber your CLI auth causing lots of confusion.

2. The file sync needs a full refresh option. Only way to currently do so is to delete the sync folder in the . databricks folder.

3. Sync needs to be 2 ways. Databricks/Spark connect is still not feature complete so you unfortunately have to use the notebook in some cases.

4. Overwrite job cluster feature installs your python whl onto your all purpose cluster but if you make any changes to the package l, it doesn't remove the old whl and update it with a new whl with your changes causing confusing errors.

[servandotorres]

If I want to leverage libraries like matplotlib I have to covnert to Pandas and once I do that the compute runs in my local. Any workaround that keeps this in my iDE or should i jsut run as workflow?

[indreshsingh]

Still confused on how did I exact populate your Bundle Resources explorer ?