ADF - Create an Azure Databricks linked service - Azure Data Factory

Create linked services
In this section, you author a Databricks linked service. This linked service contains the connection information to the Databricks cluster:

Create an Azure Databricks linked service
On the Let's get started page, switch to the Edit tab in the left panel.

Edit the new linked service

Select Connections at the bottom of the window, and then select + New.

Create a new connection

In the New Linked Service window, select Compute -- Azure Databricks, and then select Continue.

Specify a Databricks linked service

In the New Linked Service window, complete the following steps:

For Name, enter AzureDatabricks_LinkedService

Select the appropriate Databricks workspace that you will run your notebook in

For Select cluster, select New job cluster

For Domain/ Region, info should auto-populate

For Access Token, generate it from Azure Databricks workplace. You can find the steps here.

For Cluster version, select 4.2 (with Apache Spark 2.3.1, Scala 2.11)

For Cluster node type, select Standard_D3_v2 under General Purpose (HDD) category for this tutorial.

For Workers, enter 2.

Select Finish

Finish creating the linked service