Azure Data Factory | Continuous Integration & Deployment with Azure DevOps

Building CI/CD process for Azure Data Factory is not quite straightforward. Furthermore, there are a few different methods of doing that. Before we begin doing that, we must set up GIT repository associated with ADF, understand what's the differences between save and publish, and why new branch appears.
The basic knowledge of ARM templates also would be beneficial.
During this video, I will explain one of the methods recommended by Microsoft.

00:00 Intro
01:05 Existing ADF review
02:17 Publishing ADF changes
03:14 Create Build Pipeline (master branch)
05:29 Create Release Pipeline
06:47 Task: Create RG & instance of ADF
11:36 Task: Deploy ARM Template
17:10 Connection to Storage Account
18:40 Prep files for demo
20:13 adf_publish branch
21:38 Change & re-deploy
24:23 Outro

PS. Don't worry about my security. I have already changed all the keys you could have seen in the video. :)

*** Hungry more? Let's connect. ***

#devops #adf #cicd #git #azuredatafactory #cli

[clapton]

very useful, thank you for the walk through.

[Gilthonion]

Fantastic explanation, Kamil! I'm currently trying to implement CI on an ADF with a more complex structure (including a shared IntegrationRuntime from another factory and a linked service to a Databricks workspace) and multiple parameters/variables, and I hope your next video will give me some inspiration on how to create a solution.

[ArneRossmann]

Small Update: The command for creating a new ADF V2 is Set-AzDataFactoryV2. So for your script: Set-AzDataFactoryV2 -ResourceGroupName "$(ResourceGroupName)" -Name "$(DataFactoryName)" -Location "$(Location)" -Force .

[vibhavarivyas]

Very useful .. thanks for sharing .. how do you put check to ensure that you are not deleting any tables / other objects ? Assuming that code review process is not tight..

[mn]

Hi Kamil, Thanks for sharing the usefull information. I have a question If you can help with. We have a shared integration runtime that we are sharing with two data factories Dev and Prod. How can i deploy the release while using the same IR for both data factories. In the release pipeline while deploying to produciton i need to use powershell to first grant access to IR to the prod data factory and then deploy it. How i can use the powershell in this scenario. Thanks in advance

[kapa]

I'm sorry, but the idea of ADF branching was not clearly described for me. What is the purpose of master branch if deployment is done from adf_branch. Why after publishing the adf branch there are 3 commits behind and 2 ahead if we were publishing FROM master (2:50)? Why ADF CI build has to be made based on adf_publish and not on master?

[NicolasEggimann]

Actually the publish step isnt usable with the Workload Identity Federation. Is it planed, to adapt it as well?
I receive the following errror:

Unsupported authentication scheme 'WorkloadIdentityFederation' for Azure endpoint.


Thanks.

[bharanidharanm]

Hi Kamil, i have a data lake storage which is authenticated thru service principal. i am struggling to assign the client secret in config-uat.csv to override parameter of linked services. could you pls help. thank you.

[klinok]

Hi Kamil,
what about Self-hosted Integration Runtimes?
IR does not work in the new Data Factory after deploying.
What is your idea how to fix it?

[SantoshSingh-kibx]

sir, I have created a feature branch in ADF but forget to delete it. When again I am developing new changes with this branch and pull into master I am getting merge conflict issue and also don't know how to delete my unused feature branch. Please help