DP-203: 25 - Selective CI/CD for Azure Data Factory with ADF Tools (part 2)

Hey data engineers! Enough talking how selective Azure Data Factory CI/CD pipelines might look like. It's the high time to implement them using ADF Tools by Kamil Nowinski.

Join me in the 25th episode of my free DP-203 course where I build selective ADF CI/CD pipelines from scratch.

▬▬▬▬▬▬ IMPORTANT LINKS ▬▬▬▬▬▬

▬▬▬▬▬▬ MEMBERSHIP ▬▬▬▬▬▬
Join this channel to get access to perks:

▬▬▬▬▬▬ CHAPTERS ▬▬▬▬▬▬
00:00 Introduction
00:24 Current environment
01:35 Guide
08:43 Configuration per environment
20:50 Service connections
22:10 Environments
23:01 Variable groups
24:10 Pipeline deploying everything
34:08 Create selective pipeline
47:56 Summary

[oscarestorach]

Your explanation is nice, simple and brilliant for me the best

[fekasng]

Thank you very much. Waiting patiently for databricks. Keep it up sir.

[MotaRota]

Hi Piotr, a big thank you for the great content! I just wanted to ask you if this tool is available for Synapse.

[Ddimmen]

Great video! When setting up the Environments, you mentioned going back to reference another video. In which video do you cover Environment setup?

[oscarestorach]

When you fill in a DEV.csv, UIT.csv and PROD.csv files.
Should I fill in all the linked service from the Data factory DEV environment?
And some linked services like Databricks have more parameters, example from
"AzureDatabricks": {
"properties": {
"typeProperties": {
"domain": "=",
"existingClusterId":"=",
"accessToken": {
"secretName": "=} }}}
Should I fill in a DEV.csv, UIT.csv and PROD.csv all tree: typeProperties.domain, and in tree rows in csv file for every Databricks linked service?
Thanks for your time

[ugandarhari]

This is such a great video and nice explanation step by step. Thanks for providing such a wonderful content.
I have one quick question though. You have manually executed the selective pipeline and deployed Team1 code to Dev and Prod. What about main branch? Do we need to create a pull request separately for Team1 and Team2 once all the testing are done by both the teams based on the priority?

[andrejbelak]

Hi Piotr, why would two separate teams worked on the same ADF environment? Would it not be better to create a separata env for both teams? Thanks for the video!