CI/CD for Modern Data Engineering | End to End Data Engineering Project

In this video, we delve deep into the world of DevOps, focusing on Continuous Integration (CI) and Continuous Deployment (CD) within the realm of modern data engineering.

Timestamps:
0:00 Introduction
2:00 System architecture
4:00 Prerequisites and installations
9:21 Azure Infrastructure as Code Automation with Terraform
13:50 Storage account module with Terraform
32:52 Automating Azure Data Factory with Terraform
1:04:30 Testing and Validation of Results
1:09:14 Outro

Resources:

Tags:
CI/CD, Data Engineering, Azure, Terraform, DevOps, Cloud Computing, Continuous Integration, Continuous Deployment, Azure DevOps, Infrastructure as Code, Automation,Cloud Services, Azure Cloud, Terraform Automation, Data Pipeline

Hashtags:
#CICD #DataEngineering #Azure #Terraform #DevOps #CloudComputing #ContinuousIntegration #ContinuousDeployment #AzureDevOps #InfrastructureAsCode #Automation #CloudServices #AzureCloud #TerraformAutomation #DataPipeline #DevSecOps

[CodeWithYu]

Your Likes and Subscriptions are what keeps this channel alive! Don't forget to leave a LIKE and SUBSCRIBE to more Amazing Contents!

[hritikapal]

Woah! Really appreciated the frequency of the videos, haven't watched the full video yet but quality of the content you're delivering that's really phenomenal thankyou man!

[MatN-clbc]

You are such a blessing! Thank you Yusuf!

[gortsara]

thank you for this video! looks very interesting, will dive deep soon :)

[wiss]

Thanks for sharing, waiting for dbt and snowflake

[naren]

Please make video Entire Azure Devops CI-CD Dev, UAT, Prod Pipelines for Azure Data Factory pipeline involving various inside of it (ForEach, If, Filter, GroupBy, Aggregations by Dataflows) by Console or Azure Databricks, Load to Synapse, Next Analysis Dashboard at Looker

[NatureAndMyself]

thanks for the wonderful video Can we get video aws with terraform

[dreez]

Thank you for sharing all this knowledge this was very helpful, can you give me some tips on how to create a stable local environment to build side projects without getting to much dependencies errors, I believe the hardest part in data engineering is to succesfully setup an environment . Thanks

[jasonkhaihoang]

This is really cool. However, for Data Factory Linked Services, Datasets and Pipelines, I would create those via the UI (with Git integration) and not via Terraform as those components are more related to ETL logics, rather than infra.

Thanks for a very good content :)

[maazmajid]

Hey my friend, thx a lot for sharing amazing videos, waiting for snowflake and dbt, and I recommend to have live classes.
Thx in advance

[____prajwal____]

What is the best resource to learn about azure services from scratch to advanced level?

[imehmood]

Which terminal are you using? Can you pls share procedure to download it?

[deepaksaharan-ds]

Hi yusuf, can I see the content that I will be able to get access of, if I join the channel?

[voxdiary]

I am confused about the CD part. where do we do the tests to validate our sources. with the validate command?

[toyncandy]

Hey Yusuf thanks for this knowledgeable content....I hv been learning from ur videos from past 2 months... Just want to know that can a DE transition later into full time DEVOPS or MLOPS engineer? Bcz I love the deployment side. Thanks in advance.

[punamadhikari]

hey couldn't login to Azure account it says no subscription what to do?

[prabhatgupta]

Bring on azure end to end till cicd.
Thanks

[bhargaviakkineni]

We didn't getting how to make the docker setup, vscode code setup, kahka setup for practicing data engineering projects though you shared some download links but I didn't getting please make a video how to make all system setups for doing data engineering projects requesting you please