What Is A Data Pipeline - Data Engineering 101 (FT. Alexey from @DataTalksClub )

Data pipeline

A term that gets thrown around a lot in the data space.

Does it involve streaming, batch, Ipaas or all of the above?

Guests in this video include

If you enjoyed this video, check out some of my other top videos.

Top Courses To Become A Data Engineer In 2022

What Is The Modern Data Stack - Intro To Data Infrastructure Part 1

If you would like to learn more about data engineering, then check out Googles GCP certificate

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

Or check out my blog

And if you want to support the channel, then you can become a paid member of my newsletter

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[NaveenKumar-cnpc]

Wow As an Informatica ETL developer myself planning to switch to Data Engineering I was just trying to understand this concept of Data pipeline, this is as much more simplified explaination although would request you a video on what kind of pipeline would you suggest based on different business scenario that you mentioned as we have some ETL or I say ELT tool build native to cloud such as Matillion, IICS or manually coding the entire thing.

[sngx]

Great content, simple enough to understand and reflect on as to learn how the same knowledge can be interpreted by different people.

[christiansanchez]

Good job on mentioning that there are too many tools. I’m a Business Systems Analyst right now who is trying to learn data engineering and there seems to be too much to learn.

[alphar]

I am here lookong to switch from analytics and digital lead to data engineer as i see myself doing it alongside software engineering (although they might sound different). So as i lean more towards SQL rather than Python. I got trained on SQL and SSIS and i never knew that was part of data engineering. In SSIS, you really learn a lot about the ETL process. This video really taught me to revisit the SSIS as there is a learn in the system and plus it is MS product.

[Cybersader]

I do security log management using Cribl and everything youve said is everythinf ive been thinking but in a different language 😂 Great to hear from someone else. We dont have data engineers where I work. Just me

[arahso]

A lot of this content is geared for beginners and for folks trying to break into the role. What about advanced topics or in-depth technical debriefs?

[bryanneremita]

Being someone who wants to migrate from a data analytics role into a data engineering role, what tools would you guys reccommend I start out with ? From the ETL/ELT operation to data warehousing and maybe post that some work on cloud tech. Any help would be appreciated.

[Anant_Maheshwari]

apart from being excellent in sql what else one should know if a datawarehouse guy wants to become a data guy, not the seattle one but an engineer one.

[bettatheexplorer]

Hi,

Currently doing etl pipelines with airflow on gcp. Do I need to study about starflakes, snowflakes schema?

[chieeyeoh]

Ya, I think I look like a data plumber than a data engineer