What Is Data Engineering - Why Is Data Engineering Important?

What do data engineers do?

Do they build data pipelines?

Do they build data warehouses?

Sure, but why?

Or check out my blog

And if you want to support the channel, then you can become a paid member of my newsletter

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[dAntony]

Thanks for the quick rundown and the channel. I’m currently a mathematician who’s retraining in a Computer Science master’s program and have been looking to specialize in an area of CS. I think data/database systems/data engineering is the way forward for me. I like the idea of being able to provide utility for other people using computer systems.

[JackalBruit]

. . . in under 10[min]
u have reminded me
JUST HOW MUCH
i still have / want to learn

[Unexpectedly_kg]

Every upload makes me confident with the topics im learning of DE. I have a sense of what im doing. Thanks Ben!

[juan.o.p.]

You've definitely improved the video quality, I don't know if it is just the light but it looks really professional 🤩

[gavinkalaher]

Hey Ben, hope you're good! I'm still learning like crazy 6-7 days a week. Every day gets easier in regards to the knowledge and coding practice. Every day, enjoying it more annd feeling pumped from conquering the tough challenges.

[fazerydevs]

thank you Ben
I usually hard to understand your content bcs the speaking pace is a lil bit too fast but seeing this video convince me, ur already corrected it.
great content!

[sanjayplays]

Whoa, loving the new lighting and background! Also great content, as always. Just fyi, your videos have helped me progress so much in my DE career. Thanks so much Ben!

[herrunbekannt]

Good content. The part about having to deal with data curation as a Data Engineer cannot be understated. I never saw godd enough source data that did not require some kind of repair during the ingest phase

[tylerbakeman]

The DAG (directed acyclic graph) visual, is cyclic (not acyclic)

[twistedsno]

I’m naïve to this field, but I have a question. Couldn’t someone in the IT department take on most of what you’re talking about? And with a trained AI, could half of a data engineer’s job be automated? What’s stopping a company from shifting a data engineer’s role to the IT department, with some AI assistance?

[govardhandigumurthi]

Could you please tell what are the certifications that will weigh on freshers resume to get hired as a data engineer? Or what are the things that hiring managers are looking for in a freshers profile to most likey get a call

[preservemind]

Hey, I got your reference form Luke Barousse's video and after watching one, I keep watching your videos for more information about the DE, I am really curious and eagur to learn and because of that I changed my line form Computer engineering to Network security to Data Science and Now I am just looking and learning new stuff from your channel. currently working on Pysparks and if you can suggest me better resources for it.
Keep it up, and appreciate your work.
Thank you.
I feel like writing a mail to you🤣🤣

[malharyadav]

what are the base fundamentals to expertise to become a good data engineer like tools and languages may vary and change but some base concepts haven't changed for years?

[BelkisQuirogaE]

Is that the same as being a statistician?

[IamMQaisar]

0:33 DAG is represented as a cyclic figure ???

[dmitriyobidin]

How much time you spend on average doing k8s/docker/system administration?

[thegreatplanet-irlp]

Data engineering or DevOps needs some advice for the career.

[ChristianRRL]

Thank you for the videos Ben. I’ve already started reading through The Fundamentals of Data Engineering and it’s a phenomenal book so far!

One unrelated question though, I’m personally very interested in networking more in this field and I’m wondering if there are any professional or technical “data conferences” you might recommend?

[VersoOculto]

Could you make a tutorial with a project with airflow??? NICE CONTENT! thank you!!!

[claykc]

Can you explain a very basic example of an ETL moving from SQL Server to S3 to Redshift? Or a different combination of technologies. I just want to see where SQL fits in with which tool, same for python. Everyone talks about needing these languages but doesn't explain how to actually use them. I have experience using the languages but not sure how to build a production ETL with them.