The Harsh Reality of Being a Data Engineer

If you enjoyed this video, check out some of my other top videos.

Top Courses To Become A Data Engineer In 2022

What Is The Modern Data Stack - Intro To Data Infrastructure Part 1

If you would like to learn more about data engineering, then check out Googles GCP certificate

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

Or check out my blog

And if you want to support the channel, then you can become a paid member of my newsletter

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[notjpengineer]

This is so true. I'm an Environmental Engineer who just succeed in the carrier transaction to become a Data Engineer (I've been a Data Engineer for 3 months only haha), and one of the things that I chat about with my friends is that the best thing in the technology world is that there are plenty to learn, but, at the same time, the worst thing it is also that there are plenty to learn. If you try to learn everything at once the anxiety will kick in and ruin your day.

Got keep focus on the consistency, learning 1% every day is enough.

[JohnDoe]

Dealing with raw data is 1000 times better than dealing with business people who have -100 IQ when you are data Scientist/Analyst.

[tomastruchly]

I agree with you only partially concerning the old vs new technologies usage. Yes, you can stay working with Oracle or SQL Server for next X years, but the number of jobs will slowly drop and it will come time to switch. However is very different to switch when you're in your thirties or when in you're fifties. I switched from on premise world to Cloud solutions like Snowflake or AWS this year and even though I am 35 can feel it's not that easy as when I was 25 (I had no family, was ultra-motivated and literally craved for knowledge)

[alexanderpotts]

All the harsh realities you described are exactly why I love this job. I guess I'm starting to see why my old coworkers thought I was nuts when I said I really wanted to go hard into BI and modern data engineering... :) also, I'm glad you mentioned SWEs not caring about data. I had these same opinions on reddit when data contracts came up

[overidon]

I love how there are two number '1' bullet points in the infographic cartoon at 5:15 too funny

[sharu]

I’m slowly starting to relate what you are talking about which is a good thing as I’m gaining so much of valuable information. Thank you so much, really appreciate what you are doing ❤

[antonkostov]

Brilliant as always. In my humble personal beginner's opinion . Due to Hana and cloud migration DE is getting lots of demand. I can say in Bulgaria, the market for DE jobs has grown radically last few months.

[adityagoyal]

Thank you for the video. I am a trainee data engineer and being a obsessive programmer I find it difficult to accept that I won't be a developer anymore but looking at this It seems like IT is much more than just writing codes in java and C++.

[rohit-pr]

Thank you for putting this out there for everyone. The problem that I have and I feel others also face as a DE is that the organization fails to understand the complexity there is with when working with data in general. Its not all green and breezy and a SELECT * does not do all the work for us. Pipelines maintainance and optimizing applications is a tedious work in general and someone(DE obviously) has to be on top of it. Removing DE is simply a recipe for hell and a ton of work for the DE who is going to join when everything falls apart.

[frimatechnology]

thanks so much for this information it was delivered well with so much clearity.

[flaviolanfranco]

This is a great resource for everyone, thanks for sharing it

[zie]

When you said there is a lot of things to learn I almost cried because I spent the last 3 years acquiring a degree in Applied Mathematics and Computer Science which at this point I think is useless because it was too broad to a point where I do not know what to be a data engineer of. Even for entry level jobs the bar is so high you have to know a lot of things. I have an IBM Data Analyst certificate, AWS Cloud Practitioner and a couple of good projects but still can not even find an internship to at least put one foot through the door. At this point it's really stressful. 😢

[bfkgod]

Why would they want to remove the very Engineers that are responsible for fixing the bottleneck? Do they think the bottleneck will just magically disappear? Here's an idea, you could hire people specifically to try and removing Data Engineering bottlenecks! Perhaps you could even call these people... Data Engineers.

[cateclism]

I was in software QA for nearly 15 years, and it was a trainwreck for me. I prefer working with databases, data files, and such. QA really didn't interest me, so I didn't study it like I should. I think data engineering is more to my liking, so I would be a lot more motivated.

[CM_CM_]

I wanted to follow the CMU Intro to Database Systems YouTube series but the projects are written in C+ and I'm someone who's just learning python and can do some easy/medium stratascratch sql lol - I feel like I'm always going to be behind in learning DE compared to everyone else - Like you said is there's always something to learn, but it feels like the baseline for beginners is also always rising. I feel like I know nothing! Thanks for your insights, I really like that SQL poster, I just did a lot of googling haha

[kylestyle]

"Fix all these small problems that someone else produces"... Yep I'll pass

[josepablolunasanchez]

I suspect that the concept of computer science will become a skill instead of an occupation in the future, just like using a typewriter was in a distant past.

[dextermorgan]

Hi Ben, is topic of dimensionality reduction applicable to data engineering / future improvement of data engineering as well? I only know that it is for Machine learning. :o

[jamestoneythethird]

Hey Ben, although the gritty reality of being a DE doesn`t bother me that much, I`m still wondering whether or not it`s possible to find your first REMOTE job right after getting a Udemy/Coursera etc. certificate? Are DE jobs remote-friendly?🤔

[arunrajan]

Is it good to do a snowflake certification first or databricks data engineer associate certification first? Which one will you suggest