The Downfall Of The Data Engineer - The Challenges You Will Face Being A Data Engineer

While the title of this post is sensationalistic and the content quite pessimistic, keep in mind that I strongly believe in data engineering
- Maxime Beauchemin

In an article in 2017 Maxime Beauchemin wrote a great reflection on the challenges data engineers face.

He also paired it with another article called the rise of the data engineer and this was back in 2017. Before the current uptrend in data engineering content.

Here is his article!

If you enjoyed this video, check out some of my other top videos.

What Skills Do Data Engineers Need?

Data Engineering Project Ideas

If you want to learn more about machine learning, check out DataCamps Machine Learining Course

If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

Or check out my blog

Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio

_____________________________________________________________
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.

*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[troymann]

As an MLE who has been interviewing recently, it has been common to be put into DE interviews by places that do not know the difference. The state of DE makes me very sad: Spaghetti pipelines so difficult to work with many companies are hiring more DEs rather than fixing the problem, low levels of data governance including bad data quality, lack of lineage, lack of data contracts, no explorability, poor security, and outages. I use a wide suite of data tools and ML libraries but during interviews the biggest negative feedback I get is "not enough Spark experience" or "not enough DBA experience". Is that really the extent of how companies view their valuable data assets - as the domain of Spark monkeys and DBAs? If everyplace is just using tools from 10 years ago, why are DE jobs so common? Things just seem to be going in the wrong direction.

[justinhille]

Hi thanks for your insights! I'm a software engineering student possibly interested in DE. You mentioned that the automation of certain DE tasks will be beneficial because it will free up time for more impactful tasks. Will those tasks end up being programming intensive or the opposite?

[geraldgrogan]

Do you feel that Data Engineers will need to be proficient in the No-Code & Low-code solutions as well as providing the scripting for the traditional python type data pipeline scripts? Most enterprise solutions already have large scale implementations of several of the alternate solutions such as Informatica, Cognos, & Qlik?. Wouldn't this make the most sense?

[LukeBarousse]

The Data Engineer must live!!! Great content as always Ben!

[vincentbuscarello]

I'm curious what you think about this since I work at a smaller company and don't know if it applies at bigger ones, but it sounds like DEs could win some respect and $ if we really thought and communicated about the bigger picture: EG - asking "When it comes to data, what are your businesses most audacious goals and most vital needs? Are there opportunities to add value that the business is missing because they dont have our background?" vs "Where do you want the data to go and when?"

I'm torn because automation and crap scares me, but it sounds like we are solving some real problems, so there should be a way to communicate that.

[cliumay]

Imo, I think data is shifting towards to automation and tooling(as you said, some sort of tooling that can power data science cycle without a full time DE team), as well as data integration (i.e. adding new data sources and explore new data sources, as well as those datasources caveat) and data validation (i.e. anomaly detections, checking whether data is valid).

I also agree that things might be boring working as DE but I recently found making automation tools fun, as well as handling various type of protocols like avro, thrift, parquet, and etc.

[mohammedghouse]

Thanks for making this video.
Could you also make a video on how Data Engineering is done at FB?

[Sanjay-xqxr]

Software engineer vs data engineer which profession you would recommend for freshers?

[hardik]

Bro what should i opt either data science or data engineer. What will be the. Easy switch my career from automation

[hdjfjd]

please make a video explaining the role of a data analyst . Also between R & python which is better and to what degree should one need to know either of these ???

[willownot]

I'm starting now in the area, but I was afraid of this from gpt-3, and other AIs that can replace simple human noses, do you think it's possible? is it a profession with a lot of future? are there really these wages of 300usd hour?

[okolokolokolokolokolo]

Which has a better future, data engineeering or web development?

[xA]

Those who build and control the data pipeline rule the world ;-)

[Chiefnice]

I think that point you brought up drawing comparisons of boring pipelines to boring software engineering work as well is pretty good. I feel like pipelines are to data engineers as API's are to backend engineers. Honestly you could even call making API's pipelining I feel like, since their purpose as well is to move around data, just in a slightly different way in a different context. And yeah that can get quite repetitive when you are implementing CRUD operations over and over again for different cases

[tyfwsjz]

Which undergrad degree would be better to get into the DE industry - DE, SWE, CS or IT?

[shauryajain]

Do you think there is a risk of automation in this field?

[kenchu]

Data engineer will be replaced by AI/ robots or be automated or become obsolete?

[harshitsharma]

Is Hadoop itself dying ? And Databricks is boss

[GuillermoBenjamin]

I think the biggest challenge facing The Data Engineer is for it to undergo a name change again.

[antondexplorer]

I'm a web dev moving into a DE role and one of the things I'm trying to wrap my head around is what tools you should use into building data pipelines.

Cause over here on my end we just have a standard mysql DB and we have Sisense gather data from all data sources (3rd party platforms as well) then do the ETL there upto viz. And when I do some research or watching vids you see tools like Apache (Hadoop, Kafka, Spark) or AWS products being used like redshift and the likes. I'm trying to wrap my head on why other companies went that route? what are the pros and cons?