Being A Data Engineer: Expectations vs Reality

What is a data engineer?

The goals of a data engineer are much more big-picture and development focused. Data engineers build automated systems and model data structures to allow data to be efficiently processed.

This means the goal of a data engineer is to create and develop tables and data pipelines to support analytical dashboards and other data customers (like data scientists, analysts, and other engineers).

It's similar to most engineers. There is a lot of design, assumptions, limitations, and development that occurs to be able to create some sort of final robust system.

Learn more about being a DE for free with googles DE certificate

0:00 Intro
0:48 Hadoop And Spark?
3:04 Isn't it All About Data Pipelines
5:02 All The Focus Is Not On Data Engineers
6:15 What to expect for a data engineering salary?

If you need data consulting help, then reach out to our team here:

Also, if you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.

Check out my Medium here:

What Skills Do Data Engineers Need?

[jeanjurjevic]

I've been a Business Intelligence Engineer for 7 years now. You got it, one other piece that you are missing is auditing and accounting. Ensuring your numbers are correct to the source system.

[clipdat]

I'm not one to usually comment but you are incredibly good at explaining and layering information, you taught more in 7 minutes than others with 30 minute videos, thank you for taking the time to share your experiences.

[darsh_shukla]

It's a year and a half I started as Data Engineer and every word you said it's truth. Thanks for the video.

[mohammedsharikuzama]

Loved the crisp to the point no bs video. Thank you!

[protos]

I've been working 4 years as a Data Engineer with the BI Developer title. All you said is on point, and SSIS brought back memories of working at the beginning in a Microsoft only environment.

[markskywalker]

Thanks man! Appreciate your sharing and honesty! Good luck and best wishes.

[anando]

Very insightful and I can definitely relate as to how DE is a "behind the scenes" type of role. Great video 👍

[arturopresa]

Thanks for the information, I learn a lot about Data Engineer expectations in this video

[RizAngD]

Can't disagree more about Data Engineers being the middle / behind the scene man... great video :)

[nicolasdemaria]

Newt week I'm starting a position as DE I'm really excited!

[ngee]

Very helpful, thank you for explaining!

[alexanderbernard]

Nice video, I wasn’t expecting to hear SSIS but it was nice to see you mention it. I’ve used in school and in my own side projects. It’s like a love hate relationship at first because when it works I have to issues with it, but when there’s an error sometimes it’s hard to figure what the error code really means and what’s going on with the rows being affected. It seems like SSIS is dying from what I’ve been told and I’m not sure how easy it is to use Hadoop or spark as I’ve only used those programs once in lecture.

[rebeccaclafton]

Thanks for this! I've been studying to be a data analyst and have realized that I'm far more interested in a data engineering role. I've been a bit put off by the Spark, Hadoop, Kafka, etc. neverending list of "must have" skills for it though. Interested to see what else I can learn from you!

[scottfitzpatrick]

I appreciate the tips I am new to the term ETL but I have done many data migrations and relational table design. I at least have a little bit of background on the role.

[ZachRenwickData]

I'm so glad I got to skip over the Hadoop era... can't imagine how complex it would be to try and manage all of that (especially on prem)

[samuelreid]

This was a very good video, I am now trying to switch to a career in Data Engineer

[saurabhverma]

I am trying to switch career from data analyst to data engineer. This video really motivated me.

[sawonbhowmik]

Recently I got selected as a Jr. Data engineer in a company...I watched your video that helped me understand the concepts of data engineering. Thank you...

[levialberto]

I've been seeing something like that. Since I started as a data engineer as just code a bit more with SQL ( even with spark). Besides it I just use interfaces to manage jobs (jobs written in SQL + shell script but a very basic shell script demands.)

[__thytran]

I'm going to apply for this position next year. Thanks for the video.