The Harsh Reality of Being a Data Engineer

In this video, we are discussing the harsh reality of being a data engineer, the good, the bad and the ugly. Wait till the end of the video and share your thoughts in comments. Anything surprising to you?

WATCH THESE VIDEOS NEXT

Try AI for coding & data analysis:

~~~ CERTIFICATES & COURSES ~~~

Data Scientist

Data Analyst

Data Engineer

Python

SQL

Machine Learning

Statistics

EXCEL

~~~~~~~~~~~~~~~~~~~~~~~~~~~

MY FAVORITE TOOLS

SUBSCRIBE FOR MORE VIDEOS

BINGE THESE PLAYLISTS NEXT

ABOUT ME

PS. Some of the links included are affiliate links and help me keep this channel going. Thanks for the support.

[AchuVlogs]

Im a data engineer with 4 years experience. Harsh reality of being a data engineer is:

- 5% building new pipelines
- 15% automating the pipelines
- 20% adding data quality checks to the pipelines

And 98% of the time fixing bugs in OLD pipelines.

For people who like building new products, this is not the right role. Because most of the time you're only firefighting. And since you work in backend, you don't get much recognition.

[hegerwalter]

Well, everything new requires you to figure out your path. I studied to be a mathematician but needed to find a job. Salaries in programming supported a more affluent lifestyle, so just a few classes in CS helped get my foot in the door. I think that the guy who wrote a violent race car game had a history Ph. D. I was reading about a philosopher who worked as a glass lens grinder, who died from cumulatively inhaling glass fibers in his 30s. Like everything else, you might need to change careers or job titles. Being a Data Engineer and knowing programming will get you further.

[vishnuvrv]

There are two types of data engineers: Software Engineers who are also data engineers and ETL engineers. If you are an ETL engineer (building pipelines using python and SQL) then you are just considered a glorified data analyst and your pay always be less than SE. My advice for them is to take the next step and become a cloud engineer (snowflake, Azure, AWS) and see your pay increase significantly.

[abiola]

The roles of a software engineer and data engineer are totally different, even though they might use related tech stacks. Also a data scientist can't take the role of a data engineer because, apart from being technical, a data engineer is expected to communicate with stakeholders and build solutions that turn data into insigts/revenue; data scientists suck at this. So a good data engineer should be able to do the work of a data analysts effectively. According to a Gartner report, 80% of data science projects fail. The number reason is: Lack of understanding of business context and user needs.

[joelkanamugire]

I have been working as a Data Engineer for 3 years and I think it depends on companies and how they define their data engineers. I always find myself building data pipelines, API design, Web applications, Data Encryption and Database Security

[omotayosawyer]

stressing on the point of a software engineer being able to do what a data engineer does is absolutely correct. I work as a data analyst and in one of my previous roles, the software engineer was pretty much the data engineer completing all the data engineering task.

[SantoshKumarsp]

Hi Sundas, I am a bit confused. You are initially saying that SEs can do DEs role and also, with advancements in AI and possible taking over core SE's role, even they would do DE's role. Then, at the end, you are saying that DE's role will still be in demand?

[rangho]

The aspect you might not have covered is that as most Data Engineers progress, they don't remain DEs but more so SREs and System Software Engineers, or systems architect as since they have a clear idea of how the information (data) is flowing, they have a clear idea of the system flow.

[michalms]

Thank you - there are couple interesting points.
Most software engineers who I know miss the skill of data modeling. Data engineers speaks more with data users and they are closer to data analysts (at least this is what I do). SDE need to know more technologies and dependencies but DE need to know more about data she/he plays with. Point about career growth has something, however I meet some Data Engineering Managers. But probably that is it. Then maybe DE switch to DS job but it is horizontal move and not a promotion.
One last point is about logic - if SDE earns more then they will not do DE job for less. If they would be forced to do DE but paid the same it would still mean for them losing skill and time for something not related to SDE. It would be harder for them to change a job. That is why there is probably still demand for DE.

[max]

just got an internship as a data engineer(while at college) for the city of ottawa(only 1 spot btw)...wont have done it without you. Appreciate you..

[j.amethyst]

Great review, but i cant agree career growth. From Data Engineer you become a solutions architect. Or bridge being a DE to the next sensible path.

[abidrahim]

Thank you for putting this out, appreciate it! 🙌🏼

[muuwsrg]

Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me, but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql, Spark) might be a smoother path for me? Appreciate any advice!

[sergiocarmona]

Im agree with all except number 2 software engineer can make easily DE work?, i think no, many DE were DBAs in the past. DBA is complete different skill to software engineering. 3. i think there is growing for DE you can skill up into machine learning engineer more easily than other roles.

[neetsabhi]

absolutely disagree that SE can easily do DE role, working with data is different than coding, I have seen so many SE struggle with structured data let alone distributed computing and unstructured data, understanding data modeling, performance, ETL, understanding multiple storage and DBs. This is the high ground that SE need to get off, this is the reason why leetcode hiring gets you a really bad data engineer who is good at coding vs getting a really good data engineer who is average at coding.

[yousefosman]

i was confused about that thank you i will focus on data science then i will convert to data engineer because i don't see a particular road map for data engineer

[kimblevuo]

Just started Data Analysist course few weeks ago.

[mohammedabdulla]

Nice, I came to know that data engineers paid lesser than the average software engineer
Felt sad 😔

[yogajyothisomu]

Data science or data enginering which is better based on opportunites as a freshers?

[prar_th_.]

Hi I'm a high school senior right now and I want to pursue a career in data science job specifically would be wonderful if you could do a video on AIMl because I feel like there isn't much on the internet about the same