The Harsh Reality of Being a Data Engineer

Data Engineering is a great, fast growing field to pursue and build your career on but there are some major cons to keep in mind. In this video I talk about 3 things that Data Engineers face which makes the job really challenging.

Check out ProjectPro for DE, DS, ML projects. This link below gives you additional discounts!

Follow me on social media..

My POV on No Code:

0:00 Intro and Overview
2:20 2nd Class Citizens
4:56 ProjectPro - Build projects
5:55 Dependencies
9:01 Outdated Tools and Tech

[JashRadia]

Thank you for watching. What are you thoughts on these topics? Let's talk! 😀

[gavinkalaher]

The problem with the Business Analyst specialism is [as an intern/junior], employers expect you to have an MBA and/or prior business experience, a solid interest in business/finance in addition to the technical skills, which creates a significant barrier to entry. As a learner looking to change careers at a mature age, data engineering is far more attractive to me, because the emphasis is primarily on the technical skills.

[anthonychilaka]

Thank you Jash for this frank video, this puts alot in perspective. I am currently transitioning from Warehouse executive to Data sphere and training to improve soft skills in visualization tool (Power Bi) and query language (MySQL). I end goal was and still is to be a Data Engineer, before now I had no idea these three things you mentioned 1. Second tier folks in most compnay 2. Dependency (bridge between producers and stake holders) and 3. Be Update with technology. This is most appreciated. Thanks also for the link to Project Pro cause I know for a fact, a ceritficate will not get me a job its practice and documentation of work done.

[ttuurrttlle]

Your opening skit got me. My team is basically data engineering, but also core development in our small company too. And once was in a project planning meeting that had some managers, project owners, and marketing/sales in it. The main project was just parsing out text from some documents to check for some sort of risk assessment compliance. And near the end after I just briefly explained my thoughts what doing this would entail for our team which was JUST importing data, parsing for certain identifiers, and generating reports; one of the marketing/sales guys asks, "So when we have this can we advertise this as AI and machind learning because that's big right now".

And I basically facepalmed(good thing it was a zoom meeting with my camera off) and got flustered and started say explaining why this wasn't AI and it would be a whole other thing to use that for this project. Which was apparently the wrong thing to say because he just asked why can't we do that? And I'm thinking, "nobody on our team has any experience with that so how can we build that product in your timeframe?" Luckily my boss jumped in and saved me to explain that we couldn't do that now and that it would be a huge time commitment and would derail the whole thing.

[razzor]

As a data engineer, I agree with your points but many of them are barely even negatives, at least for someone who would actually enjoy this kind of job. I enjoy being on the back end and a "second class citizen" as nobody is expecting me to pump out software like they do for the software engineers. The dependencies can get annoying but honestly the investigational work of this job is some of the most fun and keeps you engaged. Learning new software all the time is fun and it's cool to find better ways to do what you were doing before. Overall this job is a privilege to be able to work in as it has amazing WLB, fantastic pay, and low stress compared to 95% of other jobs in the world. Yes it has its challenges but it is nothing compared to other careers, the only hard part really is the initial learning curve.

[arvdosal]

Brilliant, making these types of videos help with communication skills, stuff that I have been told I lack of.
One thing that you have mentioned is the very broad tech skills we have to deal with, but you haven't said that sometimes it requires to switch from one to another with a snap of a finger. Some pipelines are failing and have to fix them, and at the same time doing your normal development work.
Good stuff, I suppose your work is key and haven't been affected by Google laid offs.

[hydrilara]

My sister is data engineer and I am very proud of her. She works 10 Hours a day. Some times i feels sad for her for her long working hours and it is taking serious toll on her health.

[jimaustin]

Excellent video, important points well-presented with real-world examples!

The ‘Dependencies’ section (6:00) illustrates why data (or software or most any other type of) engineering will not be eliminated by ChatGPT. AI will get better and better at dealing with the “this ain’t right” complaints, but it will be a while.

[jaystrangelife]

Great video, really appreciated the insights on the challenges of being a data engineer

[shantayyaswami]

Introductory part was pretty awesome. Good improvements in content presentation. Keep it up so we will also keep supporting you to grow.

[ravivaddi]

Awesome !! love you bro.... your content is really helpful.

[rudroroy]

This video is so on point! data engineering is such fast-paced field that by the time you learn something, something else gains momentum and everyone starts following the trend. It is a never-ending process of learning and upgrading.

[venkatah]

Thanks Jash for bringing up much much valid situations and scenarios for data engineering.. 👍

[seniorfiance]

I'm a business analyst at a small company so some of the data engineering needs will be falling on my shoulders. This was a very informative video with great perspective!

[Abcddfgv]

This is the first video of you and I subscribed! What a content and a clear communication 👏👏

[aishwaryaradia]

The Intro 🤣👌🏻 Very well made video!!! Great Job Jash... ✌🏻

[rishabhkaushik]

It is very well explained, thanks for your effort 👍

[ronnestman]

After that opening skit, I love this guy!

[Harishiv]

3:35 the Witcher shirt already had my attention but you really had my attention when you said you were a former Zser. I’m a current Zser who’s frantically watching these videos to understand the scope of growth of my role. Literally have a connect with my PD coach this week 😂

[mannykhan]

Yeah us Data Engineers are like doctors who people think, know the answer to all questions. I get question as well like "Why cant you do this? Why are we paying you all this money?" and my answer is usually "This is the job of an architect or a data scientist or ML engineer" and this infuriates them to the core.