Data Engineer vs Software Engineer

What's the difference between a Data Engineer & a Software Engineer? How do you define each and should data engineers be doing software engineering as part of their job?

Jesse Anderson is a Data Engineer, author and trainer who runs a company called the Big Data Institute where he helps companies all over the world with their Big Data problems.

-------------------------------------------------------------------------------------

⭐ PATREON:

Join the Continuous Delivery community and access extra perks & content! If you join this month you will be entered into a book giveaway to get a free copy of Dave's latest book "Modern Software Engineering", or even a signed copy of one of Dave's books.

_____________________________________________________

🙏The Engineering Room series is SPONSORED BY EQUAL EXPERTS

-------------------------------------------------------------------------------------
📚 BOOKS:

📖 "Continuous Delivery Pipelines" by Dave Farley paperback ➡️

NOTE: If you click on one of these Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you.

-----------------------------------------------------------------------------

Also from Dave:

🎓 CD TRAINING COURSES

📧 Get a FREE guide "How to Organise Software Teams" by Dave Farley when you join our CD MAIL LIST 📧

-------------------------------------------------------------------------------------

➡️ Links

[jessetanderson]

Looks Dave forgot to cut the part where I start coughing.

[feasterfamine]

This common approach to SQL engineers as “engineers that don’t know how to program” is why I don’t do this for a living any more. If you don’t think SQL involves coding then you are not a competent data engineer.
The prevalence of software engineers that think data is easy and argue with/ignore the expertise of people that specialize in data for a living is one of the biggest problems with the industry IMO.

Particularly within the context of data sets that most organizations build without notable consideration for whether/how to use the resulting data, a whole generation of software engineers have come along that generally have no idea how to create valuable, elegant, and supportable data sets. EF and equivalents propagate this issue and invariably result in unsupportable, non-scalable, inelegant data sets that data engineers have to fix after you have moved onto the next opportunity.

In 25 years of coding I never met one software engineer that considered a data engineer to be a peer worthy of collaborating with, and not one leader that understood the process and stuck around long enough to bridge the gap.

[l_combo]

Jesse is awesome, met him in London recently at YOW. Love this point: 'If you only know one technology / language then you have no options'

[muditsingal]

Hi Dave, being a budding data engineer I would love to hear more on this hot topic. Have been following your channel for software engineering methods and most of them does make sense but the cost optimization problem is always a blocker to follow most of the laid down agile methodologies.

[bernardogalvao]

This was lovely. I'd love to hear one about MLOps one of these days!

[atkinpaul]

Love the definition of data engineers except for equating it with "big" data because 99% of companies don't have big data.

[theodorealenas]

I slowly realize how little we're taught in the informatics school. We did SQL queries, and a full stack project where I used them, and that's it. The when and why sound foreign.

[TheApothecaryAus]

Dave what's your thoughts on business analysts/architects in the modern software world - I know you've touched on Product Owners/Managers & Scrum Masters, although I think they're slightly different in purpose.

However I think that Jesse's point around seeing multi-million dollar projects fizzle out into the ether, those projects could have had been successful having that BA facility to get the lay of the land between client (or client's customers) needs and what's feasible based on your technical teams capacity/capability.

[CynicalOldDwarf]

When Jesse says "A software engineer that knows SQL", to what extent?
Just general query stuff (including joins) and stored procedures, or some of the more nuanced DBA wizardry like indexing, query optimisation, etc?

[c.ladimore]

i like programming, but i hate SQL noSQL should be taught/learned instead, esp. w/ huge databases and machine learning. SQL queries and joins and crap are just tedious, but had to learn them first for some reason

[RooftopDuvet]

These discussion format videos are really great. There's so much knowledge encoded in the things people choose to enunciate in regular conversation that's very hard to replicate in a tutor-to-audience format.