Is Data Science Dying in 2022?

Free Dataiku Learning Resource:

In this video, I talk about if data science is a dying field in 2022. I think this is a very legitimate concern since we've seen whole industries become obsolete in a matter of years as technology keeps advancing.

From LEARN Media

[_im_a_teapot]

"Prospective data scientists should focus on projects that demonstrate the ability to understand business context"
This! This is very important!

[GuppyPal]

Asking if Data Science is dying would be like asking if engineering or medicine is dying. There will always be a huge demand for these sorts of careers. Tons of people thinking they want to do them is a byproduct of that. But just like engineering and medicine, most people who think they want to be in the field can't really hack it (for whatever reason). If you are someone who is smart and hard working, you don't have to worry about the field being "saturated." Most of that saturation is from people who aren't very good or who call themselves data scientists because they completed a mooc or two. There's a huge difference between completing certs and degrees and being able to actually do things in the real world. The wheat gets separated from the chaff pretty quickly.

[ccmps]

The automation part was a matter of time, since a lot of the work with the data is repetitive. The understanding of the business and its needs have always been the most important part of Data Science.

[joshelguapo]

Honestly if you're a data scientist whose job can be automated, you're not a data scientist, you're a software engineer that uses sklearn.

[MonsieurSchue]

Personally I think you hit one key point: the importance of domain experts. With the so-called automation or new technology/ tools we'll likely see a shift to empower the domain experts and focus more on the business needs. Just because someones knows SQL or do coding doesn't mean he/she can do data science or machine learning -- or more importantly, to put out anything meaningful or even useful. Just like years ago people needed to rely heavily on someone who can do graphic design really well or who can use a handful of high-skill tools (say photoshop) really well to even modify a picture or video. These days even kids can snap a picture or video and publish to Instagram or Tiktok. Did the modern photo editing tools make graphic designer extinct? No. But it does mean that you need to be REALLY GOOD to be able to survive as a graphic designer. I see the same future for data science field too. Maybe not tomorrow, but the trend is coming for sure. If you pay attention to some of the podcasts like "me myself and ai", most the guests (chief data scientists or boss of DS departments) aren't really of these backgrounds but more business driven.

[lightning]

What a beautiful and insightful explanation of the dynamics of data science. It is true that data science demand is growing exponentially and what aspiring data scientists are fearful is lack of knowledgeable that DS is actually evolving and diverse. It is become more field specific and one need to master the area of expertise like health, business, banking etc in respect to the application of DS in it. Thanks a lot Thu Vu for your resourceful info.

[Entrepreneur-ish]

❤‍🔥❤‍🔥❤‍🔥 Dope presentation! I appreciate your depth of explanation here and your overall knowledge. Watching your videos actually convinced me to get into a data job, and now I'm a Business Analyst at a media measurement company. Keep dropping these 💎💎💎

[williamsanzvivanco]

That's the point.
For example, I work as BI Analyst, normally with Power BI, SQL and low-code platform (PowerApps, Automate...)
I have a Master in Data Science and I know how to built Neural Networks from Scratch but...for what?
Built a single linear regression model requieres a lot of steps and technologies, Azure, Python, Cloud, APIs, Devops...a lot of investment and maintenance,
instead most of the tools which I work on, have integrated machine learning tools, as image recognition, NLP, classification... with a few steps and a clear suscription.

In fact, I get my job because they demand for a person who knows analytics and have data science knwoladge so that can advice on the use of these integrated tools.
No more import keras model.fit(), you will be requiered to matinenance and advice about machine learnings integrated tools.

[theoutlet]

The whole data science industry is booming from analyst to engineers to scientist and you will see domain experts leading this field more and more

[jacktrainer]

DS/ML will never replace human judgement. Even professors at places including MIT and Columbia recognize that it is intended to supplement and guide human judgement. The discipline has been around for as long as statistics -- it comes in and out of fashion, but never goes away. Definitely saving this -- my Meta ML interview is in Sept.

[JamoSimcity]

This was very insightful. And you explained "unknown unknowns" way better than one of my previous professors from college lmao.

[annaczgli]

Agree on the need for new hires to also be good with domain knowledge & business context.

No one will pay for someone that can only model.fit(), when they can cheaply automate it.

[juliansihite]

Even data analyst is more complex in term of creativity and it can't simply automated by AI.
Data modelling (data science) of course can be automated, but insight (data analyst) in my opinion it is difficult to be automated with complex pattern.
Data Analyst deal more with soft skills, while Data Scientist deal more with hard skills. Soft skills is hard to be automated like creativity, critical thinking, finding which data is more relevant.
And for me, I don't need creativity to work with Data Science, it's purely technical skills and can be gained with 3 / 6 months bootcamp, nothing's special.
I would consider more AI researcher / ML Engineer with more complexity. But there are many people overrate Data Science too much, while not everything can be solved with Data Science.

That's why I told my boss to pay more for Data Analyst / Data Engineer / Software Engineer rather than overproud-Data Scientist.
Even the product (SaaS) is made by the software engineer team, don't talk about supply-demand, if the core is SE Team. Data Science is only support team, but they act like the most contributor team, just because they give insight directly to stakeholders and business team.

[larsjensen]

Totally agree with you. I hold a bachelor in technology management, which is really not a programming education.
But suddenly I realized that I was doing data analysis each day, to trouble shoot and optimize complex machinery in the field of green tech.
I realized that it was because of my strong "domain knowledge" as you call it. And it seems like a really few people can do it.
So now I'm looking to sign up for a master in Data Science here in Denmark, to help the world transitions to lower energy consumption and cleaner future.

Thanks for inspiration <3

[fidgetykoala]

Coming from Philosophy and then having done a degree in business, I'm fond of what you just said, namely the domain expertise, and the foundational questions. Higher education focuses on strategies rather than the new shiny automation software. However, when it comes to interviews, especially in the field of marketing, they assume the candidate knows how to use automation software. I'm puzzled to say the least and I wish I stayed on my previous path, rather than focusing so much on job market demands which anyway I'm not able to satisfy. Engineering, mechanics and statistics (just to use a more classical academic vernacular) are going to take the majority of the jobs market share...

[emarc]

TL/DR; Domain expertise is crucial over generic Data Science skills and will make you employable in whatever field you have chosen for your career.
Learn to code properly (i.e. program) and how to use API to data off from any source.
Thanks for the insight.

[Nb]

I agree that the fears of full automation and a fully saturated job market are overblown but at the same time I think there's a real pair of twin concerns that people considering advanced education should be concerned about: democratization and the experience catch 22.

I feel like your video alluded to what I mean by democratization quite well so I think we're probably in agreement there. A lot of programs focus on training people to understand the ins and outs of libraries like tensorflow, pytorch, and sklearn as well as how to mathematically derive and formally describe a bunch of models. I think a lot of the implementation of those models is routine and boiler-plate enough that it's ripe for "automation" using some kind of auto-ml, the sql-like syntax that redshift and BigQuery offer, or some managed product that abstracts the implementation away. This opens ML models up to the tech-savy but non-specialist employees who have a very firm grasp on the domain knowledge already, pushing the roles that require deep technical expertise farther out to data engineering and ML-OPs. We still need people to hyperoptimize models and do core research for critical applications but often it's just not business critical to hire full teams to read papers and install cutting edge deep learning models off of github all day.

For experience I really think there's a cruel pattern emerging of people investing enormous amounts of time and money toward masters programs and endless self-teaching motivated by the perception that the industry is starved for people. This is where I think we disagree: the job opening numbers you present are accurate but you don't talk about how that need is distributed with respect to years of experience. The largest companies have well structured well funded teams that can afford to take on new grads that need training but a lot of companies are only interested if you have a proven track record and can convince them you can deliver value extremely quickly with little need for them to train or invest in you. I would love to see a graph overlaying the number of candidates at each experience level over the number of jobs searching for people at that level. My expectation is that you'd see a massive gap favoring employers at 0-2 years of experience and a massive gap favoring employees at 5+ years of experience. So while the industry is expanding, the companies aren't hiring juniors, or at least not at the rate you need to be optimistic coming out of an expensive not top tier masters program.

Together I think these trends speak to a frightening future for the opportunities for new data science professionals. If workers with domain experience (but not a DS education) can be hired directly into the junior roles given the democratization of the field and if education alone is insufficient to get you an entry level job I think we're going to see a very desperate job market for people trying to get into the industry. Perhaps what will happen is that new grads will need to pick an industry and work a non data science job for long enough to gain years of domain experience but that can be a hard sell after spending enormous amounts of time and money on a new degree

[dakshbhatnagar]

Absolutely! Without humans, it’s just numbers that computers cannot make sense of. Hyper parameter tuning might be automated but the optimised parameters might vary depending on the business requirement which will come from domain expertise only.

[binry_dstructr]

If you're trying to get into AI/ML/DS but find the requirements in job postings too restrictive (e.g., a PhD in CS/math/stats), an alternative approach is to make a lateral transition from your current role.

I've been working cyber security for just over 10 years now and have started looking for ways to incorporate DS into my work. If I'm successful at convincing my managers to get the fire started, I can soon lead my own research teams or at least participate in side projects involving collaboration with other teams.

Don't be afraid to think outside the box of the traditional applying for a job, getting rejected, rinse and repeat model.

[Cantor-ubwj]

The real threat for data science is not automation, is that only a tiny fraction of companies really need pure, hardcore data science. Simple as that. There are many more crucial milestones than, strictly speaking, building data science products, that need to be achieved even before data science can kick in. And you know what? When it does, the reality is that the level of expertise required to produce visible and "good enough" results can be fulfilled by other, more general purpose, profiles, such as ml engineers, data analysts, bi managers. And there is no need for evidence to substantiate such a statement, just look at the market. In the UK, for instance, very few data science positions entail actually doing data science (building ml products and pushing or at least send them to production). In most cases what one ends up doing is business reports, data cleaning, sql queries, embark in dubious projects that almost never make it to the end. This also causes the data science job to become more and more vague and poorly defined, which can cause pressure as it forces the DS to constantly reinvent themselves.