What is a Data Engineer?

In this webinar, we will explore what is a data engineer. This includes discussing what are the goals, skills, and tools that they use on a daily basis. We wanted to take a moment to provide insights into the other options data professionals have besides data scientists.

Data science has attracted a lot of buzz for almost the last decade. The ability to gain insights from data has driven much of the high-tech companies like Facebook, Netflix, and Google's success. However, it is not the only piece of the data puzzle. Quietly, in the background, behind all the models, metrics, and dashboards, another type of data professional has worked to ensure data scientists have the data they need.

These are the data engineers. Data engineers work in the background managing data pipelines and data storage systems to ensure that the rest of the company can get their dashboards, metrics, and models. They use a variety of tools like SQL databases, Hadoop, and Python and are proficient in data modeling and automation.

Table of Contents:
0:00 – Introduction
6:32 – Data modeling
10:00 – Automation
13:41 – ETL Development
15:48 – Product understanding
17:15 – Data Pipeline
19:33 – Pipeline tools
27:30 – Data engineer versus data scientist
30:38 – QnA

About the Speaker:
Ben is a Seattle based Data Scientist & Engineer working in San Francisco, California. He has extensive experience designing ETL pipelines, databases, websites, and other software products for startups and pre-established corporations.
--
Watch more community talks:
--
Learn more about Data Science Dojo here:

Watch the latest video tutorials here:

See what our past attendees are saying here:
--

Also find us on:

#datascience #datascienceforbeginners

[Datasciencedojo]

About the speaker - 0:18
Overview - 0:59
Goals - 3:13
Skills - 5:51
Data Pipelines - 6:05
Automation - 9:48
ETL Development - 13:17
Product Understanding - 15:26
Why develop data pipelines? - 16:57
Tools - 19:12
Data engineer impact - 24:16
Data Engineer vs. Data Scientist - 27:08
Data Engineer vs. Business Intelligence - 29:05
Q\A - 30:31

[sleepyBear]

I really never knew what a data engineer was, after listening to this it sounds like what I support on a daily basis as a technical support engineer, i need to pursue this.

[kashifhaque]

Aweseome. This is the path I want to pursue.

[tamboyistheman]

This is really interesting, my official job title is software engineer but all the work I do really revolves around Data Engineer roles. All the things you said here, like building data warehouses, etl, automation, the tools like apache airflow. I work with all those tools inclusively. So this video served as my confirmation that I am a data engineer. I enjoying what I do honestly and would like to continue with this role. I just hope it pays well.

[OceanLoader]

Hmm. Considering Data Engineering re-skilling thru employer. Currently a DBA. I like the idea pipeline development, data extraction and automation. I hate the thought of dashboard development. BI Development is something I avoid.

[pohau]

So as a Software Architect trying to understand what Data Engineers do, Is the snowflake architecture done by a script to generate it's content via one of the slave DB to an isolated environment strictly for Data Engineers?

[_______]

Oh god the way he explained what a cron was when asked what a cron is in QA... I could tell he knew what it was but horrible job explaining it by using “basically” too much. He could have simply just said that it’s a task scheduler.

[widussss]

Do you think that this degree has a long future, or it could be automate by programmers. Thank you.

[kennylaikl]

Will Data Engineering and Cloud Engineering converge to become DATACLOUD Engineering?

[jamirajamira]

Is it safe to say that a Data Engineer is basically "a BI Developer who uses modern tools"?