ETL vs ELT | Modern Data Architectures

Understanding the differences between these two concepts is critical.

These represent two of the most common approaches for designing a data pipeline.

As a data engineer, you'll definitely be expected to know these.

While they are similar, there are some critical differences that you should know.

In this video we will discuss:
- Both ETL and ELT
- Example tools
- Which I would pick (if I were to start from scratch)

By the end you will understand the difference between ETL and ELT and why it is critical for any data engineer.

Timestamps:
00:00 - Intro
00:36 - What is E-T-L
01:38 - What is E-L-T
03:19 - Which is Better?

Title & Tags:
ETL vs ELT | Modern Data Architectures
#kahandatasolutions #dataengineering #dataops

[AlexKashie]

Man, came across your platform today and just find it so valuable.

From a Data Scientist curious to understand a little bit ELT, Pipelines and the backend.
Thank you 🙏🏽

[TA-vfyi]

A nice and concise video, thanks! Would be interesting to hear about some best practices on doing custom data ingestion (EL) pipelines (that is not using Airbyte/Fivetran/Stitch) but writing actual python scripts (which libraries are commonly used, how to structure the project etc).

[alessandroceccarelli]

Thank you for your high-quality videos! In our use case, we ingest daily a .zip file containing 3 .csv’s related to sales, inventory and orders from different shops (20-30) and CRMs (4-5 ; each one with its own naming convention, dtypes, …).

How would you improve the following pipeline?
- Raw zip files are uploaded to a GCP bucket
- The upload triggers a Python GCP Cloud function that transforms the data to create single naming/dtypes conventions and brief new columns (e.g. timestamp by merging date + time)
- Transformed data is uploaded to MongoDB – 3 separate collection for sales, inventory and orders - and raw .csv’s to a separate GCP bucket as parquet files (1 folder for each CRM and PoS as subfolder)
- A PubSub message posted by the function triggers a GCP Function that loads processed data from MongoDB, applies ML models and stores results in separate collections (1 for each analysis type; e.g. forecast, anomaly detection, …)
- A Python web app directly reads ML output data from MongoDB

Thank you so much and love your videos; 🤗

[rahulkishore]

Very elegantly explained. Very concise & straight to the point. Loved the visual showing the different silos of data for Billing & CRM!

[alexperrine]

I get to move my company into the modern ELT approach, thanks for the information!

[woolfolkdoesthings-onemans]

Great video! Super helpful and clear about ELT being the best approach. Question…I see you prefer dbt but how do you feel about Matillion? Thanks!

[JJ-kimw]

Thank you for expalining it thats super easy to understand

[SameerSrinivas]

Would you also call building data models from analytical event tables as ETL? Or is it just abstracted as T of ELT? Thanks for making the video.

[rguez]

Is Airflow another ELT/ETL tool? I mean, can you manage to create an entire data pipeline just with Talend/FiveTran/DBT or how does Airflow enters to the tool set?

[Satyaamevjayathe]

This seem you are suggesting various data type and formats be brought into the single platform and then use tools there to transform

[teamwasted]

I don't understand how you can load the data into a "more permanent" table before you transform the data because many times when you transform the data by applying business logic, you are changing the grain and schema of the data. Am I missing something?

[michaelsegel]

Sorry Michael, but you should have attended more CHUG meetups and learned something about Big Data and doing ETL.
There is no such thing as ELT. Its really ETL.