Getting started with Dagster | Create Python ETL | Orchestrate ETL Pipelines with Dagster

In this video, we will cover an exciting new application called Dagster. It used to orchestrate your Python pipelines. Dagster has a user-friendly user interface and gives us better options of logging and history of the jobs we run with it. Dagster comes as a python library and you can quickly get setup and running with it.

Get started with Dagster in just three quick steps: Install Dagster, Define Ops and Materialize the assets.

Create a virtual environment: python -m venv env
Activate the virtual environment: env\Scripts\activate

To install Dagster into an existing Python environment, run: pip install dagster dagit

For projects using newer version 1.1.20 or 0.17.20 the command to create a new project has changed. To get started, you can run:
pip install dagster
dagster project scaffold --name my-dagster-project

Additional libraries required: Pandas, psycopg2

Create a new project: dagster new-project etl
dagit
dagster-daemon run

Subscribe to our channel:

---------------------------------------------
Follow me on social media!

---------------------------------------------

#Python #ETL #Dagster

Topics covered in this video:
0:00 - Introduction ETL with Dagster
1:17 - ETL Direct Acyclic Graph (DAG)
2:25 - Dagster Setup
3:32 - Dagster Project Overview
4:48 - Run Dagster
5:25 - Dagster UI Overview
6:47 - Write Python ETL Pipeline with Dagster
11:18 - Run ETL Pipeline from Dagster UI
12:55 - Run relatively Large dataset test

[bralabala]

very helpful, but a few things have changed since the project was recorded. for example `dagster new-project <name>` is now `dagster project scaffold --name <projectx>`

[ianyoung_]

Thanks for the helpful tutorial. I'd love to see a follow-up on how to deploy to a production environment using CI/CD. The workflow from local changes to production deployment would be very useful.

[rizzrak]

Helpful tutorial. Thanks for this. Pls make more videos

[tkeus]

thanks a lot man ! i'm starting out with dagster and i'm completely clueless . this will help out a little bit :)

[siddharthasahu]

I work in Windows Subsystem for Linux, Just so because Linux is more comfy for me.. Nice Tutorial btw

[hungnguyenthanh]

hi, i watched the video and it's great. You said in the video that Dagster is only suitable for ETL with small to medium data sources, you rate Dagster as medium to good. But I have the following advice: your data pipeline is using python, so I think this ETL performance depends on the ETL tool here, python, not Dagster. If we use Dagster to manage the data pipeline for ETL work like Apache Kafka, Pyspark, Dbt tools then I think it's much faster. I'd say that ETL performance is in the technology used and not the management tool. thanks for reading.

[ishatripathi]

Thanks for the video it was really helpful. Could you make some more videos on dagster like a tutorial or something like that.

[BiInsightsInc]

Related videos on Dagster & ETL orchestration topic:

[alexzir]

Thanks 🙏 Continue about Dragster please

[Pasdpawn]

hi, great video. have one question though. how do i run the scheduled dagster job even when my pc is turned off? Cos when my pc is off, dagster daemon wont run and therefore the job will also not run. how do i overcome this?

[alexzir]

What is better for you Airflow or Dragster?

[lokendrasinghtanwar]

having issue in setting up environment variable, what will be the directory for DAGSTER_HOME variable

[pybokeh]

Aren't you missing a workspace.yaml file? You can't just run the dagit command @4:50 by itself without the workspace.yaml file.

[Vasavi-zl]

Now im trying the exact same thing but getting errors. get the provide the new version video or documents that helps us

[hungnguyenthanh]

I don't know if you can make a video on how to install it on docker.

[harshitamehta]

command for creating a new project is not working, dagster new-project etl. Getting error, AttributeError: module 'pendulum' has no attribute 'Pendulum'

[ExploreWithArcha]

command for creating a new project is not working, dagster new-project etl, what to do

[julesm]

No jobs
Your definitions are loaded, but no jobs were found.