End to end ETL pipeline project using Docker, Airflow, PostgresDB and Metabase | Data Engineering

🌦️ Build a Complete Weather Data Pipeline with Airflow, Docker, PostgreSQL & Metabase 🚀

In this video, I walk you through a full end-to-end data engineering project where I collect real-time weather data from the OpenWeatherMap API, orchestrate the pipeline using Apache Airflow, store the data in PostgreSQL, and visualize it beautifully in Metabase — all containerized with Docker Compose.

🔧 Technologies Used:
- Apache Airflow (for ETL orchestration)
- Docker & Docker Compose (for infrastructure)
- PostgreSQL (for storage)
- pgAdmin (for DB inspection)
- Metabase (for dashboards)
- OpenWeatherMap API (data source)
- Python (for scripting ETL)

All codes mentioned in the video are available in above github repository.

📌 Project Highlights:
- Uses Airflow to fetch and load weather data for 10 UK cities
- Coordinates-based API requests for accuracy
- PostgreSQL table schema auto-initialized
- Fully containerized setup — run with a single command
- Clean, professional Metabase dashboard

⏱️ Timestamps:

0:00 - Introduction
1:08 - Openweather API
2:44 - Docker Desktop
3:22 - Project Overview
4:18 - Project folder overview
4:53 - Docker compose file
9:57 - Project Demo
10:58 - Airflow code
14:19 - Run the project
19:12 - Visualization
31:49 - Conclusion

📌 Want to try it yourself?
Clone the repo, add your own OpenWeatherMap API key in the `.env` file, and run:
```bash
docker compose up --build