All case studies
2025
Weather ETL Pipeline - Apache Airflow
Scalable ETL pipeline for real-time weather collection, transformation and storage, orchestrated with Apache Airflow over PostgreSQL and deployed by one Docker Compose command.
Stack
- Apache Airflow
- PostgreSQL
- Docker
- Python
- ETL
- REST API
- SQL
- Data Validation
Problem
Weather data is only useful if it arrives on a schedule, survives API failure, and lands somewhere queryable. That is an orchestration problem more than a data problem.
Architecture
Airflow DAG (TaskFlow API) -> Open-Meteo API extract -> transform -> PostgreSQL load, with data-quality views over the result. All services in Docker Compose.
What it does
- Apache Airflow orchestration using the TaskFlow API
- Real-time extraction from the Open-Meteo API
- PostgreSQL storage with data-quality views
- One-command Docker Compose deployment