Tutorial: Airflow + PySpark + dbt

You'll build a complete Analytics pipeline from scratch. By the end you'll have:

• Airflow scheduling and running your pipeline
• PySpark transforming raw sales data
• dbt creating clean analytical models
• Postgres storing everything
• pgAdmin and Grafana for visual inspection

The tutorial has 6 short steps. Each one adds one piece to the puzzle.

---

What you'll build

A sales data pipeline that:

1. Extracts synthetic daily sales records
2. Cleans and aggregates them with PySpark
3. Creates a sales_summary mart with dbt
4. Runs automatically via Airflow on a daily schedule
5. Exposes results in Grafana dashboards

raw_sales.csv  →  PySpark clean  →  Postgres (staging)
                                           ↓
                                    dbt run  →  sales_summary
                                           ↓
                                        Grafana

---

Prerequisites

• Python 3.12+
• pipx
• Docker Desktop running

Install DEGEN:

pipx install degen-cli
degen version   # degen 0.7.0

---

Steps

1

Init & Install

Scaffold the project and install dependencies.

2

Start Services

Launch Postgres, pgAdmin, Grafana, and Airflow.

3

Create the DAG

Write the Airflow DAG that orchestrates the pipeline.

4

PySpark Transform

Write the Spark job that cleans and loads the data.

5

dbt Models

Create staging and mart models with tests.

6

Run End-to-End

Trigger the DAG, verify results, open Grafana.

Begin: Init & Install →