Data Engineering Environment

Enterprise data engineering lab with PySpark, DuckDB, Apache Airflow, dbt, Delta Lake, Apache Superset, MinIO (S3-compatible), Redpanda (Kafka-compatible), and 20 guided exercises covering the full data engineering lifecycle.

Who Is This For

Prerequisites: Basic Python, basic SQL, comfort with terminal/CLI.

Not for: Complete beginners who haven't written code before. Start with Python fundamentals first.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                    Data Engineering Stack                                 │
├─────────────────────────────────────────────────────────────────────────┤
│  Processing        │  Storage            │  Orchestration & BI           │
│  ─────────────     │  ────────────       │  ─────────────────           │
│  PySpark 4.1       │  MinIO (S3)         │  Apache Airflow              │
│  DuckDB            │  Delta Lake         │  Apache Superset             │
│  Polars            │  Parquet/CSV        │  Jupyter Lab                 │
│  pandas            │  DuckDB (OLAP)      │  dbt-core                    │
│                    │  Redpanda (Kafka)    │  Great Expectations          │
├─────────────────────────────────────────────────────────────────────────┤
│  CLI Framework: data-help | data-stack-up | data-demo | data-run        │
└─────────────────────────────────────────────────────────────────────────┘

What's Inside

Core Tools

Infrastructure Services

Additional Tools

Quick Start

# Start infrastructure services (MinIO + Redpanda + Superset)
data-stack-up

# Run end-to-end demo pipeline (validates everything works)
data-demo

# List all exercises
data-run

# View an exercise guide
data-run 01

# Run the exercise script (live execution with colored output)
data-run --exec 01

# Show all available commands
data-help

CLI Framework

Stack Management

Pipeline & Demo

Interactive Tools

Exercises & Learning

Data Operations

Cloud Integration

Services & Credentials

MinIO (S3-compatible Storage)

Pre-created buckets: data-lake, raw-zone, staging-zone, curated-zone Sample data pre-loaded in raw-zone/sales/, raw-zone/customers/, raw-zone/products/

Apache Superset

Pre-configured: DuckDB database connection, sample datasets (sales, sales_by_category, sales_by_region)

Redpanda (Kafka-compatible)

Exercises (20 total)

Learning Path

Exercises are organized into three tiers. Start where your skill level matches:

Beginner (no services needed): 01, 05, 06, 07, 13, 16

• SQL fundamentals, data quality, Polars, Parquet, profiling, validation
• Pure Python — runs without data-stack-up

Intermediate (local services): 00, 02, 03, 04, 08, 09, 11

• PySpark ETL, Airflow DAGs, dbt models, Delta Lake, Superset dashboards
• Needs data-stack-up for some exercises

Advanced (full stack): 10, 12, 14, 15, 17, 18, 19

• MinIO ingestion, Kafka streaming, lakehouse, catalog/lineage, end-to-end
• Requires data-stack-up for MinIO and Redpanda

Exercise List

Running Exercises

Each exercise has two modes:

# View the exercise guide (instructions + code snippets)
data-run 01

# Execute the reference implementation (live output)
data-run --exec 01

The --exec mode runs a Python script that demonstrates the exercise concepts with real data processing, colored output, step-by-step progress, and timing information.

Data Demo Pipeline (data-demo)

Runs 10 stages to validate the entire stack:

┌─────────────────────────────────────────────────┐
│ 1. Generate sample data        → output/data/   │
│ 2. PySpark ETL                 → output/etl/    │
│ 3. Delta Lake tables           → output/delta/  │
│ 4. DuckDB analytics            → output/analytics/ │
│ 5. dbt transforms              → dbt models     │
│ 6. Data quality validation     → pass/fail      │
│ 7. Airflow registration        → DAG list       │
│ 8. OneLake upload (optional)   → cloud sync     │
│ 9. Superset dashboard setup    → superset init  │
│ 10. Execution report           → output/reports/ │
└─────────────────────────────────────────────────┘

Project Structure

workspace/
├── datasets/                    # Sample CSV datasets (pre-loaded)
│   ├── sales_transactions.csv   # 102 rows
│   ├── customers.csv            # 101 rows
│   └── products.csv             # 102 rows
├── exercises/                   # 20 guided exercises (markdown)
├── scripts/                     # 20 runnable Python exercise scripts
│   ├── common/                  # Shared utilities (paths, output, checks)
│   ├── exercise_01_sql_fundamentals.py
│   ├── exercise_02_spark_transforms.py
│   ├── ...
│   ├── exercise_19_end_to_end.py
│   └── registry.json            # Exercise metadata & categories
├── pipelines/                   # ETL scripts (sample_etl.py, etl_spark.py, create_delta.py)
├── samples/                     # Enterprise sample projects
│   ├── dbt-project/             # dbt with DuckDB
│   ├── data-quality-suite/      # Great Expectations
│   └── fabric-lakehouse/        # Microsoft Fabric integration
├── output/                      # Pipeline output (generated by exercises)
│   ├── exercise_01/             # Per-exercise output artifacts
│   ├── exercise_02/
│   └── ...
├── .env.dev                     # Dev environment config
├── .env.staging                 # Staging environment config
├── .env.production              # Production environment config
├── .sqlfluff                    # SQL linter config
├── Mastery.md                   # Learning guide & standards
└── README.md                    # Quick reference

Virtual Environments

Cloud Storage Pattern (MinIO as S3)

MinIO mimics AWS S3 / Azure Blob / GCP Cloud Storage locally:

# Same as `aws s3` commands
mc ls local/                     # List buckets
mc cp file local/raw-zone/       # Upload to S3
mc ls local/raw-zone/ --recursive

# Python (boto3) — identical to real AWS S3
import boto3
s3 = boto3.client("s3", endpoint_url="http://localhost:9000",
    aws_access_key_id="minioadmin", aws_secret_access_key="minioadmin")

# PySpark reads from S3
spark.read.csv("s3a://raw-zone/sales/sales_transactions.csv")

# DuckDB reads from S3
SELECT * FROM read_csv_auto('s3://raw-zone/sales/sales_transactions.csv');

Enterprise Mapping

Security

• Root access restricted to: apt, apt-get, pip, python3
• Python packages: use pip install <pkg> (no sudo needed, venv is writable)
• System packages: use sudo apt install <pkg>
• Superset runs in isolated venv (/opt/superset-venv)

Ports Exposed

VS Code Extensions (Pre-installed)

Tips

• data-stack-up seeds MinIO with sample data — you can query it via boto3, PySpark, or DuckDB immediately
• data-demo --verbose shows full error output for any failing stages
• DuckDB queries CSV/Parquet/JSON directly without loading: SELECT * FROM read_csv_auto('file.csv')
• Use Polars instead of pandas for datasets > 1M rows
• Airflow DAGs auto-discover from ~/airflow/dags/ — just save a .py file
• data-run with no args shows exercise completion progress
• Type opencode for AI-powered coding assistance