Observability-Driven Testing Environment

Enterprise-grade observability testing with OpenTelemetry, Jaeger, Prometheus, Grafana, and Tracetest.

What's Inside

Quick Start

# Start the observability stack (Jaeger + Prometheus + Grafana)
odt-stack-up

# Start the instrumented demo app
odt-demo-up

# Run trace-based tests
odt-test

# Validate SLOs against Prometheus
odt-slo

# Run k6 load test with trace propagation
odt-load

# Detect latency regressions
odt-regression

Helper Commands

Accessing Services

Each service is exposed via a subdomain on your workspace URL. After running odt-stack-up, access them from the preview panel or directly:

Note: The exact URLs are shown in the Preview panel on the right side of your workspace. Click any preview URL to open the service in a new tab.

Exercises

1. Instrument Your First App — Add OpenTelemetry tracing, export to Jaeger, view in Grafana
2. Trace-Based Testing — Assert on span attributes and trace structure
3. SLO Validation — Define SLOs in YAML, validate against Prometheus, confirm in Grafana
4. Latency Regression Detection — Define baselines, detect regressions, visualize spikes
5. Load Testing with Observability — Run k6 while watching Grafana dashboards live
6. Custom Metrics & Alerts — Add Prometheus metrics, write PromQL alert rules
7. Distributed Tracing Across Services — Propagate context, validate end-to-end traces
8. CI/CD Integration — Run observability tests in pipelines, fail on SLO violations
9. Incident Simulation & Root Cause Analysis — Inject chaos (latency/errors/db-failure), diagnose with traces

Project Structure

workspace/
├── app/                    # Instrumented demo application
│   └── demo_app.py        # Flask + OpenTelemetry + Prometheus metrics
├── tests/                  # Trace-based test suites
│   ├── test_traces.py     # Jaeger trace assertions
│   ├── test_slo.py        # SLO validation against Prometheus
│   └── test_regression.py # Latency regression detection
├── slos/                   # SLO definitions
│   └── slo_definitions.yaml
├── load/                   # Load test scripts
│   └── k6_traces.js       # k6 with trace propagation
├── config/                 # Prometheus + OTel config
│   └── prometheus.yml
├── tracetest/              # Tracetest definitions
│   └── trace_test.yaml
├── reports/                # Test outputs
└── README.md

Grafana Dashboard

The workspace comes with a pre-provisioned Observability Overview dashboard (no login required). After running odt-stack-up, open the Grafana preview URL to see:

• Request Rate — Total requests per second over time
• Error Rate — Percentage of failed requests with threshold indicators
• Latency P50 — Median response time in milliseconds
• Latency P95 — 95th percentile response time
• Latency P99 — 99th percentile response time
• Requests by Endpoint — Breakdown of traffic per API route

Demo App Endpoints

The instrumented Flask app exposes these endpoints with full OpenTelemetry tracing:

Tips

• Always run odt-stack-up before odt-demo-up — the app needs Jaeger and Prometheus running
• Grafana dashboards update in real-time — open them before running load tests
• Use odt-traces to quickly query Jaeger traces from the terminal
• SLO definitions are in slos/slo_definitions.yaml — add your own and run odt-slo
• The /slow endpoint is intentionally slow for regression testing exercises
• Use Ctrl+Shift+B keyboard shortcut to start the observability stack
• Traces take 2-3 seconds to appear in Jaeger after a request
• All services communicate internally on localhost — the ingress subdomains proxy to the correct ports

Security

• Root access is restricted. Use sudo apt install <package> for system packages.
• pip, npm, python3, docker, apt, and apt-get are whitelisted for sudo access.