2026

Real-Time Crime Analytics - Lambda Architecture

End-to-end big-data system over Chicago open data: Spark recomputes batch views into PostgreSQL while Kafka and Storm detect district-level anomalies in a sliding window, both surfaced through one Streamlit dashboard.

Source datasets joined: 5
Orchestrated services: 7
Serving stores (SQL + document): 2

Repository

Real-Time Crime Analytics - Lambda Architecture cover

Stack

Apache Spark
Apache Kafka
Apache Storm
PostgreSQL
MongoDB
Streamlit
Docker
Python
Java
PySpark
K-Means
SQL

Problem

City-scale crime data is large, continuously updated, and asked both strategic questions (long-horizon trends, hotspots) and tactical ones (unusual bursts). A Lambda architecture answers both without one compromising the other.

Architecture

Spark batch job -> PostgreSQL analytics tables. Kafka replay -> Storm topology -> PostgreSQL alerts plus MongoDB alert documents. Streamlit reads both stores.

What it does

Batch layer in Spark: typed ingest, cleaning, time-trend aggregation, arrest-rate joins and K-Means geospatial hotspots written to PostgreSQL
Speed layer in Kafka and Storm: sliding-window counts per district with a threshold anomaly test
Dual serving stores, PostgreSQL for relational alert rows and MongoDB for raw alert documents
Streamlit dashboard reading both paths so batch and streaming appear as one system
Whole stack orchestrated by a single Docker Compose file: ZooKeeper, Kafka, Storm, Spark, PostgreSQL, MongoDB and the dashboard

Challenges & learnings

Explicit StructType schemas mean malformed CSV rows land in _corrupt_record instead of silently shifting columns, which is the failure mode that makes bad analytics look fine.
Alerts stayed empty against a live producer because the anomaly threshold was tuned for aggregate volume while traffic split across districts; the fix was a shorter window and a lower per-district threshold.
Resubmitting a Storm topology consumed nothing because the superseded instance still held the Kafka consumer group on a single partition.
The sex-offender extract has no district key, so rows are assigned deterministically by hashing the block. Documented as a relative signal rather than presented as a real geospatial join.

Real-Time Crime Analytics - Lambda Architecture

Source datasets joined

Orchestrated services

Serving stores (SQL + document)

What it does

Batch layer in Spark: typed ingest, cleaning, time-trend aggregation, arrest-rate joins and K-Means geospatial hotspots written to PostgreSQL
Speed layer in Kafka and Storm: sliding-window counts per district with a threshold anomaly test
Dual serving stores, PostgreSQL for relational alert rows and MongoDB for raw alert documents
Streamlit dashboard reading both paths so batch and streaming appear as one system
Whole stack orchestrated by a single Docker Compose file: ZooKeeper, Kafka, Storm, Spark, PostgreSQL, MongoDB and the dashboard

Challenges & learnings

Explicit StructType schemas mean malformed CSV rows land in _corrupt_record instead of silently shifting columns, which is the failure mode that makes bad analytics look fine.
Alerts stayed empty against a live producer because the anomaly threshold was tuned for aggregate volume while traffic split across districts; the fix was a shorter window and a lower per-district threshold.
Resubmitting a Storm topology consumed nothing because the superseded instance still held the Kafka consumer group on a single partition.
The sex-offender extract has no district key, so rows are assigned deterministically by hashing the block. Documented as a relative signal rather than presented as a real geospatial join.