Batch and stream processing

Stream versus batch, windowing and watermarks, late and out-of-order data, exactly-once counting, approximate algorithms, and the lambda/kappa argument in one paragraph.

Analytics, counters, aggregations, recommendations, ETAs and abuse detection all end up in the same place: a pipeline that turns a firehose of events into numbers people can query. The interview questions are always about time, duplicates and late data.

Batch or stream

The honest default: stream for anything a user sees within a minute, batch for anything that must be exactly right. View counts and trending topics stream; billing, payouts and financial reporting batch.

That is also the whole lambda-versus-kappa argument. Lambda runs both — a fast approximate stream layer and a slow correct batch layer that overwrites it. Kappa runs only the stream and reprocesses from the retained log when it needs to fix something. Kappa is the modern default because a replayable log makes the batch layer redundant; lambda still appears where the batch system is the system of record (a warehouse, a ledger).

Event time versus processing time

Every event has at least two timestamps: when it happened on the device (event time) and when your pipeline saw it (processing time). They differ by milliseconds normally and by hours when a phone was offline in a tunnel.

Aggregate by event time. Processing-time aggregation is trivially easy and produces numbers that move when your consumer lags, which makes them useless for comparing days.

Windows

• Tumbling: fixed, non-overlapping (every minute). The default for counters.
• Hopping / sliding: fixed size, advancing by a smaller step (5-minute window every minute). Used for trends and rate alerts.
• Session: closes after a gap of inactivity. Used for user sessions and trips.

State the window size, because it determines how much state the job keeps: windows in flight × keys × bytes per aggregate. That product is the memory footprint people forget.

Watermarks and late data

A watermark is the pipeline's claim that it has seen every event with an event time before T. It is usually computed as the maximum event time seen minus an allowed lateness (say 30 seconds). When the watermark passes the end of a window, the window is closed and emitted.

Then there are three choices for events that arrive after that:

1. Drop them, and report how many you dropped — an actual metric people alert on.
2. Allow late updates for a grace period and re-emit the corrected aggregate; downstream must accept updates, which means the sink is keyed and upsertable.
3. Send to a side output and fix in a nightly batch job.

Say which one you picked; "we handle late data" without naming the mechanism is the answer that invites the follow-up.

Exactly-once, honestly

There is no exactly-once delivery. There is exactly-once effect, and it comes from one of three things:

• Idempotent writes: the sink is keyed by (window, key) and the write is an upsert, so reprocessing the same events produces the same row. This is the practical answer for aggregation.
• Transactional sink: offsets and output commit atomically (Kafka transactions, a database that stores both the aggregate and the offset in one transaction).
• Deduplication with an id set over a bounded time window — the sink keeps event ids seen in the last N minutes.

Checkpointing is what makes recovery work: the job periodically snapshots its operator state plus the input offsets; on restart it restores both and replays from the checkpointed offset. Without aligned checkpoints, recovery double-counts.

Approximate algorithms

When exactness is not required, approximate structures cut memory by orders of magnitude, and naming them is a cheap way to sound experienced:

HyperLogLog unions cleanly, which is why "unique users per day" can be computed per shard per hour and merged for any range.

Backfills and reprocessing

Every pipeline needs an answer for "the aggregation had a bug for three days":

• Keep raw events in the log (7–30 days) and in cold storage (indefinitely), so reprocessing is possible at all.
• Make the job versioned and the output keyed so a rerun overwrites rather than appends.
• Run the corrected job into a shadow table, compare, then swap.
• Watch the cost: replaying a month of a large stream is a real compute bill and can saturate the sink.

Sizing

• Pipeline throughput is bound by the slowest stage; state size and shuffles usually break first, not CPU.
• One minute of a 100 k events/s stream at 500 bytes is 3 GB; a day is 4.3 TB. That is why raw events go to object storage and aggregates go to the serving store.
• Lag, in seconds and in events, is the single health metric for a streaming job.

Serving the results

Aggregates land in a store shaped for the query: a wide-column or time-series store keyed by (entity, time bucket), a columnar warehouse for ad-hoc analysis, or a cache for the handful of counters shown on every page. Pre-aggregate the rollups the product actually shows (minute, hour, day) rather than computing them at read time.

Common mistakes

• Aggregating by processing time and wondering why yesterday's numbers changed.
• No watermark policy, so the job either waits forever or drops silently.
• Counting with INCR on a single hot key instead of sharded counters merged on read.
• Treating the streaming job as the system of record with a three-day retention behind it.
• Forgetting that a schema change to the event breaks every consumer; version the event, add fields, never repurpose one.

Checklist

• Stream or batch, and the latency the product needs.
• Event time, window type and size, watermark and late-data policy.
• How exactly-once effect is achieved at the sink.
• Retention for replay, and the reprocessing procedure.
• Lag metric and its alert threshold.