Data Concepts

Streaming and Real-Time Data

A batch job sees the whole day before it answers. A stream has to answer now, so the hard part isn't throughput, it's deciding when a window is finished.

Events land out of order on an event-time line, and the watermark trailing behind the newest event is what decides when the window is allowed to answer

Why Does This Exist?

A batch job runs against a finished day. Midnight passes, the table stops changing, and the job sees every row before it computes anything. Order doesn't matter. Completeness is free.

A stream gets neither. Events turn up one at a time, forever, roughly in the order they happened but not exactly, and something downstream wants a number now.

Scope, before you go further, because the word covers a lot of ground: what follows is streaming as it changes a feature value. Not partitions, not consumer groups, not how to size a cluster. The question is what logins_in_the_last_5_minutes means when the last five minutes haven't finished arriving.

Throughput isn't where this gets hard. Machines are quick and the tooling is good. The hard part is deciding when a window is finished, because nothing in an unbounded sequence ever announces that the rest of it isn't coming.

Think of It Like This

Declaring a result while boxes are still in traffic

A count for a local election. Every ballot was cast at some moment during the day, and every ballot box reaches the counting hall at some later moment. The two aren't the same, and one box is always stuck behind a lorry.

The returning officer has a problem with no clean answer. Declare at 10pm and the box in traffic doesn't count. Wait until every box is definitely in and there's no deadline at all, because "definitely" never arrives.

So they pick a rule and say it out loud: boxes received after 10pm aren't counted. That rule is the whole idea. It turns an open-ended wait into a fixed one, and it puts the cost where people can see it. Move the deadline later and the result is more complete and slower. Move it earlier and you get a number fast, knowing a box might have turned it over.

How It Actually Works

Two clocks

An event happened at 10:00:03. It reached your stream at 10:04:11. Event time is the first, processing time the second, and every hard part below grows out of that gap.

A feature keyed on arrival is a different feature from one keyed on occurrence. Worse, it's an unstable one: replay the same log tomorrow and every processing time changes while every event time stays put. Event time is the only clock you can reproduce, so it's the only clock a training set can be built on.

Events arrive out of order because reality is out of order. A phone in a lift buffers and flushes twenty minutes later. A failed write gets retried. One region's collector falls behind. None of that is exotic, and none of it is fixable upstream.

Windows

You can't aggregate over "all of it", so you cut the stream into windows.

Tumbling. Fixed length, no overlap, every event in exactly one window. Five-minute counts, hourly sums.
Sliding. Fixed length, advancing by a smaller step, so a 5-minute window every minute puts each event in five of them. Smoother, and five times the work.
Session. No fixed length at all. It opens on an event and closes after a gap of inactivity you choose, so 30 minutes of silence ends it. The length is a property of the data rather than a parameter you set.

Watermarks

A watermark is a claim about completeness: no event with an event time older than T will arrive. That's the load-bearing idea on this page, because it turns an unbounded wait into a bounded one. When the watermark passes a window's end, the window emits.

The usual construction is the newest event time seen so far, minus an allowed lateness you choose. Choosing it prices the trade out loud.

Late watermark. High latency, and answers that survive contact with the tail.
Early watermark. Low latency, and answers a late event would have changed.

No setting gives you both. What you pick is which one you'd rather explain.

What happens to a late event

Three policies, and you're on one of them whether you chose it or not.

Drop it. Cheapest, and the default nearly everywhere. Your counts run low.
Emit, then correct. Publish the window, republish it when a late event lands. Needs a consumer that can take a retraction, not a store that only appends.
Hold a grace period. Keep the window's state alive for a fixed span past the watermark, then close for good. Memory in exchange for accuracy.

Delivery semantics, where they touch a number

At-least-once delivery means an event can be handed to you twice. For a log, harmless. For failed_payments_in_last_hour, it's a feature that reads 4 when the truth is 3.

Two fixes, both about making a repeat a no-op: deduplicate on an event id carried in the payload, or make the update idempotent so applying it twice lands in the same place. That's the part of the exactly-once argument that reaches a feature value.

One window, defined once

A streaming feature almost always has a batch twin. The stream computes orders_in_prior_30d for serving and a warehouse job computes it for training, and if the two disagree your model trains on one number and predicts on another. That's online/offline skew, and it happens over small things: different boundaries, one path on event time and one on arrival, a different late-arrival policy. Define the window logic once and have both paths call it.

Backfill is the other half. A stream can't be re-run over last year, so history comes from somewhere else, and that somewhere decides what you can reproduce. A replayable log rebuilds the exact values a given day would have produced. A table that's been updated in place since then can't, and nothing warns you. See data versioning.

Worked example

One 5-minute tumbling window, 10:00:00 to 10:05:00. Four events belong in it: 10:00:40, 10:02:00, 10:03:30 and 10:04:20. Three arrived within seconds. The 10:03:30 one went missing and turned up at 10:07:30, four minutes late. Meanwhile a 10:07:00 event arrived at 10:07:04, and a 10:10:02 event at 10:10:04.

Allowed lateness of 90 seconds. The 10:07:00 event pushes the newest event time to 10:07:00, so the watermark moves to 10:05:30, past the window's end. The window fires at 10:07:04 with 3. The late event lands 26 seconds after that, with nothing left to add it to.

Allowed lateness of 5 minutes. The watermark only reaches 10:05:00 once the newest event time hits 10:10:00, so the window fires at 10:10:04 with 4.

Three minutes of extra latency, exactly, bought one event. And the fast setting isn't wrong at random: it's low by 1 in 4 here, and it'll be low on every window with a tail.

Show Me the Code

1import numpy as np2
3event: np.ndarray = np.array([40, 120, 210, 260, 420, 602])    # seconds past 10:00:00, when it happened4arrival: np.ndarray = np.array([45, 124, 450, 266, 424, 604])  # when it reached the stream; row 2 is late5order: np.ndarray = np.argsort(arrival)                        # a stream only ever sees arrival order6newest: np.ndarray = np.maximum.accumulate(event[order])       # the newest event time so far7
8
9def tumbling_count(lateness: int, window_end: int = 300) -> tuple[int, int]:10    watermark = newest - lateness              # "nothing older than this is still coming"11    fires = int(np.flatnonzero(watermark >= window_end)[0])12    fired_at = int(arrival[order][fires])13    counted = (event < window_end) & (arrival <= fired_at)14    return int(counted.sum()), fired_at15
16print(tumbling_count(90))    # -> (3, 424)   emits 10:07:04, and the late event has nowhere to go17print(tumbling_count(300))   # -> (4, 604)   emits 10:10:04, correct, 180 seconds slower

Same events, same window, same code. The only thing that moved was one constant.

Watch Out For

Features keyed on processing time, because that timestamp is the one you already have

ingested_at is sitting right there. The event's own timestamp needs parsing, might be in a local zone, and is occasionally rubbish. So the window gets built on arrival, and everything works.

Then you rebuild the training set. Every event now arrives at a different moment than it did the first time, so every window holds a different set of events, so every feature value differs from the one served in production. Your training set isn't reproducible, and it never will be, because the input that defined it was the clock on the machine.

Nothing throws. The tell is that two runs over the identical log give different features. Do that comparison deliberately, once, and key on event time even when the field is awkward. See time series data.

A watermark tuned for latency, quietly shaving the tail off every window

Someone wanted the dashboard fresher, so allowed lateness went from five minutes to thirty seconds. Latency improved. Nothing broke.

Every window now closes before its slowest events land, so every count runs low. Not noisily, and not in both directions: the bias is one-way and roughly constant, a couple of percent for as long as the setting stands. Nobody sees an error, because dropping a late event is normal behaviour working as configured.

The same shape shows up from the other side under at-least-once delivery, where a redelivered event gets counted twice and the number runs high instead. Both are invisible for the same reason: no exception, no null, no gap.

Instrument it. Count dropped-late events and emit that as a metric next to the feature, watch the distribution of arrival delay rather than guessing at it, and set allowed lateness from its 99th percentile. Then a change to the setting shows up as a number instead of a slow drift in a count. See data validation and contracts.

The Quick Version

A stream answers from an unbounded, out-of-order sequence. The hard part isn't throughput, it's knowing when a window is done.
Event time is when it happened, processing time is when it arrived, and only event time survives a replay. Key features on it.
Tumbling windows partition, sliding windows overlap, session windows end on a gap of inactivity you pick.
A watermark claims nothing older than T is still coming, which bounds the wait. Late means high latency and correct answers, early means low latency and revisable ones.
Late events get dropped, corrected, or held in a grace period. Pick deliberately.
At-least-once delivery double-counts, so a running count needs an event id to deduplicate on or an idempotent update.
The streaming feature and its batch twin must agree, so define the window once and call it from both. A replayable log is what makes backfill reproducible.
One window, two watermark settings: 3 versus 4, three minutes apart. The fast one is low every time, not sometimes.