Live Demo: Logs

Two hours of an invented web shop’s logs run through the Precomputing log reducer, right here in your browser. The reducer learns each kind of line as it arrives, keeps every line in a local file, and sends upstream only what a six-panel dashboard needs, plus the errors and anything new. The dashboard is then drawn from what was sent alone, and every point is checked against the raw lines.

Simulated logs, real reducer. The shop and its log lines follow a fixed script. The reducer, the template miner and the Engine are the real Go program, compiled to WebAssembly, writing into SQLite’s own WebAssembly build.

How to Run It

1. Press Play. Five services start writing log lines, about 30 a second. Two hours pass in about a minute, slowing down when something happens.
2. Break something. Break a search shard, or turn the rate up, and watch the templates, the alerts and the gap between raw and sent.
3. Review the results. The two hours finish and every panel is checked against a recount of the raw lines.
4. Search and ask. Search the lines kept on site, or ask the file in SQL, then download it.

On a wide screen the demo can use the whole window: open it full screen.

The Controls

The Shop

An invented outdoor gear shop on 29 September 2026, from 12:00 to 14:00 UTC. Five services write plain log lines: the web server, search, checkout, payments and login. A line looks like this:

2026-09-29T12:00:00.600Z INFO  payments charge ok provider=northpay result=ok amount=76.73 ms=289

Two things happen on schedule:

• 12:40 to 12:48. The payment provider northpay fails. Payments log a new kind of error, and checkout logs failed orders.
• 13:05. A deploy makes search log three debug lines with every query. From then on the shop writes about 45% more lines.

The Dashboard and the Policy

The dashboard has six panels: requests a minute by route, p95 latency by route, errors a minute by service, payments by provider and result, revenue a minute, and top searches. The importer reads its definition and writes a policy with one stream from logs per panel. It always adds two more streams: every line kept on site for 48 hours and counted per template, and error lines kept whole, up to five a minute of each kind. Part of the result:

logs {
  format "<timestamp> <level> <service> <message>"
}

stream lines from logs {
  key    service text
  key    level text
  key    template integer
  raw    keep 48h
  rollup 1m keep 30d
  rollup 10m keep 30d
  samples 1 per 10m
}

# "p95 latency by route": p95 of ms
stream web_ms_by_route from logs where service = "web" {
  key    route text
  value  ms real
  rollup 1m keep 30d quantiles ms
  anomalies ms log z > 4 keep 5 per 1m
}

What Goes Upstream

Every minute, the rows of each panel’s 1-minute window, the error lines kept whole and any unusual requests. Every ten minutes, one example line of each template. A new template goes at the next checkpoint, at most five seconds after its first line, and a new kind of WARN or ERROR line is also an alert. Everything else stays in the file on site.

What the Numbers Mean

• Raw logs, kept on site: the bytes of every line written so far. All of them are in the local file for 48 hours.
• Sent upstream: the bytes of every batch sent so far, as JSON.
• Fewer bytes: raw divided by sent.
• Templates learned: kinds of line, each with <*> where its lines differ. Lines of different services or levels never share a template.
• What it costs: the same month of logs sent two ways, at Datadog’s list prices checked on 29 September 2026: $0.10 per GB ingested and $1.70 per million log events indexed for 15 days, billed annually. Everything sent upstream is counted as log events.

Things to Try

1. Let the two hours finish untouched. Press Play and wait, or Review the results at once. The run ends with 281,164 lines, 19 templates and 120 times fewer bytes sent, and every panel checked against the recount: counts and sums equal, p95 within 1%.
2. Watch 12:40. Within seconds a new ERROR template appears for payments, marked as new, and an alert goes upstream. The errors panel and the payments panel show the incident from what was sent alone.
3. Break a search shard. A new kind of error, caught at the next checkpoint. The alert says how many seconds it took.
4. Watch 13:05. The deploy adds a noisy DEBUG template. The raw bytes climb much faster than what is sent, because the dashboard never asked for debug lines.
5. Search the lines kept on site. Search for northpay to find every line that names the provider, or for status=503 to see only the failed charges. Nothing that went upstream was needed for this.
6. Change the dashboard. In Try the import, add a panel, for example the count of login lines grouped by method, and import it to see the stream it adds.

If It Does Not Start

The demo works in any current Chrome, Edge, Firefox or Safari, on a computer or a phone, with JavaScript switched on. It needs WebAssembly, which all of them support. It downloads SQLite (1.5 MB, 577 KB compressed) and the reducer with the Engine (5.0 MB, 1.3 MB compressed) from this site. If the demo says it could not start, try another browser, or allow scripts on this page if an extension blocks them.

In testing no check has failed. If one ever does, the results say so in red, and the project would be glad to hear about it through the contact page.

What Is Real Here

Real:

• The log reducer, the Drain template miner and the Engine: the same Go code as precomputing put --lines, compiled to WebAssembly.
• The dashboard import, the policy it writes and the SQL it compiles to.
• Every byte kept and sent, and the checks: separate code recounts each panel from the raw lines and never reads the file.

Simulated:

• The shop, its services, its customers’ searches and its incident.
• The upstream service. The demo draws the dashboard from what it would have sent, and does not send anything.
• Time: two hours pass in about a minute.

The web shop logs case study follows the two hours.