Live Demo: Incident Déjà Vu

When something breaks, the first useful question is whether it has happened before, and what fixed it last time. In this demo the invented web shop from the Logs demo carries on for two more hours. Its file already holds the day before: every kind of log line counted a minute at a time, and six incidents the on-call team wrote down. Every ten seconds the page asks the file what stands out right now and which of those incidents it looks like.

Simulated logs, real code. The shop and its log lines follow a fixed script, incidents included. The log reducer and the Engine are the real Go program built for WebAssembly, and the comparison is plain SQL kept in the same SQLite file.

How to Run It

1. Press Play. Two hours pass in about a minute and a half, slowing down as each incident begins.
2. Watch Right now. A kind of incident the file knows is named within seconds, with what fixed it last time. A kind it has never seen is flagged as new.
3. Start one of your own. Pick a kind and press Start it now. Write a new one down, start it again, and the file recognizes it.
4. Review the results. The two hours finish, and every count in the file is checked against a recount of the lines.

On a wide screen the demo can use the whole window: open it full screen.

The Controls

The Shop’s Day Before

The file the demo starts from covers 24 hours of the shop, from 29 September 2026 at 12:00 to 30 September at 12:00 UTC: 2,133,322 log lines, 25 kinds of line. The same code that runs in the page made it, and making it again from the same lines gives the same bytes. Two deploys that log extra debug lines and a newsletter that brings more visitors are in there too, and none of them are incidents.

Today’s Two Hours

From 12:00 to 14:00 UTC on 30 September, five incidents happen on schedule:

• 12:18. Card payments start failing at northpay, for eight minutes.
• 12:52. The checkout database runs out of connections, for seven minutes.
• 13:14. The certificate on an image server expires, and product images start failing. The file has never seen this kind. At 13:22 the on-call team writes it down.
• 13:34. A login attack, for five minutes.
• 13:48. The certificate on a second image server expires.

How the File Tells

The policy counts every kind of log line by service and level, a minute at a time for a week and an hour at a time for 90 days, and keeps every line whole for ten minutes:

stream lines from logs {
  key    service text
  key    level text
  key    template integer
  raw    keep 10m
  rollup 1m keep 7d
  rollup 1h keep 90d
}

The comparison is plain SQL beside the policy’s tables, in the same file:

• What stands out. A kind of line stands out when its rate over the last minute or two is at least three times its usual minute, higher or lower, and at least six lines a minute away. Usual is the median minute of the hour before, so an earlier incident in that hour doesn’t move it. A kind of line going quiet must stay quiet over one minute more, since quiet takes longer to be sure of. DEBUG lines are left out.
• The fingerprint. Each kind of line that stands out gets a score, the logarithm of how far it moved. A new error from nothing scores high. A line that went quiet scores below zero.
• Writing it down. Saving an incident saves the fingerprint of its first two minutes beside it, with the newest raw line of each kind in it while the raw lines are still kept. A trigger does it on the insert. The saved fingerprint outlives the week the minute counts are kept.
• The comparison. The view deja_vu compares the fingerprint of now with every saved one by the cosine of their scores. At 0.7 or more, now looks like that incident. At 0.4 or more, it is partly like it.
• Two that look the same. The northpay and quickcard outages leave the same kinds of line, so their fingerprints match equally well. The page then compares the words those lines hide, in the newest raw line against the example saved with each incident. provider=northpay decides.

Things to Try

1. Let the two hours finish untouched. The run ends with all five incidents right. The three kinds from the day before were named 8 to 10 seconds after they began. The new kind was flagged as new after 5 seconds and, once written down, its return was named 10 seconds in. Nothing stood out outside an incident in 721 checks, and all 1,661 counts by minute and hour equal a recount of the lines.
2. Watch 12:18. Within seconds Right now says it looks like #1, with what fixed it, and that the raw lines tell it from #6: provider=northpay.
3. Start the order queue incident. The file has never seen it, so the page offers to write it down. Write down what it was and what fixed it, wait for it to pass, then start it again.
4. Start two at once. Overlapping incidents mix their fingerprints. See what the file makes of it.
5. Ask the file. SELECT * FROM deja_vu while something stands out, or Every fingerprint in the file.

What the Numbers Mean

• Stood out after: seconds from an incident’s first line to the first check where something stood out. A check runs every ten seconds, after the Engine writes the file, so this is at most ten seconds late.
• Similarity: the cosine of two fingerprints. 1 means the same kinds of line moved in the same proportions, and 0 means they share none.
• A check takes: the time to read what stands out and the closest incidents from the file, both SQL views. On our test machine, 4 to 5 ms on average in Chromium, at most 35 ms.
• Counts checked: the lines a minute and an hour for each service and level, from the file, against a recount of the lines by separate code that never reads it.

Measured Beyond the Script

The same code runs headless in Node (tools/run-demo6.mjs), which is how these were measured:

If It Does Not Start

The demo works in any current Chrome, Edge, Firefox or Safari, on a computer or a phone, with JavaScript switched on. It downloads SQLite (1.5 MB, 577 KB compressed) and the reducer with the Engine (5.0 MB, 1.3 MB compressed) from this site, then the file of the day before (3.7 MB, 0.73 MB compressed). If the demo says it could not start, try another browser, or allow scripts on this page if an extension blocks them.

What Is Real Here

Real:

• The log reducer, the template miner and the Engine: the same Go code as precomputing put --lines, compiled to WebAssembly.
• The comparison: views and a trigger in plain SQL, in the same SQLite file as the counts.
• The day before: made from its 24 hours of lines by the same code, and the same bytes when made again.
• The checks: separate code recounts the lines and never reads the file.

Simulated:

• The shop, its incidents and what the on-call team wrote down.
• Time: two hours pass in about a minute and a half.

The incident déjà vu case study follows the two hours.