Live Demo: Wikipedia

Every change to every Wikipedia, and to every other Wikimedia wiki, goes out on a public stream the moment it is made. This demo connects your browser to that stream and hands each change to the Precomputing Engine, running in the page. The Engine keeps the answers ready in an SQLite file as the changes arrive, from how many come in a second to what people are editing right now. A recount kept apart from the Engine checks every count in the file as it goes.

Real changes, real code. Unlike the other demos, the data here is real and live, straight from Wikimedia’s stream to your browser. If the stream can’t be reached from your network, the demo offers simulated changes of the same shape instead, marked as simulated.

How to Run It

  1. Press Connect to Wikipedia. Changes start to arrive once the connection opens.
  2. Watch what people are editing. Articles changed by two or more people in about the last ten minutes, most people first, with bots left out.
  3. Pick a wiki. The chart and the lists switch at once, since every wiki’s answers are already in the file.
  4. Pause and ask the file. Ask it anything in SQL, then download it if you like.

On a wide screen the demo can use the whole window: open it full screen.

The Controls

Control What it does
Connect to Wikipedia, Pause, Reconnect Opens the stream, closes it and opens it again. Changes made while it is closed are not received
Watch simulated changes Invented changes in the shape of the real stream, for when it can’t be reached. Switching between the two starts a new file
Clear Starts a new, empty file
Chart and lists for Every wiki, every Wikipedia, or one wiki, for the chart and the lists. The counters always cover every wiki
Check every count now Checks every count and sum in the file against the recount at once. The regular checks look at the ones that could have changed since the last
Run Runs your SQL against the file
Download the file Saves the file as it is, to open in any SQLite tool

The Stream

Wikimedia publishes its recent changes at stream.wikimedia.org as server-sent events, one JSON object per change, following a published schema. The stream carries every edit, new page, log entry and category change, usually tens of them a second. From each change the demo reads:

  • When: the time of the change on the wiki.
  • Where: the wiki’s address, such as en.wikipedia.org, and the page with its namespace.
  • What: the kind of change and how much longer the page got.
  • Who: whether a bot made it, and a number made from the user name and a random value picked when the page opened.

It skips the stream’s own test events, and a change that arrives twice counts once, by its id. No user name or edit summary is shown or kept, so the file never holds a name.

The Policy

The stream edits takes every change. Changes to articles also go into article_edits, and the ones made by people into article_editors, which counts who made them.

stream edits {
  key    wiki text
  key    kind text
  key    who text
  value  bytes integer
  raw    keep 10m
  rollup 10s keep 1h
  rollup 1m  keep 7d
  rollup 1h  keep 1y
}

stream article_edits {
  key    wiki text
  key    title text
  value  bytes integer
  rollup 1m keep 1h
}

stream article_editors {
  key    wiki text
  key    title text
  key    editor integer
  value  bytes integer
  rollup 1m keep 1h
}

precompute changes     = count(edits)
precompute changes_day = count(edits) by wiki per day
precompute growth_day  = sum(edits.bytes) by wiki per day

An article is a page in a wiki’s main namespace, changed by an edit or created as a new page. What people are editing is one query over article_editors: the people on each article in about the last ten minutes, counted distinct.

What the Numbers Mean

  • Changes counted: every change the Engine took in since the page connected, test events and repeats left out.
  • Changes a second: every change in the last minute of whole 10-second windows, or in as much of it as there is.
  • Made by bots: the share of changes in about the last ten minutes that the stream marks as made by a bot.
  • Counts checked: every 30 seconds the recount checks each count and sum in the file that could have changed since the last check. Check every count now checks them all.
  • The file: the size of the Engine’s SQLite file in the page. It grows for the first hour, then the minute windows of articles start to fade.

The Checks Behind It

Live changes can’t be replayed, so the same code was also run headless in Node (tools/run-demo5.mjs) on 75 minutes of simulated changes, with repeats, lines that don’t read and changes that arrive late mixed in:

What Result
The changes 112,985 counted; 96 test events skipped, 1,096 repeats dropped, 140 lines that don’t read set aside
Checks during the run 150, one every 30 simulated seconds, 0 differences
The full check at the end 99,992 of 99,992 counts and sums equal the recount
The native Engine precomputing put --format json took the same changes and wrote the same file: 119,668 rows, 1,596,566 values, 0 differences

If It Does Not Start

The demo works in any current Chrome, Edge, Firefox or Safari, on a computer or a phone, with JavaScript switched on. It downloads SQLite (1.5 MB, 577 KB compressed) and the Engine (5.0 MB, 1.3 MB compressed) from this site, then connects to stream.wikimedia.org. Some networks, and some pages that embed other pages, don’t allow that connection. The demo then says it can’t reach the stream, and Watch simulated changes still shows how it works.

What Is Real Here

Real:

  • The changes, once connected: straight from Wikimedia’s stream to your browser, soon after they are made.
  • The Engine and SQLite: the same Go code as precomputing put, compiled to WebAssembly.
  • The answers: every number on the page is read from the Engine’s file.
  • The checks: a recount kept apart from the Engine, which never reads the file.

Simulated, only when you choose it:

  • Watch simulated changes: invented changes to invented pages, with a few people editing one article now and then.

The Wikipedia live post tells the story behind the demo.