live data

Wikipedia Live: Every Change to Every Wiki, Counted in the Browser

Wikipedia never stops changing. Every edit, new page, log entry and category change on every Wikipedia, and on Wikidata, Commons and the other Wikimedia wikis, goes out on a public stream the moment it is made, usually tens of changes a second. That makes the stream a good test for keeping answers ready: the data is real and never pauses. Anyone can watch it.

The Wikipedia demo connects your browser straight to that stream. Each change goes into the Precomputing Engine, running in the page, which keeps its answers in an SQLite file as the changes arrive. The page draws everything from that file, once a second, and a recount kept apart from the Engine checks every count in it as it goes.

Every change to every Wikimedia wiki, counted in the page. The stream at stream.wikimedia.org sends every change as it is made. In the browser, the page reads each change for when, which wiki, what kind, bot or person, which page and how much it grew, drops test events and repeats, and the Engine writes the file every second. One SQLite file holds three streams: edits, article_edits and article_editors. Read from the file every second: changes a second, what people are editing, the busiest wikis, bots and people, and any question in SQL.

From the stream to the answers, all inside the page.

The Stream

Wikimedia publishes its recent changes at stream.wikimedia.org as server-sent events, one JSON object per change, following a published schema. The browser opens it with the standard EventSource interface, so nothing sits in between. From each change the page reads the time, the wiki, the page and its namespace, the kind of change, whether a bot made it and how much longer the page got. It skips the stream’s own test events, and a change that arrives twice counts once, by its id.

What the File Keeps

The policy has three streams. edits takes every change, keyed by wiki, kind, and bot or person. Its windows run from 10 seconds, kept for an hour, up to an hour, kept for a year. Changes to articles also go into article_edits, one row per article per minute, for an hour. Those made by people go into article_editors as well, keyed by article and by a number made from the user name.

stream article_editors {
  key    wiki text
  key    title text
  key    editor integer
  value  bytes integer
  rollup 1m keep 1h
}

That last stream answers the most interesting question on the page, what people are editing right now. It is one query: the distinct people on each article over about the last ten minutes, bots left out, most people first. A burst of editors on one article often means something just happened in the world.

What You See

  • Changes a second, over the last minute, and the share made by bots over about ten minutes.
  • A chart of changes every ten seconds, by people and by bots, for up to half an hour.
  • What people are editing: articles changed by two or more people in about the last ten minutes, each linked to its page.
  • The busiest wikis and the kinds of change, from edits to category changes.

A menu narrows the chart and the lists to the Wikipedias or to a single wiki. The switch is instant, because every wiki’s answers are already in the file.

The Checks

Beside the Engine, the page keeps its own recount of every change it hands over, window by window. Every 30 seconds it compares each count and sum in the file that could have changed, and a button checks them all. On the live stream these checks run in your browser, on real data.

Live changes can’t be replayed, so the published figures come from the same code run headless in Node on 75 minutes of simulated changes, with repeats and unreadable lines mixed in, and some changes arriving late. It counted 112,985 changes, after dropping 1,096 repeats and 96 test events. All 150 checks during the run found no difference, and the full check at the end found 99,992 of 99,992 counts and sums equal. The native Engine, given the same changes with precomputing put --format json, wrote the same file: 1,596,566 values, no differences.

Privacy

The page shows no user names and no edit summaries, and the file keeps neither. People are counted by a number made from the user name and a random value chosen each time the page opens, so the numbers mean nothing outside that visit. Nothing the page receives is sent anywhere.

What It Shows, and What It Doesn’t

The demo shows the Engine keeping up with a real public stream in a browser tab, answering from a file that stays small because detail fades: after an hour the article windows start to go, while the counts by wiki stay. It doesn’t show a team’s own traffic, and the stream is modest next to a busy production system. Some networks block the connection to the stream, and the page then offers simulated changes of the same shape, clearly marked.

Try It

Open the Wikipedia demo and press Connect to Wikipedia. The chart fills in as changes arrive, and articles with several people on them show up as it happens. Pick a single wiki, then pause and ask the file anything in SQL, or download it and open it in any SQLite tool.