| entry | source |
|---|---|
| 01-npm-registry | 786 Kb |
| 02-hn-thread | 193 Kb |
| 03-natural-earth | 4 Mb |
| 04-gharchive | fetch.sh |
| 05-fhir-bundle | 2 Mb |
| 06-espn-qbr | 176 Kb |
| 07-graphql-introspection | 140 Kb |
| 08-open-meteo | 12 Kb |
| 09-stripe-openapi | fetch.sh |
| 10-wikidata | 1 Mb |
| 11-jupyter-notebook | 1 Mb |
| 12-agent-trace | fetch.sh |
| 13-package-lock | 759 Kb |
| 14-nyc-311 | fetch.sh |
| 15-github-issues | 686 Kb |
| 16-movie-ratings | 7 Kb |
| 17-openlibrary | 64 Kb |
| 18-openfda-events | 3 Mb |
| 19-chicago-salaries | 923 Kb |
| 20-homebrew-formulae | fetch.sh |
| 21-crossref-works | fetch.sh |
| 22-dockerhub-tags | 476 Kb |
| 23-cratesio-summary | 41 Kb |
| 24-cargo-metadata | 27 Kb |
| 25-usgs-quakes | fetch.sh |
| 26-gharchive-scale | fetch.sh |
| 27-grafana-dashboard | 667 Kb |
| 28-home-assistant-i18n | 590 Kb |
| 29-mdn-browser-compat | fetch.sh |
| 30-aws-redshift-pricing | 4 Mb |
4 The corpus
Real files only. Toy JSON is hard in ways nobody suffers from, and a tool tuned against invented documents is tuned against invented problems. Every entry is something that came off a real API, a real export or a real tool.
An entry is finished only when all of these are true, and anything less is a file somebody downloaded:
source.jsonis committed, or afetch.shis, with the file described as it was on the day.- Its notes carry provenance with a date, a measured number in every grading row that can carry one, and a section saying what the file disconfirmed.
- One attempt file per tool, in both languages, each with its scoring header filled in.
- An entry in the findings saying what the file changed.
4.1 The disconfirmation rule is the point
That third bullet in the notes — what this file disconfirmed — is why the corpus is worth having.
The first file was chosen as a polymorphism specimen and has none. That is worth more than a file which confirmed what somebody already believed, and it is only worth anything because the expectation was written down first.
Each of the first three files contradicted something. A later one was chosen specifically as the case where an existing describer should finally be right, and it dropped the field held by the majority of records. Another was chosen because two spellings of one field are its whole point, and a database engine silently collided them and returned the other population’s data.
4.2 Why a corpus and not a benchmark
A benchmark asks which tool is fastest at a task everyone agrees on. There is no agreed task here — that is the problem. The corpus exists to find out whether there is a capturable pattern underneath, by asking the same questions of documents that are as unlike each other as could be found.
Gaps are tracked as deliberately as entries. The one that matters most is scale: nothing here has ever strained memory, so the axis has never had a chance to separate anything.