7  What could one row be?

This half is ported and verified. The Rust core prices candidates identically to the prototype on every corpus file — including reproducing pandas.json_normalize, which was named as the port’s largest risk before any Rust was written. Parity has that story.

Rectangling is not the goal, and getting that wrong was the first correction this project made to itself.

A table is the shape of your answer, never the shape of the document.

An extract may be any shape — a value, a vector, a list. The rectangular ones are what flow onward into the next tool in the chain; non-rectangular extracts are terminal, for looking at or for feeding back into another call.

But when you do want a table, the document does not tell you which one, and pricing the candidates is the part nothing else offers.

7.1 Every defensible answer, and what each costs

  ONE ROW COULD BE — give any of these to rows()
    the whole document                      1 rows x   18 cols
    an entry of time                      320 rows x    2 cols
    an entry of users                   2,648 rows x    2 cols
    an entry of versions                  288 rows x  140 cols   60% empty
    an entry of dependencies            4,645 rows x    2 cols
    an entry of devDependencies         3,175 rows x    2 cols
    an entry of scripts                   888 rows x    2 cols
    an item of contributors                 7 rows x    2 cols
      └─ 1,589 more at versions.*.contributors — not counted above
    an item of maintainers                  5 rows x    2 cols
      └─ 1,039 more at versions.*.maintainers — not counted above
    an item of signatures                 288 rows x    2 cols

Read the two extremes. One row per version is a wide table that is mostly holes. One row per dependency edge is a long table with the same name repeated thousands of times. The cost changes in kind with the row you pick — shallow gives you holes, deep gives you duplication — and neither is wrong.

7.2 Computed on the fold, never on the data

The first version of this recursed into raw values and emitted one candidate per version, producing 1,239 lines on a 786 KB file — reproducing the exact failure the whole project exists to prevent.

The fold is not a display step. Anything computed from raw values rather than from the folded structure will be proportional to the data no matter what it prints.

7.3 The candidate that is really several tables

A document of ten kinds has ten answers to “what is one row”, and the operation that finds them has already run by the time the candidates are priced. An earlier version printed the split near the top of the report and then, twenty lines later, offered a single 319-column table that was 93% empty — same program, same fold, two numbers, no join.

  ONE ROW COULD BE — give any of these to rows()
    the whole document                      1 rows x   13 cols
    a node at any depth (13 levels)       336 rows x   13 cols   23% empty
      └─ or 2 tables, split on type — 0% empty: comment 335, story 1
    an item of children                    25 rows x   13 cols   23% empty
      └─ 310 more at 11 other paths — not counted above

The └─ line is the report contradicting its own previous line, on purpose.

7.4 What porting this costs

The prototype prices a candidate by building the table with a data-frame library and reading three things off it: its shape, its fraction of missing cells, and the most-repeated value in each column.

That was the single largest parity risk in the port, named as such before any Rust was written. The core has to reproduce that library’s flattening rule and its missing-value semantics exactly, or every percentage in the block above moves — and a percentage that moves by one point is a diff that looks like a bug in the fold.

It reproduces them, and two of the behaviours had to be measured because the obvious guess was wrong. astype(str) leaves a missing value missing rather than rendering it as the string "nan", so the distinct count drops it. And the inferred column type decides the digits: a column of integers with no hole prints 1, and the same column with one hole prints 1.0. Same data, different count, and nothing in the document changed.

The cost is a standing liability. That part of the core is a model of one library’s behaviour at one version. Anything pandas changes there is a parity failure rather than a bug in fathom, and the harness is what would catch it.