6 The book’s data

Where do gapminder_2007, medals and winds come from, and how do you get them?

Every chapter of this book draws from a small set of shared tables. They are published beside the book as CSV files, one per table, so you can run any example in this book yourself. Each file sits at a fixed address:

https://psychometrician.github.io/gog-book/data/<name>.csv

gapminder_2007.csv holds the 142 countries the first chapter plots. winds.csv holds the wind observations behind the polar chapter. The full list is at the end of this page.

Reading one takes a single call, book_table(), which comes with the package in all four languages. Nothing else needs installing.

6.1 Reading a table

book_table() fetches any of the thirty-five by name. It is included with the package, so there is nothing to copy:

gapminder_2007 <- book_table("gapminder_2007")
data(gapminder_2007) + point + x(gdp) + y(life)

gapminder_2007 = book_table("gapminder_2007")
data(gapminder_2007) + point + x(col.gdp) + y(col.life)

gapminder_2007 = book_table("gapminder_2007")
data(gapminder_2007) + point + x(:gdp) + y(:life)

const gapminder_2007 = await book_table("gapminder_2007");
plot(data(gapminder_2007), point, x(col.gdp), y(col.life));

It also gives each column the right type. A CSV is text, so numbers arrive as text and have to be converted. A column becomes numbers when every value in it is a number, and stays text otherwise. The text argument forces a column to stay text. Three columns in these tables need that, and the next section explains which and why.

This one function is the whole reason the four packages carry anything that is not grammar. R reads a CSV in a single call and Python and Julia in a handful, but JavaScript has no CSV reader in its standard library, so the same work there is more than thirty lines. One country in gapminder_2007.csv is "Congo, Dem. Rep.", and its name holds a comma, so splitting each line on commas gives that row seven fields where the header has six. Asking a reader to paste that before drawing anything is what moved it into the packages.

Nothing else is needed. Downloads in Julia, urllib in Python and fetch in JavaScript are all standard, so the helper costs you no dependency beyond the package you already installed to draw.

The plot is the same in all four, because the table is the same file and the engine is the same program.

Reading a different table is the same call with a different name: book_table("winds"), book_table("medals"). The list of names is at the end of this page.

6.2 Two things a CSV cannot say

A CSV is text. It records what a value is, never what kind of thing it is. Two kinds of information are therefore lost in the file and have to be restored when you read it. Both matter to the plot, so this book states them rather than hiding them.

The order of a category. A column such as direction holds the eight points of the compass. Their order is N, NE, E, SE, S, SW, W, NW, which is the order a compass runs in. A CSV cannot record that, so every language sorts them alphabetically instead: E, N, NE, NW, S, SE, SW, W. The plot still draws, and it is wrong in a way that is easy to miss. Declaring the order fixes it:

winds_local <- read.csv("data/winds.csv")
compass <- c("N", "NE", "E", "SE", "S", "SW", "W", "NW")
winds_local$direction <- factor(winds_local$direction, levels = compass)

data(winds_local) + bar * count + x(direction) + polar()

“Given the winds read from the file: bars derived by count, x is direction.”

The compass now runs the way a compass runs. Without the third line it would start at E and the wind rose would point the wrong way.

Each language spells the declaration differently: R uses factor(levels = ), Python and JavaScript use ordered(), and Julia uses ordered() too. The Data chapter covers this in full, because declared order is part of the grammar and not a detail of file formats.

Labels that look like numbers. Three columns in these tables hold text that a reader will turn into numbers unless told otherwise. census.csv has an age column of 0, 5, 10, which are the names of age bands rather than quantities. sessions.csv has a session column of 01, 02, 03, and a number cannot keep a leading zero, so 01 comes back as 1. That is the whole reason book_table() takes a text argument:

census <- book_table("census", text = "age")
sessions <- book_table("sessions", text = "session")

census = book_table("census", text=("age",))
sessions = book_table("sessions", text=("session",))

census = book_table("census"; text = ["age"])
sessions = book_table("sessions"; text = ["session"])

const census = await book_table("census", ["age"]);
const sessions = await book_table("sessions", ["session"]);

Without it, age becomes the numbers 0, 5 and 10, and an axis that should read as five labels is drawn as a scale from 0 to 10. The plot looks reasonable, which is what makes the mistake worth naming.

6.3 The tables

Table	Rows	Columns
actuals	5	2
cashflow	6	3
census	36	3
commutes	16	3
day_cycle	25	2
forecast	3	2
gapminder_2007	142	6
gapminder_asia	60	6
gdp_rug	142	1
gdp_threshold	1	1
gm_all	1704	6
gm_continents	1056	6
gm_eras	284	7
gm_europe	30	6
iris_flowers	150	4
life_bands	3	2
life_rug	142	1
listening	104	3
maunga_whau	1364	3
medals	5	4
milestones	2	3
quakes_fiji	1000	5
quarterly	19	2
recessions	2	3
ripples	5400	5
score_band	21	4
sessions	14	5
six_weeks	42	4
speed_target	1	1
spending	12	4
target_band	1	2
thermal_marks	34	5
thermals	340	5
tide	8	2
winds	264	4
world_borders	4150	5

The tables are not all equally serious. The gapminder cuts are real figures from the Gapminder Foundation, released into the public domain. iris_flowers, maunga_whau and quakes_fiji are reshaped from tables included with R, and carry R’s own license. Everything else is written for this book to make a particular point, and census in particular is two plausible city profiles rather than any real census. The terms for each are in data/LICENSE.md.