31 Cluster

Which categories in a table are most alike? A table of numbers holds the answer, but a reader cannot see it in the rows. The cluster transform computes it. It treats each category of an axis as a leaf of a tree. It finds the two closest leaves and joins them into one branch. Then it measures again, treating that new branch as one item, and joins the closest two again. It repeats until one tree holds every leaf. The finished tree is called a dendrogram. As with a pie chart, no atom spells that name.

This chapter builds the tree, uses it to order a tile plot, and then puts the two together as a clustered heatmap. nutrients holds one row per pair of food and nutrient. There are eight everyday foods, and each one is measured on five nutrients:

`nutrients`: first 5 of 40 rows
food	nutrient	amount
salmon	protein	25.4
salmon	fat	12.4
salmon	carbs	0.0
salmon	fiber	0.0
salmon	iron	0.5

31.1 The tree

Which two of the eight foods are most alike, and which one is least like all the rest? To answer that, the engine needs to know what to compare. Each leaf is described by a profile. cluster(amount, over = nutrient) says that one leaf’s profile is its amount in every category of nutrient. So each food’s profile is five numbers. Two numbers place a point on a scatter plot, and five numbers place a point the same way, along five axes that are never drawn. The distance between two leaves is the straight-line distance between their points, the Euclidean distance. The distance between two branches is the average of the distances from every leaf on one branch to every leaf on the other. At each step the two closest branches join, and the join is drawn at that distance. This rule is called average linkage.

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  title("Eight foods, joined by what they are made of")

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  title("Eight foods, joined by what they are made of"))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  title("Eight foods, joined by what they are made of")

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  title("Eight foods, joined by what they are made of"))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food.”

path draws the tree. A tree is a set of points that must be joined in a fixed order, and line would sort them by x instead. path draws the rows in the order they arrive, and cluster writes them in the order the tree needs. Here you bind food to x, so x carries the leaves. This chapter calls that axis the leaf axis. You bind nothing to the other position, and cluster supplies it. The transform computes the distance at which each pair of branches joins, and it names that axis Distance.

The place where two branches meet is a join. The further a join is drawn from the leaf axis, the more different the two branches are. Beans and lentils join lowest, and rice and oats join just above them. Almonds join at the greatest distance of all, so almonds are the food least like the rest.

The transform writes the distances into a column called distance. Name it in y() when you want to change the distance axis, for example to set its range with limits. Here the axis runs from 0 to 60:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  y(distance, limits = c(0, 60))

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  y(col.distance, limits = [0, 60]))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  y(:distance, limits = [0, 60])

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  y(col.distance, { limits: [0, 60] }))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food, y is distance from 0 to 60.”

The distance column can also go on color. Each place on the tree is at one distance, its height. color(distance) colors each straight segment by the distance at its middle. A join’s horizontal line takes the join’s distance, and the vertical segments below it are lighter:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  color(distance)

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  color(col.distance))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  color(:distance)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  color(col.distance))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food, color by distance.”

The color repeats what the height already shows. The darkest line is at the top, where almonds join the other seven foods.

Bind food to y instead of x, and the same tree is drawn sideways, with the leaves down the left edge. The distance axis is then x, so its settings would go in x(distance). This tree needs none:

data(nutrients) + path * cluster(amount, over = nutrient) + y(food)

data(nutrients) + path * cluster(col.amount, over=col.nutrient) + y(col.food)

data(nutrients) + path * cluster(:amount, over = :nutrient) + y(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), y(col.food))

“Given nutrients: paths derived by cluster on amount, over nutrient, y is food.”

Whichever axis carries the leaves, the tree is the same. Two things about how cluster builds it cannot be changed. First, distance is measured on the values as they are, with no rescaling. Fat runs from 0.3 to 49.9 and iron from 0.5 to 3.7, so fat has far more effect on the distance than iron does. If you want every nutrient to count equally, rescale amount separately within each nutrient, in the host language, before you draw. Second, branches join at their average distance, so a join is never closer to the leaf axis than the joins inside its two branches. And two leaves with identical profiles are at distance zero, so their join is drawn at zero, where every leaf’s line begins.

31.2 The order

Reading the finished tree along the leaf axis also gives an order, with similar foods next to each other. That order helps on a tile plot, because a pattern in the tiles is hard to see when similar foods are far apart. A tile plot of the same table draws its slots in the order the rows arrived. That order says nothing about the foods:

data(nutrients) + zone + x(food) + y(nutrient) + color(amount) +
  title("In the order the rows arrived")

(data(nutrients) + zone + x(col.food) + y(col.nutrient) + color(col.amount) +
  title("In the order the rows arrived"))

data(nutrients) + zone + x(:food) + y(:nutrient) + color(:amount) +
  title("In the order the rows arrived")

plot(data(nutrients), zone, x(col.food), y(col.nutrient),
  color(col.amount), title("In the order the rows arrived"))

“Given nutrients: zones, x is food, y is nutrient, color by amount.”

zone * cluster draws the same tiles. The leaf axis now runs in the tree’s order. Chicken and salmon are next to each other on the left, and oats, rice, beans and lentils are together between spinach and almonds:

data(nutrients) + zone * cluster(over = nutrient) +
  x(food) + y(nutrient) + color(amount) +
  title("In the tree's order")

(data(nutrients) + zone * cluster(over=col.nutrient) +
  x(col.food) + y(col.nutrient) + color(col.amount) +
  title("In the tree's order"))

data(nutrients) + zone * cluster(over = :nutrient) + x(:food) +
  y(:nutrient) + color(:amount) + title("In the tree's order")

plot(data(nutrients), layer(zone, cluster({ over: col.nutrient })),
  x(col.food), y(col.nutrient), color(col.amount),
  title("In the tree's order"))

“Given nutrients: zones derived by cluster over nutrient, x is food, y is nutrient, color by amount.”

On a tile plot, amount is already bound to color, so cluster does not name it again. over names the profile column, nutrient, which is on y. The other axis, food on x, holds the leaves, and that is the axis the tree reorders.

31.3 The clustered heatmap

So far one tree has ordered one axis. Put a tree on two sides of the tiles, above and to the right, and each axis takes its order from the tree along that side. The result is usually called a clustered heatmap. It is three plots joined by two operators: a tree above, the tile plot below it, and a second tree to the right of the tiles. The plots align because they share axes, as the marginal plot in Composition does. A second rule from that chapter sets their order: a shared categorical axis takes the order a clustered panel computed for it. The tile plot in the middle carries no cluster of its own. It is the same plain tile plot drawn earlier in this chapter, without its title. theme(legend = "bottom") puts its legend under the tiles, so the tree on the right is next to the tiles it orders. The tile plot takes its column order from the tree above and its row order from the tree on the right.

(data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
   theme(height = 150)) /
  ((data(nutrients) + zone + x(food) + y(nutrient) + color(amount) +
      theme(legend = "bottom")) |
     (data(nutrients) + path * cluster(amount, over = food) + y(nutrient) +
        theme(width = 150)))

((data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
   theme(height = 150)) /
  ((data(nutrients) + zone + x(col.food) + y(col.nutrient) + color(col.amount) +
      theme(legend = "bottom")) |
     (data(nutrients) + path * cluster(col.amount, over=col.food) + y(col.nutrient) +
        theme(width = 150))))

(data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  theme(height = 150)) /
  ((data(nutrients) + zone + x(:food) + y(:nutrient) + color(:amount) +
  theme(legend = "bottom")) |
  (data(nutrients) + path * cluster(:amount, over = :food) +
  y(:nutrient) + theme(width = 150)))

below(plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  theme({ height: 150 })),
  beside(plot(data(nutrients), zone, x(col.food), y(col.nutrient),
  color(col.amount), theme({ legend: "bottom" })),
  plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.food })), y(col.nutrient),
  theme({ width: 150 }))))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food; above zones, x is food, y is nutrient, color by amount, beside paths derived by cluster on amount, over food, y is nutrient.”

Each column of tiles is one food. Follow the tree upward from a column to its first join: that join shows which food, or which group of foods, the column’s food is most like. The tree on the right swaps the roles. Its leaves are the five nutrients, and each nutrient’s profile is its amount in the eight foods. Iron and fiber join first. The trees are not decoration. They decide the order of the foods across the page and of the nutrients down it.

31.4 What cluster refuses

The tree decides the order of the leaf axis. You saw order() on a bar chart, where it sorted the categories along an axis, and you may want to write it here as well. An axis can hold only one order. order() sorts by a column, cluster derives an order from the tree, and writing both is a contradiction:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  order(food)

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  order(col.food))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  order(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  order(col.food))

Error:
! gog: `order(food)` and `cluster` both order the `food` axis — one sorts by a column, the other derives the order from the tree, and an axis holds one order. Drop one.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

A profile holds one value for every category of the profile column, so that column must hold categories. In gm_all, year holds numbers, so the grammar reads it as continuous, and a continuous column has no categories:

`gm_all`: first 5 of 1704 rows
country	continent	year	life	population	gdp
Afghanistan	Asia	1952	28.801	8425333	779.4453
Afghanistan	Asia	1957	30.332	9240934	820.8530
Afghanistan	Asia	1962	31.997	10267083	853.1007
Afghanistan	Asia	1967	34.020	11537966	836.1971
Afghanistan	Asia	1972	36.088	13079460	739.9811

data(gm_all) + path * cluster(life, over = year) + x(continent)

data(gm_all) + path * cluster(col.life, over=col.year) + x(col.continent)

data(gm_all) + path * cluster(:life, over = :year) + x(:continent)

plot(data(gm_all), layer(path, cluster(col.life, { over: col.year })),
  x(col.continent))

Error:
! gog: `cluster(over = year)` names a numeric column, and a profile's entries are indexed by names — every leaf holds one value per level of it. Make it text where your data lives, or bin it there first.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

Each entry of a profile is one value. If more than one row holds the same leaf and the same profile category, gog refuses. It will not choose a mean, a sum or a median for you, and the message names the pair it found. gm_eras keeps two of gm_all’s years, 1957 and 2007, as the categories of a column called era, so over = era does name a categorical column. It still fails, because every continent holds one row per country in each era:

`gm_eras`: first 5 of 284 rows
country	continent	year	life	population	gdp	era
Afghanistan	Asia	1957	30.332	9240934	820.8530	1957
Afghanistan	Asia	2007	43.828	31889923	974.5803	2007
Albania	Europe	1957	59.280	1476505	1942.2842	1957
Albania	Europe	2007	76.423	3600523	5937.0295	2007
Algeria	Africa	1957	45.685	10270856	3013.9760	1957

data(gm_eras) + path * cluster(life, over = era) + x(continent)

data(gm_eras) + path * cluster(col.life, over=col.era) + x(col.continent)

data(gm_eras) + path * cluster(:life, over = :era) + x(:continent)

plot(data(gm_eras), layer(path, cluster(col.life, { over: col.era })),
  x(col.continent))

Error:
! gog: `continent` = Asia at `era` = 1957 holds 33 rows, and a profile's entry is one value — the mean, the sum and the median are all defensible, so gog will not choose. Aggregate to one row per pair where your data lives.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

Aggregate to one row per continent and era in the host language first, for example with each continent’s mean life in each era.

31.5 Where the limits are

Each refusal above names a fix for the sentence you wrote. The limits below are different, because they are about what the engine draws and what the grammar can say.

Trees are often drawn in a circle, with the leaves around the rim, so you may want to draw one that way. polar() is what bends a plot, and a tree inside it is valid grammar that this engine does not draw. A circular tree needs curved lines, and path draws straight ones. Write polar(), and the engine says so:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) + polar()

data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) + polar()

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  polar()

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  polar())

Error:
! gog: a circular cluster tree is valid grammar, but this engine does not draw it: the tree's horizontal bars would bend into arcs, and a path's segments draw straight. Drop `polar()`.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

You may want to color the tree by food. color(food) gives each food its own color, but every join connects two or more foods. A join’s lines would need several colors at once, so the engine refuses color(food):

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  color(food)

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  color(col.food))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  color(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  color(col.food))

Error:
! gog: `color(...)` under `cluster` has nothing to read — the tree's strokes are computed from every row, so no column is left to split them by. `style(color = )` paints the tree; coloring its branches by a cut is not designed.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

To draw the whole tree in one color instead, name the color in style(), such as style(color = "steelblue"). Each place on the tree has one distance, so color(distance) works (The tree).

Another request is to cut the tree at a chosen distance and give each group below the cut its own color. The grammar has no word for that, so gog cannot draw it.

And a tree uses its second position for the distance, so it cannot share a plot with the tiles it orders. That is why the clustered heatmap composes three plots.

31.6 What comes next

cluster reorders an axis, and the plot still uses two positions. Space adds the third position, z, and draws the marks inside a cube you can turn.