31 Cluster

Which things are most alike? A table of numbers holds the answer, but a reader cannot see it in the rows. The cluster transform computes it. It treats each category of an axis as a leaf of a tree. It joins the two closest leaves, then the next two, and keeps joining until one tree holds them all. The world calls the picture a dendrogram, and as with the pie there is no atom that spells the name.

This chapter builds the tree, then uses it to order a tile plot. nutrients holds one row per pair of food and nutrient: eight everyday foods, each measured on five nutrients:

`nutrients`: first 5 of 40 rows
food	nutrient	amount
salmon	protein	25.4
salmon	fat	12.4
salmon	carbs	0.0
salmon	fiber	0.0
salmon	iron	0.5

31.1 The tree

Which two of the eight foods are most alike, and which one is least like all the rest? To answer that, the engine needs to know what to compare. Each leaf is described by a profile. cluster(amount, over = nutrient) says that one leaf’s profile is its amount in every category of nutrient. The distance between two leaves is the straight-line distance between their two profiles. Two branches join at the average distance between their leaves.

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  title("Eight foods, joined by what they are made of")

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  title("Eight foods, joined by what they are made of"))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  title("Eight foods, joined by what they are made of")

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  title("Eight foods, joined by what they are made of"))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food.”

path draws the tree. A tree is a set of corners joined in a fixed order, and path joins rows in the order the table gives them. The bound position is the leaf axis. The other position stays unbound: the tree computes the distance at which two branches join, and names that axis Distance. The place where two branches meet is a join, and its height says how different they are. Beans and lentils join lowest, rice and oats join just above them, and almonds join last of all.

Bind y instead of x, and the same tree lies on its side:

data(nutrients) + path * cluster(amount, over = nutrient) + y(food)

data(nutrients) + path * cluster(col.amount, over=col.nutrient) + y(col.food)

data(nutrients) + path * cluster(:amount, over = :nutrient) + y(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), y(col.food))

“Given nutrients: paths derived by cluster on amount, over nutrient, y is food.”

Three things about how cluster builds the tree cannot be changed. Distance is straight-line distance on the values as given, so a nutrient with large values changes the distance more than one with small values. Rescale where your data lives if that is not what you mean. Branches join at their average distance, so every join stands further from the leaves than the joins inside it. And two leaves with identical profiles join at zero, drawn as a join resting on the leaf axis.

31.2 The order

Similar foods should stand next to each other on a tile plot, or the pattern in the tiles is hard to see. The tree can also be read as an order. A tile plot of the same table draws its slots in the order the rows arrived, and nothing chose that order:

data(nutrients) + zone + x(food) + y(nutrient) + color(amount) +
  title("In the order the rows arrived")

(data(nutrients) + zone + x(col.food) + y(col.nutrient) + color(col.amount) +
  title("In the order the rows arrived"))

data(nutrients) + zone + x(:food) + y(:nutrient) + color(:amount) +
  title("In the order the rows arrived")

plot(data(nutrients), zone, x(col.food), y(col.nutrient),
  color(col.amount), title("In the order the rows arrived"))

“Given nutrients: zones, x is food, y is nutrient, color by amount.”

zone * cluster draws the same tiles with the leaf axis in the tree’s order. Chicken and salmon now stand together on the left, and the four foods that are mostly carbohydrate stand together between spinach and almonds:

data(nutrients) + zone * cluster(over = nutrient) +
  x(food) + y(nutrient) + color(amount) +
  title("In the tree's order")

(data(nutrients) + zone * cluster(over=col.nutrient) +
  x(col.food) + y(col.nutrient) + color(col.amount) +
  title("In the tree's order"))

data(nutrients) + zone * cluster(over = :nutrient) + x(:food) +
  y(:nutrient) + color(:amount) + title("In the tree's order")

plot(data(nutrients), layer(zone, cluster({ over: col.nutrient })),
  x(col.food), y(col.nutrient), color(col.amount),
  title("In the tree's order"))

“Given nutrients: zones derived by cluster over nutrient, x is food, y is nutrient, color by amount.”

On a tile plot the values are already bound to color, so cluster need not name them again. over names the profile column, and the other categorical axis is the leaf axis the tree reorders.

31.3 The clustered heatmap

The full figure puts a tree on two sides of the tiles, so you can see where each order came from. The world calls it a clustered heatmap. It is three plots and two operators: a tree above, the tile plot, and a second tree on the right, beyond the legend. The alignment comes from one rule in Composition: a shared categorical axis takes the order a clustered panel computed for it. The tile plot itself stays plain. Its column order arrives from the tree above, and its row order from the tree on the right.

(data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
   theme(height = 150)) /
  ((data(nutrients) + zone + x(food) + y(nutrient) + color(amount)) |
     (data(nutrients) + path * cluster(amount, over = food) + y(nutrient) +
        theme(width = 150)))

((data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
   theme(height = 150)) /
  ((data(nutrients) + zone + x(col.food) + y(col.nutrient) + color(col.amount)) |
     (data(nutrients) + path * cluster(col.amount, over=col.food) + y(col.nutrient) +
        theme(width = 150))))

(data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  theme(height = 150)) /
  ((data(nutrients) + zone + x(:food) + y(:nutrient) + color(:amount)) |
  (data(nutrients) + path * cluster(:amount, over = :food) +
  y(:nutrient) + theme(width = 150)))

below(plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  theme({ height: 150 })),
  beside(plot(data(nutrients), zone, x(col.food), y(col.nutrient),
  color(col.amount)),
  plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.food })), y(col.nutrient),
  theme({ width: 150 }))))

“Given nutrients: paths derived by cluster on amount, over nutrient, x is food, above zones, x is food, y is nutrient, color by amount, beside paths derived by cluster on amount, over food, y is nutrient.”

Follow a column of tiles up into the tree, and the tree says which columns that food was joined with. The trees are not decoration: they decide which column stands next to which.

31.4 What cluster refuses

Having learned order() on a bar chart, you may write it on the tree as well. An axis holds one order. order() sorts by a column, cluster derives an order from the tree, and saying both is a contradiction:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  order(food)

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  order(col.food))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  order(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  order(col.food))

Error:
! gog: `order(food)` and `cluster` both order the `food` axis — one sorts by a column, the other derives the order from the tree, and an axis holds one order. Drop one.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

A profile holds one value for every category of the profile column, so that column must hold categories. A continuous column has none:

`gm_all`: first 5 of 1704 rows
country	continent	year	life	population	gdp
Afghanistan	Asia	1952	28.801	8425333	779.4453
Afghanistan	Asia	1957	30.332	9240934	820.8530
Afghanistan	Asia	1962	31.997	10267083	853.1007
Afghanistan	Asia	1967	34.020	11537966	836.1971
Afghanistan	Asia	1972	36.088	13079460	739.9811

data(gm_all) + path * cluster(life, over = year) + x(continent)

data(gm_all) + path * cluster(col.life, over=col.year) + x(col.continent)

data(gm_all) + path * cluster(:life, over = :year) + x(:continent)

plot(data(gm_all), layer(path, cluster(col.life, { over: col.year })),
  x(col.continent))

Error:
! gog: `cluster(over = year)` names a numeric column, and a profile's entries are indexed by names — every leaf holds one value per level of it. Make it text where your data lives, or bin it there first.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

A profile’s entry is one value, so a pair that holds several rows is refused, and the message names the pair. In gm_eras every continent holds one row per country in each era. Aggregate to one row per pair first, where your data lives:

`gm_eras`: first 5 of 284 rows
country	continent	year	life	population	gdp	era
Afghanistan	Asia	1957	30.332	9240934	820.8530	1957
Afghanistan	Asia	2007	43.828	31889923	974.5803	2007
Albania	Europe	1957	59.280	1476505	1942.2842	1957
Albania	Europe	2007	76.423	3600523	5937.0295	2007
Algeria	Africa	1957	45.685	10270856	3013.9760	1957

data(gm_eras) + path * cluster(life, over = era) + x(continent)

data(gm_eras) + path * cluster(col.life, over=col.era) + x(col.continent)

data(gm_eras) + path * cluster(:life, over = :era) + x(:continent)

plot(data(gm_eras), layer(path, cluster(col.life, { over: col.era })),
  x(col.continent))

Error:
! gog: `continent` = Asia at `era` = 1957 holds 33 rows, and a profile's entry is one value — the mean, the sum and the median are all defensible, so gog will not choose. Aggregate to one row per pair where your data lives.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

31.5 Where the limits are

Trees are often drawn in a circle, with the leaves around the rim, so you may write polar() here. A circular tree is valid grammar that this engine does not draw. Adding polar() says exactly that:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) + polar()

data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) + polar()

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  polar()

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  polar())

Error:
! gog: a circular cluster tree is valid grammar this engine does not draw yet — the treads would bend into arcs, and a path's segments draw straight. Drop `polar()`, or wait for the feature.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

The grammar has no way to cut the tree into groups and color each group, and color() on the tree says so:

data(nutrients) + path * cluster(amount, over = nutrient) + x(food) +
  color(food)

(data(nutrients) + path * cluster(col.amount, over=col.nutrient) + x(col.food) +
  color(col.food))

data(nutrients) + path * cluster(:amount, over = :nutrient) + x(:food) +
  color(:food)

plot(data(nutrients),
  layer(path, cluster(col.amount, { over: col.nutrient })), x(col.food),
  color(col.food))

Error:
! gog: `color(...)` under `cluster` has nothing to read — the tree's strokes are computed from every row, so no column is left to split them by. `style(color = )` paints the tree; coloring its branches by a cut is not designed.
gog: nothing was rendered. Fix the above, or set GOG_STRICT=0 to draw anyway.

And one plot clusters one axis. The heatmap above orders both, because it composes two clustered plots, one for each axis.