Trying to sort 632 seats into types

1. Setting up the experiment

The input is 632 GB constituencies (England, Scotland, Wales pooled into one clustering run, not fit separately) described by 34 2021 Census variables per seat — ethnicity, religion, age bands, education, NS-SEC occupation class, housing tenure, plus a non-dom-derived Superwealthy proxy. Deliberately no vote share or swing goes into the features at this stage: the whole point is to ask whether demographics alone separate seats into discrete types, independently of how they happen to vote. Feeding vote data in here would answer a circular question — "do seats that vote alike get clustered together" is trivially yes if votes are one of the clustering inputs.

Four algorithms, two ways of reducing 34 dimensions down to something clustering can work with, and four ways of scoring the result — a full grid, not a single best-guess run:

• K-means — assigns every seat to whichever of k centroids is nearest, then recomputes centroids as the mean of their assigned seats, and repeats. Assumes round, evenly-sized clusters.
• Hierarchical agglomerative clustering (HAC), Ward linkage — merges the two closest seats (or clusters) repeatedly until everything's in one tree, which can then be cut at any k.
• Gaussian Mixture Models (GMM) — a softer K-means: every seat gets a probability of belonging to each cluster, fitted as a mixture of overlapping bell-curve-shaped clumps. Still a hard-clustering technique in the sense that matters here — see the closing section.
• DBSCAN — needs no k at all. Grows clusters from dense regions and labels anything outside a dense region as noise; useful both for flagging genuine outlier seats and for revealing when there's no dense, separated structure to find in the first place.

And two ways of getting 34 census variables down to a workable number of dimensions before any of the above runs: PCA (a linear projection onto the directions of greatest variance) and UMAP (a nonlinear technique that explicitly reshapes distances to preserve which points are each other's near neighbours). Both get carried through the entire grid below, since — as Section 3 shows — the choice between them turns out to matter a lot.

import json
import numpy as np
import pandas as pd
from pathlib import Path

OUTPUT = Path("../output")

selection = pd.read_csv(OUTPUT / "whole_seat_model_selection.csv")
diagnostics = json.load(open(OUTPUT / "whole_seat_clusters_diagnostics.json", encoding="utf-8"))

print(f"{diagnostics['n_seats']} seats, {diagnostics['n_features']} demographic features")
print(f"PCA: {diagnostics['n_pca_components']} components, "
      f"{diagnostics['pca_variance_explained']:.0%} of variance retained")
print(f"k searched from 2 to {diagnostics['k_max_searched']}")
selection.head()

632 seats, 34 demographic features
PCA: 12 components, 91% of variance retained
k searched from 2 to 20

2. The sweep: does anything here agree on how many types there are?

Every (embedding, algorithm) pair gets its own recommended k from each of the four requested metrics — Distortion (elbow), Silhouette (max), Davies-Bouldin (min), Calinski-Harabasz (max) — plus a consensus taken as their median, since the four routinely disagree and picking one silently would hide that disagreement rather than report it.

def find_elbow(xs, ys):
    xs, ys = np.array(xs, dtype=float), np.array(ys, dtype=float)
    xn = (xs - xs.min()) / (xs.max() - xs.min() + 1e-12)
    yn = (ys - ys.min()) / (ys.max() - ys.min() + 1e-12)
    x1, y1, x2, y2 = xn[0], yn[0], xn[-1], yn[-1]
    dist = np.abs((y2 - y1) * xn - (x2 - x1) * yn + x2 * y1 - y2 * x1) / np.hypot(y2 - y1, x2 - x1)
    return int(xs[np.argmax(dist)])

rows = []
for (emb, algo), grp in selection[selection["algorithm"] != "DBSCAN"].groupby(["embedding", "algorithm"]):
    grp = grp.sort_values("k")
    picks = {
        "distortion": find_elbow(grp["k"], grp["distortion"]),
        "silhouette": int(grp.loc[grp["silhouette"].idxmax(), "k"]),
        "davies_bouldin": int(grp.loc[grp["davies_bouldin"].idxmin(), "k"]),
        "calinski_harabasz": int(grp.loc[grp["calinski_harabasz"].idxmax(), "k"]),
    }
    rows.append({"Embedding": emb, "Algorithm": algo, **picks,
                 "Consensus k": int(np.median(list(picks.values())))})

pd.DataFrame(rows).sort_values(["Embedding", "Algorithm"])

Two things jump out immediately. First, within a single (embedding, algorithm) pair, the four metrics themselves often disagree wildly — PCA + GMM gets told k=10 by Distortion, k=2 by Silhouette, k=13 by Davies-Bouldin and k=2 by Calinski-Harabasz, a spread of 11 across four metrics measuring the same partition. Second, PCA and UMAP disagree systematically with each other, not just noisily: PCA's consensus recommendations cluster in the 4–6 range, UMAP's sit at 7 almost everywhere. That's not four algorithms independently converging on a different answer — it's the same four algorithms, run on the same underlying data, reduced to the same number of dimensions, giving a different answer purely because of which dimensionality-reduction method sits upstream of them.

3. Why that split isn't a coin flip: PCA vs. UMAP on the exact same partition

The gap above is worth pinning down precisely rather than shrugging at, because it decides which numbers in the rest of this post can be trusted. Take the identical clustering — GMM, k=6 — and score it once on each embedding:

same_k6 = selection[(selection["algorithm"] == "GMM") & (selection["k"] == 6)]
same_k6[["embedding", "silhouette", "davies_bouldin", "calinski_harabasz", "distortion"]]

Same seats, same algorithm, same k — and the Silhouette score jumps from 0.058 on PCA to 0.455 on UMAP, nearly an eightfold difference, on a metric that's supposed to be measuring one fixed property of one fixed partition. That's not GMM finding a better split on UMAP's coordinates; it's UMAP's coordinates lying about how well-separated the points already were. UMAP is explicitly built to warp distances so that points which were near-neighbours in the original 34-dimensional space stay near each other in the low-dimensional output — a design choice that's genuinely useful for visualization, but it means the distances UMAP hands back are no longer a faithful measure of separation in the original data. Silhouette, Davies-Bouldin and Calinski-Harabasz all assume the coordinates they're given are a faithful distance measure. Feed them UMAP's warped ones and they report a rosier picture than the underlying census data actually supports. PCA's projection is linear — it can only rotate and rescale, never selectively stretch some neighbourhoods and compress others — so its distances stay an honest (if lower-dimensional) read of the real thing. Every result from here on uses PCA for exactly that reason.

4. DBSCAN: does GB even have dense, separated clusters at all?

DBSCAN needs no k — it grows clusters from dense regions and calls everything else noise — which makes it a useful honesty check on the other three: if there genuinely isn't dense, separated structure in this data, DBSCAN is the algorithm that will say so instead of forcing an answer.

dbscan = selection[selection["algorithm"] == "DBSCAN"][
    ["embedding", "eps_percentile", "k", "n_noise", "degenerate", "silhouette"]
]
dbscan

On PCA, three of the four density thresholds tried come back degenerate — DBSCAN finds only one real cluster plus a pile of noise points, not a genuine multi-way split — and the one PCA threshold that does produce two clusters splits off a fairly small dense core (53 seats flagged as noise) rather than partitioning the full 632. UMAP's warped geometry, true to Section 3's finding, manufactures a cleaner-looking multi-cluster split at every threshold — which is the same artifact showing up a second way, not a second, independent piece of evidence. Read honestly, DBSCAN's verdict on the real (PCA) geometry is that Great Britain's 632 constituencies mostly form one dense mass with a soft, fuzzy edge, not a small number of crisply separated islands.

5. Picking a working configuration anyway, and checking it against judgement

Every metric disagreeing is itself the finding, but a concrete partition is still worth building and inspecting — averaging out four disagreeing per-metric picks with a median gives PCA + GMM, k=6 as a working consensus (Section 2), so that's what gets fit and profiled.

profiles = json.load(open(OUTPUT / "whole_seat_clusters_profiles.json", encoding="utf-8"))

pd.DataFrame([{
    "Segment": p["segment"], "n seats": p["n_seats"],
    "Distinguishing profile": p["auto_label"],
    "Leading 2024 vote": ", ".join(f"{party} {share:.0f}%" for party, share in p["leading_parties_2024"].items()),
} for p in profiles])

That's a genuinely legible six-way split — a young/graduate/urban segment, a white-British working-class segment, a young-families/routine-occupation segment, an owner-occupier professional segment, an older/veteran/intermediate-occupation segment, and a low-White-British, high-Muslim/Asian segment — each with a distinct implied vote. Real structure is in here.

tribe_comparison = json.load(open(OUTPUT / "whole_seat_tribe_comparison.json", encoding="utf-8"))
print(f"Adjusted Rand Index vs. this project's own hand-assessed dominant category per seat: "
      f"{tribe_comparison['adjusted_rand_index']}")
print(f"Seats where the hard cluster's majority category disagrees with the hand assessment: "
      f"{tribe_comparison['n_disagreements']} of {tribe_comparison['n_seats']} "
      f"({tribe_comparison['n_disagreements'] / tribe_comparison['n_seats']:.0%})")

Adjusted Rand Index vs. this project's own hand-assessed dominant category per seat: 0.125
Seats where the hard cluster's majority category disagrees with the hand assessment: 266 of 632 (42%)

An Adjusted Rand Index of 0.125 is weak agreement — far above the 0 you'd get from random labels, so the clustering isn't picking up nothing, but nowhere near the 1.0 that would mean "this hard partition and hand judgement are describing the same categories." 42% of seats get a different dominant category from the two approaches. Combined with Section 2's internal metric disagreement and Section 4's DBSCAN read of one soft, fuzzy mass rather than crisp islands, three independent checks are converging on the same honest answer: there's real, recoverable structure in how British constituencies differ demographically, but a hard partition — one seat, one label — isn't the right shape of answer for it.

6. Why segments, not clusters

Every algorithm tried here, GMM's "soft" probabilities included, still answers the same question underneath: which single cluster does this seat most plausibly belong to, and how confident are we? A seat sitting at 55% Segment 3-ish and 45% Segment 5-ish gets assigned to whichever is larger and its GMM confidence reported alongside it — the 45% doesn't survive the trip into the output. Section 5's 42% disagreement rate is exactly the visible cost of that: a constituency isn't neatly one thing, and forcing it into the label of whichever thing it's most like throws away the part of the picture explaining why judgement and the algorithm often land on different labels for the same seat.

The fix isn't a better k, a better algorithm, or a better embedding — Section 2 through 4 already ran the full available grid of all three. It's dropping the assumption that "seat → single type" is even the right question. What's needed instead is a technique that answers a different one: not which type a seat belongs to, but how much of each recurring type is present in it — a genuine compositional breakdown, in percentages that sum to 100%, the same way a census breakdown of age or ethnicity already does for the underlying population. That's where the next post picks up.