Do Britain's 632 constituencies fall into clean types?

1. Setting up the experiment

The input is 632 GB constituencies (England, Scotland, and Wales pooled into one clustering run, not fit separately) described by 34 variables per seat: ethnicity, religion, age bands, education, NS-SEC occupation class, housing tenure (mostly derived from ONS data and other government sources). Deliberately no vote share or swing goes into the features at this stage, because the whole point is to ask whether demographics alone separate seats into discrete types, independently of how they happen to vote. Feeding vote data in here would answer a circular question: "do seats that vote alike get clustered together" is trivially yes if votes are one of the clustering inputs, and I'd rather not spend a blog post proving something I already knew going in.

Four algorithms, two ways of reducing 34 dimensions down to something clustering can actually work with, and four ways of scoring the result: a full grid, not a single best-guess run, because a single best-guess run is how you end up confidently wrong.

• K-means - assigns every seat to whichever of k centroids is nearest, then recomputes centroids as the mean of their assigned seats, and repeats. Assumes round, evenly-sized clusters, whether or not the data agrees to cooperate.
• Hierarchical agglomerative clustering (HAC), Ward linkage - merges the two closest seats (or clusters) repeatedly until everything's in one tree, which can then be cut at any k.
• Gaussian Mixture Models (GMM), a softer K-means: every seat gets a probability of belonging to each cluster, fitted as a mixture of overlapping bell-curve-shaped clumps. Still a hard-clustering technique in the sense that matters here (more on that in the closing section).
• DBSCAN needs no k at all. It grows clusters from dense regions and labels anything outside a dense region as noise, which is useful both for flagging genuine outlier seats and for admitting honestly when there's no dense, separated structure to find in the first place.

And two ways of getting 34 census variables down to a workable number of dimensions before any of the above runs: PCA (a linear projection onto the directions of greatest variance) and UMAP (a nonlinear technique that explicitly reshapes distances to preserve which points are each other's near neighbours). Both get carried through the entire grid below, since the choice between them turns out to matter rather more than it should, as Section 3 shows.

import json
import numpy as np
import pandas as pd
from pathlib import Path

OUTPUT = Path("../output")

selection = pd.read_csv(OUTPUT / "whole_seat_model_selection.csv")
diagnostics = json.load(open(OUTPUT / "whole_seat_clusters_diagnostics.json", encoding="utf-8"))

print(f"{diagnostics['n_seats']} seats, {diagnostics['n_features']} demographic features")
print(f"PCA: {diagnostics['n_pca_components']} components, "
      f"{diagnostics['pca_variance_explained']:.0%} of variance retained")
print(f"k searched from 2 to {diagnostics['k_max_searched']}")
selection.head()

632 seats, 34 demographic features
PCA: 12 components, 91% of variance retained
k searched from 2 to 20

2. Does anything here agree on how many types there are?

Every (embedding, algorithm) pair gets its own recommended k from each of the four requested metrics: Distortion (elbow), Silhouette (max), Davies-Bouldin (min), and Calinski-Harabasz (max), plus a consensus taken as their median. The four routinely disagree with each other, and picking just one and quietly moving on would hide that disagreement rather than report it, which felt like cheating.

def find_elbow(xs, ys):
    xs, ys = np.array(xs, dtype=float), np.array(ys, dtype=float)
    xn = (xs - xs.min()) / (xs.max() - xs.min() + 1e-12)
    yn = (ys - ys.min()) / (ys.max() - ys.min() + 1e-12)
    x1, y1, x2, y2 = xn[0], yn[0], xn[-1], yn[-1]
    dist = np.abs((y2 - y1) * xn - (x2 - x1) * yn + x2 * y1 - y2 * x1) / np.hypot(y2 - y1, x2 - x1)
    return int(xs[np.argmax(dist)])

rows = []
for (emb, algo), grp in selection[selection["algorithm"] != "DBSCAN"].groupby(["embedding", "algorithm"]):
    grp = grp.sort_values("k")
    picks = {
        "distortion": find_elbow(grp["k"], grp["distortion"]),
        "silhouette": int(grp.loc[grp["silhouette"].idxmax(), "k"]),
        "davies_bouldin": int(grp.loc[grp["davies_bouldin"].idxmin(), "k"]),
        "calinski_harabasz": int(grp.loc[grp["calinski_harabasz"].idxmax(), "k"]),
    }
    rows.append({"Embedding": emb, "Algorithm": algo, **picks,
                 "Consensus k": int(np.median(list(picks.values())))})

pd.DataFrame(rows).sort_values(["Embedding", "Algorithm"])

Two things jump out immediately, neither of them reassuring. First, within a single (embedding, algorithm) pair, the four metrics themselves often disagree wildly: PCA + GMM gets told k=10 by Distortion, k=2 by Silhouette, k=13 by Davies-Bouldin, and k=2 by Calinski-Harabasz, a spread of 11 across four metrics supposedly measuring the same partition. Second, PCA and UMAP disagree systematically with each other, not just noisily: PCA's consensus recommendations cluster in the 4 to 6 range, while UMAP's sit at 7 almost everywhere. This isn't four algorithms independently arriving at different answers by chance. It's the same four algorithms, run on the same underlying data, reduced to the same number of dimensions, giving a different answer purely because of which dimensionality-reduction method happens to sit upstream of them.

3. Why that split isn't a coin flip: PCA vs. UMAP on the exact same partition

The gap above is worth pinning down properly rather than shrugging at, because it decides which numbers in the rest of this post are actually trustworthy. Take the identical clustering, GMM at k=6, and score it once on each embedding:

same_k6 = selection[(selection["algorithm"] == "GMM") & (selection["k"] == 6)]
same_k6[["embedding", "silhouette", "davies_bouldin", "calinski_harabasz", "distortion"]]

With the same seats, the same algorithm, and the same k, the Silhouette score jumps from 0.058 on PCA to 0.455 on UMAP, nearly an eightfold difference, on a metric that's supposed to measure one fixed property of one fixed partition. GMM didn't find a better split on UMAP's coordinates; UMAP's coordinates simply lied about how well-separated the points already were. UMAP is explicitly built to warp distances so that points which were near-neighbours in the original 34-dimensional space stay near each other in the low-dimensional output. That's a design choice that's genuinely useful for visualization, but it means the distances UMAP hands back are no longer a faithful measure of separation in the original data. Silhouette, Davies-Bouldin, and Calinski-Harabasz all assume the coordinates they're given are a faithful distance measure, and given UMAP's warped ones, they duly report a rosier picture than the underlying census data actually supports. PCA's projection is linear, so it can only rotate and rescale, never selectively stretch some neighbourhoods and compress others, and its distances stay an honest, if lower-dimensional, read of the real thing. Every result from here on uses PCA for exactly that reason.

4. DBSCAN: does Great Britain even have dense, separated clusters at all?

DBSCAN doesn't need a k. It grows clusters from dense regions and calls everything else noise, which makes it a useful honesty check on the other three algorithms: if there genuinely isn't dense, separated structure in this data, DBSCAN is the one that will actually say so, rather than forcing an answer just because an answer was requested.

dbscan = selection[selection["algorithm"] == "DBSCAN"][
    ["embedding", "eps_percentile", "k", "n_noise", "degenerate", "silhouette"]
]
dbscan

On PCA, three of the four density thresholds tried come back degenerate: DBSCAN finds only one real cluster plus a pile of noise points, not a genuine multi-way split, and the one PCA threshold that does produce two clusters splits off a fairly small dense core (53 seats flagged as noise) rather than partitioning the full 632. UMAP's warped geometry, true to Section 3's finding, manufactures a cleaner-looking multi-cluster split at every threshold, which is the same artifact turning up a second time, not a second, independent piece of evidence. Read honestly, DBSCAN's verdict on the real (PCA) geometry is that Great Britain's 632 constituencies mostly form one dense mass with a soft, fuzzy edge, rather than a small number of crisply separated islands, which is a less satisfying sentence to have to write than I'd have liked.

5. Picking a working configuration anyway, and checking it against judgement

Every metric disagreeing with every other metric is itself the finding, but a concrete partition is still worth building and inspecting properly rather than just waving at the disagreement and walking off. Averaging out four disagreeing per-metric picks with a median gives PCA + GMM, k=6 as a working consensus (Section 2), so that's what gets fit and profiled below.

profiles = json.load(open(OUTPUT / "whole_seat_clusters_profiles.json", encoding="utf-8"))

pd.DataFrame([{
    "Segment": p["segment"], "n seats": p["n_seats"],
    "Distinguishing profile": p["auto_label"],
    "Leading 2024 vote": ", ".join(f"{party} {share:.0f}%" for party, share in p["leading_parties_2024"].items()),
} for p in profiles])

That's a genuinely legible six-way split: a young, graduate, urban segment; a white-British working-class segment; a young-families, routine-occupation segment; an owner-occupier professional segment; an older, veteran, intermediate-occupation segment; and a low-White-British, high-Muslim/Asian segment, each with a distinct implied vote. There's real structure in here, whatever the metrics spent the last two sections arguing about.

tribe_comparison = json.load(open(OUTPUT / "whole_seat_tribe_comparison.json", encoding="utf-8"))
print(f"Adjusted Rand Index vs. this project's own hand-assessed dominant category per seat: "
      f"{tribe_comparison['adjusted_rand_index']}")
print(f"Seats where the hard cluster's majority category disagrees with the hand assessment: "
      f"{tribe_comparison['n_disagreements']} of {tribe_comparison['n_seats']} "
      f"({tribe_comparison['n_disagreements'] / tribe_comparison['n_seats']:.0%})")

Adjusted Rand Index vs. this project's own hand-assessed dominant category per seat: 0.125
Seats where the hard cluster's majority category disagrees with the hand assessment: 266 of 632 (42%)

An Adjusted Rand Index of 0.125 is weak agreement: far above the 0 you'd get from random labels, so the clustering isn't picking up on nothing, but nowhere near the 1.0 that would mean "this hard partition and hand judgement are describing the same categories." 42% of seats get a different dominant category from the two approaches, which is not a rounding error. Combined with Section 2's internal metric disagreement and Section 4's DBSCAN read of one soft, fuzzy mass rather than crisp islands, three independent checks are converging on the same honest answer: there's real, recoverable structure in how British constituencies differ demographically, but a hard partition, where one seat is assigned exactly one label, simply isn't the right shape of answer for it.

6. Why segments, not clusters

Every algorithm tried in this post, GMM's "soft" probabilities included, still answers the same question underneath: which single cluster does this seat most plausibly belong to, and how confident are we? A seat sitting at roughly 55% Segment 3 and 45% Segment 5 gets assigned to whichever is larger, its GMM confidence dutifully reported alongside it, and the other 45% quietly does not survive the trip into the output. Section 5's 42% disagreement rate is exactly the visible cost of that small act of data destruction: a constituency isn't neatly one thing, and forcing it into the label of whichever thing it's most like throws away precisely the part of the picture that explains why judgement and the algorithm keep landing on different labels for the same seat.

I would love to tell you the fix is a better k, a better algorithm, or a better embedding, not least because that would have been considerably less work, but Sections 2 through 4 already ran the full available grid of all three, and none of them was it. The actual fix is dropping the assumption that "seat, therefore single type" is even the right question to be asking. What's needed instead is a technique that answers a different one entirely: not which type a seat belongs to, but how much of each recurring type is present in it, a genuine compositional breakdown in percentages that sum to 100%, the same way a census breakdown of age or ethnicity already does for the underlying population. That's where the next post picks up.