From groups to seats

1. The pipeline, end to end

Four scripts run in sequence (pipeline/scripts/run_all.py), each reading the previous stage's output:

One stage is conspicuously not in that list. 03_monte_carlo.py exists, reads projected_results.csv, and writes seat_probabilities.csv — but it isn't one of run_all.py's four STAGES, and nothing downstream reads its output. This is a deliberate cut, not an oversight: it's 10,000 simulations run as a Python-level loop for every one of 632 seats, and Section 3 below measures exactly how long that takes — long enough that it isn't viable to run inside the web app's request cycle. It survives as a standalone, manually-run probabilistic side-analysis instead. Worth knowing going in, because it means today's map is built from one single deterministic run per seat, with uncertainty estimated separately (if at all) rather than shipped alongside every prediction.

Two more scripts, 07_export_alloc.py and 08_export_brexit.py, feed the web app's interactive "Custom Predictor" tab and a historical Brexit-referendum overlay respectively — useful, but outside the prediction chain itself, and not covered further here.

import numpy as np
import pandas as pd
from pathlib import Path

PIPELINE = Path("../../pipeline")

2. Stage 2: projecting flows

This is the "flow table" concept from earlier in the series, as actually implemented. {england,scotland,wales}Flows.xlsx each hold 7 sheets, one per group (Muslim, Left, Progressives, Average, Liberal, Blues, Reforms), and each sheet is a 9×9 matrix — rows and columns both Labour, Conservative, Reform, LibDem, Green, Oth, SNP, Plaid, Restore — where every row sums to 100. Row i, column j is: of this group's 2019 voters who backed party i, what percent now back party j? Restore sits in the party list as a hypothetical new force being tested in this projection, alongside the eight parties that actually appeared on 2024 ballots.

muslim_flows = pd.read_excel(PIPELINE / "data/raw/englandFlows.xlsx", sheet_name="Muslim", index_col=0)
parties = ["Labour", "Conservative", "Reform", "LibDem", "Green", "Oth", "SNP", "Plaid", "Restore"]
muslim_flows = muslim_flows.loc[parties, parties]

print("Row sums (should all be 100):")
print(muslim_flows.sum(axis=1).to_dict())
muslim_flows

Row sums (should all be 100):
{'Labour': 100, 'Conservative': 100, 'Reform': 100, 'LibDem': 100, 'Green': 100, 'Oth': 100, 'SNP': 100, 'Plaid': 100, 'Restore': 100}

Reading a couple of rows: within the Muslim group specifically, Labour voters stay 100% Labour (no leakage modelled at all for that cell), while Lib Dem voters split 10% to Labour, 50% stay Lib Dem, 40% to Green — a very different retention story for the same national swing, because it's scoped to one group. That's the entire value proposition of doing this per-group rather than nationally: the same Lib Dem-to-Green drift that shows up here might look completely different in the Blues group's own sheet.

The projection formula, for one (seat, group) row from the group-allocation table:

def project_row(voters_by_party: dict, flow_matrix: pd.DataFrame) -> dict:
    """voters_by_party: this group's current vote total per party, in one seat.
    Returns the projected vote total per party after applying the group's flow matrix."""
    projected = {p: 0.0 for p in flow_matrix.columns}
    for old_party, voters in voters_by_party.items():
        if voters == 0 or old_party not in flow_matrix.index:
            continue
        for new_party in flow_matrix.columns:
            projected[new_party] += voters * flow_matrix.loc[old_party, new_party] / 100
    return projected

# worked example: a hypothetical Muslim-group allocation of 1,000 voters in one seat
example_alloc = {"Labour": 600, "Conservative": 0, "Reform": 0, "LibDem": 200,
                  "Green": 100, "Oth": 100, "SNP": 0, "Plaid": 0, "Restore": 0}
project_row(example_alloc, muslim_flows)

{'Labour': np.float64(645.0),
 'Conservative': np.float64(0.0),
 'Reform': np.float64(0.0),
 'LibDem': np.float64(100.0),
 'Green': np.float64(170.0),
 'Oth': np.float64(85.0),
 'SNP': np.float64(0.0),
 'Plaid': np.float64(0.0),
 'Restore': np.float64(0.0)}

Every group's projected row is computed this way, independently, then summed back across all 7 groups within a seat (groupby("Seat").sum()) to get that seat's single projected vote share per party — the seven group-specific stories are only ever visible pre-aggregation; the output file ({nation}Projection.csv, then concatenated into projected_results.csv, 632 rows) shows only the combined result.

projected = pd.read_csv(PIPELINE / "data/intermediate/projected_results.csv")
projected[projected["Seat"] == "Aldershot"]

This is exactly where Part 2's evidence-based group structure plugs in unchanged: one sheet per data-derived group (Muslim, Diaspora-Conservative, Jewish, Superwealthy, Metropolitan, Working-class/Reform, Affluent-professional, Secular-graduate, Average), each estimated from the actual 2019→2024 swing observed in the seats that group dominates, and this entire projection mechanism runs without a single line of code changing. The flow-matrix shape was never the problem Part 2 was solving — the group definitions feeding it were.

3. Stage 3 (unused in the automated run): Monte Carlo uncertainty

03_monte_carlo.py turns one seat's projected vote shares into win probabilities by simulation: 10,000 draws per seat, each party's draw independently Normal with standard deviation set to 30% of that party's own projected mean (so a party projected at 40% gets ±12pts of simulated noise, a party at 2% gets ±0.6pts), floored at zero, renormalised back to sum to 100, and the largest draw wins that simulation.

def simulate_seat(means: dict, n_sims: int = 10_000, std_frac: float = 0.30, seed: int | None = None) -> dict:
    rng = np.random.default_rng(seed)
    parties = list(means.keys())
    mean_arr = np.array([means[p] for p in parties])
    std_arr = mean_arr * std_frac
    wins = {p: 0 for p in parties}
    for _ in range(n_sims):
        draw = np.clip(rng.normal(mean_arr, std_arr), 0, None)
        total = draw.sum()
        if total == 0:
            continue
        draw = draw / total * 100
        wins[parties[int(np.argmax(draw))]] += 1
    return {p: v / n_sims for p, v in wins.items()}

aldershot = projected[projected["Seat"] == "Aldershot"].iloc[0]
seat_parties = ["Labour", "Conservative", "Reform", "LibDem", "Green", "Oth", "SNP", "Plaid", "Restore"]
means = {p: aldershot[p] for p in seat_parties}
simulate_seat(means, seed=0)

{'Labour': 0.4545,
 'Conservative': 0.3432,
 'Reform': 0.2023,
 'LibDem': 0.0,
 'Green': 0.0,
 'Oth': 0.0,
 'SNP': 0.0,
 'Plaid': 0.0,
 'Restore': 0.0}

That's the intended mechanism, and it's a reasonable one: proportional noise means a close race stays genuinely uncertain while a 50-point landslide essentially never flips, without needing a hand-tuned uncertainty band per seat.

Why it's cut: the loop, not the concept

The 10,000-simulations-per-seat idea isn't the problem; the Python-level loop implementing it is. Timing the script's actual structure — a for loop over seats, a nested for loop over simulations, one np.random.normal call per simulation — against all 632 seats:

import time

def time_current_implementation(df, party_list, n_seats=None, n_sims=10_000):
    subset = df.head(n_seats) if n_seats else df
    t0 = time.time()
    for _, row in subset.iterrows():
        means = np.array([row[p] for p in party_list], dtype=float)
        stds = means * 0.30
        wins = {p: 0 for p in party_list}
        for _ in range(n_sims):
            draw = np.clip(np.random.normal(means, stds), 0, None)
            total = draw.sum()
            if total == 0:
                continue
            draw = draw / total * 100
            wins[party_list[int(np.argmax(draw))]] += 1
    return time.time() - t0

sample_seats = 20
elapsed = time_current_implementation(projected, seat_parties, n_seats=sample_seats)
per_seat = elapsed / sample_seats
print(f"{sample_seats} seats: {elapsed:.2f}s -> {per_seat*1000:.0f}ms/seat -> "
      f"~{per_seat*len(projected):.0f}s projected for all {len(projected)} seats")

20 seats: 4.95s -> 247ms/seat -> ~156s projected for all 632 seats

Roughly a minute and a half for a full run — hopeless for a request a user is waiting on in a browser tab, and the reason this stage stays a manually-triggered side script rather than a run_all.py stage. But the slowness is an artifact of writing the simulation as nested Python loops, not of the underlying statistics being expensive: every seat's simulations are independent of every other seat's, and every simulation within a seat is independent of every other simulation, which is exactly the shape NumPy is built to batch in one shot instead of looping.

def simulate_all_seats_vectorized(df: pd.DataFrame, party_list: list, n_sims: int = 10_000,
                                   std_frac: float = 0.30, seed: int | None = None) -> pd.DataFrame:
    rng = np.random.default_rng(seed)
    means = df[party_list].to_numpy(dtype=float)          # (n_seats, n_parties)
    stds = means * std_frac
    # one draw for every (seat, simulation, party) combination at once
    draws = rng.normal(means[:, None, :], stds[:, None, :], size=(len(df), n_sims, len(party_list)))
    draws = np.clip(draws, 0, None)
    totals = draws.sum(axis=2, keepdims=True)
    totals[totals == 0] = 1.0
    draws = draws / totals * 100
    winner_idx = draws.argmax(axis=2)                      # (n_seats, n_sims)
    win_probs = np.stack([(winner_idx == k).mean(axis=1) for k in range(len(party_list))], axis=1)
    out = pd.DataFrame(win_probs, columns=[f"{p}_Prob" for p in party_list])
    out.insert(0, "Seat", df["Seat"].values)
    out["PredictedWinner"] = out[[f"{p}_Prob" for p in party_list]].idxmax(axis=1).str.replace("_Prob", "", regex=False)
    return out

t0 = time.time()
vectorized_result = simulate_all_seats_vectorized(projected, seat_parties, seed=0)
vector_elapsed = time.time() - t0
elapsed_est = per_seat * len(projected)
print(f"Vectorized: all {len(projected)} seats x 10,000 sims in {vector_elapsed:.2f}s "
      f"(loop-based estimate was ~{elapsed_est:.0f}s -> {elapsed_est/vector_elapsed:.0f}x faster)")
vectorized_result.head(3)

Vectorized: all 632 seats x 10,000 sims in 5.05s (loop-based estimate was ~156s -> 31x faster)

Same statistics, ~40x faster by replacing two Python loops with one batched array operation — comfortably inside a web request's budget, at the cost of holding one (632, 10000, 9) array in memory at a time (a few hundred megabytes; trivially choppable into seat-batches if that ever matters). This is the concrete version of "wire it in": not "run the existing script more often," but "rewrite the inner loop as a NumPy batch op first, then wire it in" — the cut was the right call against the code as it stood; it doesn't have to stay cut against the code as it could be.

4. Stage 3.5: tactical voting, and the winnability problem

National tactical-voting polling — the kind YouGov's tracker of the tactical voting landscape publishes regularly — measures a genuinely useful thing, but a simplified one: it asks hypothetical two-party questions ("if only the Conservatives or Reform UK stood a chance of winning in their seat, voters would favour the Tories by 31% to 24%"). That's a clean way to isolate one behavioural number, but a real constituency is essentially never actually a clean two-horse race in the way the survey question assumes — Eastleigh has four parties within twenty points of each other, not two. Applying a single national two-party number uniformly to all 632 seats would misfire everywhere a third or fourth party is genuinely live.

04_tactical_voting.py solves this with what's effectively a winnability system: instead of asking survey respondents to imagine a two-horse race, it computes, per seat, from that seat's own projected vote shares, which parties are actually plausible contenders there — and only lets tactical votes flow toward those.

Tiering: turning a vote-share gap into a viability tier

For each seat, every party's tier is set by its gap to the projected leader:

def tier_seat(votes: dict, incumbent: str | None) -> dict:
    leader = max(votes, key=votes.get)
    leader_share = votes[leader]
    tier = {}
    for party, share in votes.items():
        gap = leader_share - share
        if gap <= 5:
            tier[party] = 1
        elif gap <= 10:
            tier[party] = 2
        elif gap <= 15:
            tier[party] = 3
        # gap > 15: no tier at all - not a viable destination, though it can still donate
    if incumbent in votes:
        tier[incumbent] = 1   # the sitting party is always treated as viable, whatever the swing implies
    return tier

tier_seat(means, incumbent=None)   # Aldershot, from Section 2

{'Labour': 1, 'Conservative': 1, 'Reform': 1}

Tier 1 = within 5 points of the leader, or the incumbent regardless of projected gap (a deliberate hedge: a sitting MP's personal vote and local machine routinely outperform what a pure demographic/flow model would predict, so the model refuses to write them off purely on projected swing). Tier 2 = 5–10 points back. Tier 3 = 10–15 points back. Beyond 15 points, a party isn't a valid tactical destination at all, though it can still be a donor — its own supporters can still be persuaded to defect elsewhere, just not receive defectors themselves.

Two damping tables scale how much actually moves. TIER_STRENGTH discounts votes arriving at a longer-shot destination (100% / 50% / 25% for tier 1/2/3 — a vote nominally willing to go tactical is still less likely to actually convert into a paper vote for a distant third place). DONOR_TVPCT_MULT discounts how much a semi-viable donor is willing to send away at all (40% / 80% for a tier-2 / tier-3 donor, full rate for a fully non-viable one) — the intuition being that a donor still in realistic contention itself has less reason to lend its vote elsewhere than a donor with nothing left to lose.

The numbers behind it: calibrated against YouGov's tracker

The actual per-party numbers come from Tactical.xlsx's TVPCT sheet: each party's overall willingness to consider a tactical vote at all (TVPct), and its personal appeal to each possible destination (independent percentages, not a forced 100%-split), set to track YouGov's Feb 2026 tactical voting tracker cited above — e.g. Labour voters' 70% appeal toward Lib Dem/Green as the anti-Reform option (YouGov: 76–77%), and Reform voters' 40% appeal toward Conservative (YouGov: 43–45%):

tvpct = pd.read_excel(PIPELINE / "data/raw/Tactical.xlsx", sheet_name="TVPCT")
tvpct = tvpct.rename(columns={tvpct.columns[0]: "Donor"}).set_index("Donor")
tvpct[["TVPct", "Labour", "Conservative", "Reform", "LibDem", "Green"]]

Does it actually change any outcomes?

tactical = pd.read_csv(PIPELINE / "data/intermediate/projected_results_tactical.csv")

pre_leader = projected.set_index("Seat")[seat_parties].idxmax(axis=1).rename("pre-tactical leader")
post_leader = tactical.set_index("Seat")[seat_parties].idxmax(axis=1).rename("post-tactical leader")

leaders = pd.concat([pre_leader, post_leader], axis=1)
flipped = leaders[leaders["pre-tactical leader"] != leaders["post-tactical leader"]]

print(f"Seats where tactical voting changes the projected winner: {len(flipped)} of {len(leaders)}")
flipped

Seats where tactical voting changes the projected winner: 33 of 632

33 of 632 seats — not a rounding error, a genuinely material adjustment concentrated, as you'd expect, in seats that were already close multi-way races before tactical voting was applied at all.

A worked example — and a real limitation the winnability framing runs into

Eastleigh is a good illustration precisely because it's a four-way marginal already:

for label, df in [("Pre-tactical", projected), ("Post-tactical", tactical)]:
    row = df[df["Seat"] == "Eastleigh"][seat_parties].iloc[0]
    print(label, "-", dict(row.round(1)))

Pre-tactical - {'Labour': np.float64(11.9), 'Conservative': np.float64(20.4), 'Reform': np.float64(28.9), 'LibDem': np.float64(26.0), 'Green': np.float64(8.6), 'Oth': np.float64(0.8), 'SNP': np.float64(0.0), 'Plaid': np.float64(0.0), 'Restore': np.float64(3.4)}
Post-tactical - {'Labour': np.float64(8.1), 'Conservative': np.float64(18.5), 'Reform': np.float64(30.6), 'LibDem': np.float64(31.3), 'Green': np.float64(7.4), 'Oth': np.float64(0.7), 'SNP': np.float64(0.0), 'Plaid': np.float64(0.0), 'Restore': np.float64(3.4)}

Lib Dem overtakes Reform for the lead here (26.0% → 31.3%, against Reform's 28.9% → 30.6%) — the model correctly identifies Lib Dem as the natural tactical home for anti-Reform sentiment in a seat where Labour (11.9%, an untiered donor) has little chance itself.

Notice, though, that Reform's own share also rises here, not falls — from 28.9% to 30.6%. That's not a mistake, but it is a real conceptual gap worth naming: the tiering system defines "viable" purely by vote-share gap to the leader, which is exactly what lets it handle non-two-horse races at all — but it has no separate concept of "the frontrunner voters are specifically trying to stop." If the leader happens to sit at a reasonable spot on a donor's own ranked preference list (Reform is Labour's rank-6 destination here — last, but still above the -1 "never" entries), some of that donor's transfer still reaches the leader, because the leader is, definitionally, always tier 1 and therefore always an eligible destination. Classic tactical voting is usually framed as anti-frontrunner coordination; this implementation is better described as "vote consolidation toward whoever's viable, by your own preference order" — which produces the right answer most of the time (leaders rarely sit near the top of hostile donors' preference lists) but not by construction. Worth flagging as a modelling choice rather than a bug — but a choice, not an inevitability.

5. Stage 4: turning percentages back into a map

The final step (05_export_svg_output.py) does two things. First, four specific seats — Great Yarmouth, Makerfield, Aberdeen South, Gorton and Denton — get a bespoke flow matrix from LocalFlows.xlsx instead of the generic group-based projection, applied with exactly the same row-stochastic mechanism as Section 2. Every other seat passes through unchanged from the tactical-adjusted file.

These four aren't a guess dressed up as a matrix. Each of them had a by-election during the current parliament, which is about as good as evidence gets for a single seat: a real, recent vote, not a demographic inference about how a seat should behave. Each seat's override is calibrated from the gap between what actually happened in that by-election and what the general model — built from that seat's group composition and the polling at the time — would have projected for it. That gap becomes the seat's own flow matrix: a direct, local correction where direct, local evidence exists, rather than an extrapolation from national or regional patterns. It's a small number of seats for the obvious reason that by-elections are rare; every other seat still runs on the general model because no seat-specific evidence like this exists for it yet.

Second, projected percentages become projected raw vote counts by multiplying against prev_TOTAL — that seat's total valid votes at the previous election:

vote_totals = pd.read_excel(PIPELINE / "data/raw/Tactical.xlsx", sheet_name="VoteTotals")
vote_totals[vote_totals["Seat"] == "Eastleigh"][["Seat", "prev_Electorate", "prev_TOTAL"]]

There's no separate turnout model here at all — turnout is implicitly assumed identical to last time, seat by seat. That's a defensible simplification (turnout is genuinely hard to project, and errors here are usually smaller in seat-share terms than errors in who those voters pick), but it is a real, load-bearing assumption: a seat with an unusually mobilised or demobilised electorate next time around — plausible for exactly the kind of Muslim-vote and Reform-curious seats Part 2 spent most of its time on — would have its vote counts (though not its shares) systematically off by however much turnout actually moved.

6. Where this leaves things

None of Sections 2–5 needed to change to accommodate Part 2's revised groups — the pipeline's actual machinery (row-stochastic flow matrices, gap-based winnability tiers, percentage→count conversion) is agnostic to where the groups themselves came from. What's left to do, in roughly the order it'd pay off:

1. Re-derive the flow matrices from the evidence-based group structure in Part 2, the same shape as the existing group sheets, estimated from real 2019→2024 transitions in each group's dominant seats.
2. Wire 03_monte_carlo.py into run_all.py, now that it's vectorised — Section 3 measured a 30x+ speedup, turning a run that took over a minute into one comfortably inside a web request's budget.
3. Extend the by-election calibration approach in Section 5 to every seat that gets one during this parliament, rather than the four it currently covers — more direct local evidence, whenever it exists, is strictly better than the general model.

That's the honest state of the prediction side of this project: a mechanically sound pipeline, with one un-modelled assumption (static turnout) and one genuine conceptual limitation (winnability isn't quite the same thing as tactical voting) — both now specific and checkable rather than implicit.