A more 'objective' value of a house in the UK (sans London)

What listings sites don't tell you

Property sites are generous with the property itself: square footage, photos, floor plans, asking price, the lot. They're considerably shyer about everything around it. Rightmove will show you a map of nearby schools without the faintest hint of whether any of them are good; Zoopla's valuation tool leans almost entirely on what the house next door recently sold for. Neither tells someone moving to an unfamiliar area much about the deprivation, crime or income levels of the neighbourhood they're about to commit several hundred thousand pounds to, and a local forum mostly offers anecdote, which is not, strictly speaking, data.

This project's answer is a house-price model that takes the surrounding neighbourhood seriously as an input in its own right, rather than an afterthought bolted onto square footage and property type, and that actually measures how much of a house's price that neighbourhood explains.

The data: property records meet the census

England was the target, and not out of any lack of ambition. Scotland and Northern Ireland record small-area socioeconomic data under entirely different systems, so pooling all three would have meant reconciling incompatible geographies for comparatively little payoff, given that the great majority of the UK's internal relocators and new arrivals move within England anyway. Two datasets, joined at LSOA/MSOA level:

• HM Land Registry Price Paid Data, November 2025: every residential sale, with property type, price and postcode.
• The English Indices of Deprivation (2024–25), with two of its seven domains swapped out, because both were quietly measuring the wrong thing. The IoD's own Income domain only measures the share of people on a low income, so a neighbourhood with one very poor resident and everyone else wealthy would register as more deprived than one where everyone earns exactly the national average: the wrong shape of variable for pricing a house. Actual small-area income estimates were used instead. Education has the mirror problem (it measures the share lacking qualifications, not the presence of good schools), so it's replaced with Attainment 8: average GCSE score by Local Authority, weighted to pupil residence rather than school location, so it reflects the catchment area rather than whatever happens to sit inside the boundary on a map.

Land Registry doesn't record floor area, unhelpfully, so each sale was matched to its Energy Performance Certificate for that figure instead.

This was a lengthy process that consumed many afternoons.

The EPC register holds ~25 million certificates, and took distributed computing to match, with a bit of NLP to address inconsistent flat addressing rules, all just to add a floor area to some property sales.

The code and the merged dataset behind every table below are the dissertation's own

import pandas as pd
import numpy as np
from pathlib import Path
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

CODE_DIR = Path("../socioeconomichosueingvaluator")

df = pd.read_csv(CODE_DIR / "final_data_house_sales2025.csv", encoding="latin1")
df = df.drop(columns=["LSOA"])
df = pd.get_dummies(df, columns=["Type"], drop_first=True)   # Type_F, Type_S, Type_T vs. Detached baseline

X = df.drop(columns=["Paid Price", "Postcode", "Address"])
y = df["Paid Price"]

X_train, X_test, y_train, y_test, addr_train, addr_test, pc_train, pc_test = train_test_split(
    X, y, df["Address"], df["Postcode"],
    test_size=0.2, random_state=88,
)
print(f"{len(df):,} properties -> {len(X_train):,} train / {len(X_test):,} test")
X_train.head()

16,997 properties -> 13,597 train / 3,400 test

Five models, one dataset

Every model below trains and tests on an 80/20 split. Two are linear: a straight line through the features, and the same line with a penalty added to stop the coefficients running wild with enthusiasm. Three are built to catch the kind of non-linear, interacting relationships a straight line simply can't:

def rmse(y_true, y_pred):
    return np.sqrt(mean_squared_error(y_true, y_pred))

scaler = StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Hyperparameters as tuned in the dissertation's own model.py
models = {
    "Linear Regression": (LinearRegression(), False),
    "Ridge Regression":  (Ridge(alpha=50), False),
    "Random Forest":     (RandomForestRegressor(n_estimators=100, max_depth=None,
                                                 min_samples_split=5, min_samples_leaf=2,
                                                 random_state=50), False),
    "Gradient Boosting": (GradientBoostingRegressor(n_estimators=300, learning_rate=0.1,
                                                     max_depth=4, min_samples_split=5,
                                                     min_samples_leaf=2, random_state=50), False),
    "Neural Network":    (MLPRegressor(hidden_layer_sizes=(100, 50), activation="relu",
                                        solver="adam", learning_rate_init=0.01, max_iter=500,
                                        random_state=50, alpha=0.001), True),   # scaled input
}

predictions, rows = {}, []
for name, (model, needs_scaling) in models.items():
    Xtr, Xte = (X_train_scaled, X_test_scaled) if needs_scaling else (X_train, X_test)
    model.fit(Xtr, y_train)
    y_pred = model.predict(Xte)
    predictions[name] = y_pred
    rows.append({"Model": name, "RMSE (£)": round(rmse(y_test, y_pred)),
                 "MAE (£)": round(mean_absolute_error(y_test, y_pred)),
                 "R2": round(r2_score(y_test, y_pred), 3)})

results = pd.DataFrame(rows).sort_values("RMSE (£)").reset_index(drop=True)
results

Linear and Ridge Regression land in almost the same place, which isn't much of a surprise: Ridge is just Linear Regression with a penalty on large coefficients, and here that penalty barely lifts a finger. All three non-linear models beat both by a wide margin. Random Forest and the Neural Network come out close to tied: Random Forest wins on MAE (a lower typical error), the Neural Network wins on RMSE and R² (fewer really big misses), and Gradient Boosting trails a clear step behind both while still leaving the two linear models for dust. Random Forest is the one actually running the live web app, on the grounds that it wins on MAE.

That's the full model comparison, and also the last time this post treats the five as rivals. What's more interesting than which one wins is a mistake every single one of them makes.

Where the straight line breaks first: the opposite end of the country

Before getting to London, it's worth seeing what the linear models get wrong first, since it turns out to be a mirror image of what's coming. A straight line fitted to minimise overall error will always extrapolate past the point where the relationship it learned still makes any sense, and pushed far enough into deprived, low-income, high-crime territory, that extrapolation cheerfully crosses zero altogether:

linear_check = pd.DataFrame({
    "Address": addr_test.values,
    "Postcode": pc_test.values,
    "Actual (£)": y_test.values,
    "Linear predicted (£)": predictions["Linear Regression"],
}).sort_values("Linear predicted (£)").head(8).reset_index(drop=True)
linear_check

Every one of Linear Regression's worst underpredictions is a cheap flat or terrace in the North of England: FY1 and FY4 are Blackpool; BB9 is Nelson, in Lancashire's old cotton-mill belt; WA9 is St Helens, a former glass-and-chemicals town in Merseyside; L36 is Huyton, on Liverpool's eastern edge; further down the list sit ST2 (Stoke-on-Trent) and HU4 (Hull). Blackpool is not, it turns out, a borderline case: on the 2025 Indices of Deprivation it's ranked the single most deprived local authority in England, first nationally on employment and health, second on crime (GOV.UK Indices of Deprivation 2025; Local Deprivation Explorer), which happens to be the exact dataset this model trained on.

The model isn't wrong to price that profile low; it's wrong about how low, because it's still drawing one straight line all the way from Kensington's income and school figures down to Blackpool's, and a line steep enough to fit the top of that range overshoots badly at the bottom. A model with no concept of a floor at zero will happily inform you that a house is worth minus a quarter of a million pounds, if that's what the slope demands at that end of the data. Rather humorously, it does suggest that the regression models consider these parts of England so undesirable you should only move to them if you were paid to do so, though on a more serious note, it is a nonsense suggestion. The other 3 models do not have such folly, although a problem of a similar ilk (but different magnitude) does arise

The mistake every model makes, in the same handful of postcodes

Moving from a straight line to Random Forest, Gradient Boosting or a Neural Network does fix that specific problem: none of the three ever predicts a negative price, since tree splits and a learned network function aren't forced through one global slope the way a straight line is. But all three independently land on a different failure, in exactly the same postcodes, and this time it's the opposite end of the country entirely:

def most_underpredicted(name, n=8):
    resid = predictions[name] - y_test.values
    idx = np.argsort(resid)[:n]
    return set(addr_test.values[idx])

# addresses appearing in every non-linear model's own worst-underpredicted list
shared = most_underpredicted("Random Forest") & most_underpredicted("Gradient Boosting") & most_underpredicted("Neural Network")

rows = []
for addr in shared:
    mask = addr_test.values == addr
    row = {"Address": addr, "Postcode": pc_test.values[mask][0], "Actual (£)": int(y_test.values[mask][0])}
    for name in ["Random Forest", "Gradient Boosting", "Neural Network"]:
        row[name] = round(predictions[name][mask][0])
    rows.append(row)

shared_underpredicted = pd.DataFrame(rows).sort_values("Actual (£)", ascending=False).reset_index(drop=True)
shared_underpredicted

Every one of these addresses sits in inner or well-to-do outer London: NW8 is St John's Wood, W11 is Notting Hill, SW7 is South Kensington, TW9 is Richmond upon Thames. All three flexible models independently put it among their own worst underpredictions, without ever once being told to go looking for London specifically. 73 Marlborough Place, NW8, sold for £8.6 million; every model's best guess, built from that property's actual crime, income, education, employment and health figures, lands somewhere between £2.7 and £3.8 million, which is a multi-million-pound gap that has nothing to do with the models being unsophisticated, since these are precisely the models built to catch this kind of non-linear, location-driven effect.

There's a perfectly mechanical reason Random Forest in particular does this: it predicts by averaging a lot of decision trees, each trained on a bootstrap sample, which structurally drags extreme, rare values back toward the mean of whatever similar-looking properties it's actually seen plenty of. But that only pushes the real question back a step: why is a handful of London addresses so rare and so extreme, relative to everywhere else with a similar income, school and crime profile, that every flexible model quietly files them under the same kind of outlier?

worst = shared_underpredicted.iloc[0]
gap = worst["Actual (£)"] - worst["Random Forest"]
income_coefficient = 7.51   # £ of house price per £1 of local income, from the linear model above

print(f"{worst['Address']}: actual £{worst['Actual (£)']:,} vs. Random Forest's £{worst['Random Forest']:,}")
print(f"Shortfall: £{gap:,.0f} — equivalent to {gap / income_coefficient:,.0f} separate £1-of-local-income increases")

73 MARLBOROUGH PLACE: actual £8,600,000 vs. Random Forest's £2,695,406
Shortfall: £5,904,594 — equivalent to 786,231 separate £1-of-local-income increases

Random Forest's shortfall on that one property alone works out to the equivalent of over three-quarters of a million separate £1-of-local-income increases, all stacked on a single street. Income, schools, crime and employment explain a genuinely large share of what a house is worth almost everywhere in England, and then this one handful of addresses carries a premium built out of none of those things, at a scale no amount of model sophistication ever quite closes.

London isn't an expensive version of England. It's a different market.

In December 2025, the average house in England sold for £292,000 (GOV.UK: UK House Price Index, England, December 2025). London's own average, from the same release two months earlier, was £547,000, nearly double the England-wide figure. That gap isn't simply "London vs. everywhere else," either; it holds up inside London too. Kensington and Chelsea, the most expensive local authority in the December 2025 release, averaged £1,178,497. Barking and Dagenham, London's cheapest borough, averaged £353,512, which is still above the England-wide average of £292,000, but somewhat less of an arbitrarily large number. For comparison, Kingston upon Hull (one of the cheapest areas in England, and the same HU postcode area that showed up among Linear Regression's worst underpredictions above) averaged £131,323 in the same release, roughly a ninefold gap between Hull and Kensington and Chelsea.

That scale of gap, England's cheapest and most expensive local authorities sitting roughly nine times apart, is exactly what sends a straight line into negative numbers at Blackpool's end of the distribution, and exactly what three structurally different flexible models still can't close at the other end, no matter how clever they get. Indeed, the magnitude in errors seen when evaluating the models can mostly be explained by how wrong they are about London.

The dissertation arrives at the same conclusion from the results themselves, rather than assuming it going in: every model, however flexible, systematically underprices the same prime London properties, and the fix isn't a sixth algorithm, however tempting that always feels. It's treating London as its own market, not a pricier tail of the national one. That could mean adding distance to central London as its own feature, or simply giving the capital a model of its own and letting the rest of England get on with things separately. Sophistication bought real accuracy almost everywhere else in the country. In St John's Wood, Notting Hill and South Kensington, all it bought was a smaller wrong answer.

The Random Forest model itself is live at socioeconomichousevaluationweb.uk: enter an LSOA code, property type and floor area, or just click the map, for an actual prediction from the model this whole post has been about. Though if you're to move to St John's Wood, it's probably not your best tool.