What a house is worth, and why London breaks the curve

The gap the project is trying to close

Property listing sites are data-rich about the property itself — square footage, photos, floor plans, asking price — and comparatively thin on everything around it. Rightmove shows a map of nearby schools without saying whether they're any good; Zoopla's valuation tool leans almost entirely on nearby sold prices. Neither tells a buyer relocating to an unfamiliar area much about the deprivation, crime, or income profile of the neighbourhood they're about to move into, and asking a local forum tends to return anecdote rather than anything comparable across the country.

The project's response is to build a house-price model that takes socioeconomic geography seriously as an input in its own right, not just an afterthought to square footage and property type — and to see how much of a house's price that geography actually explains.

Data: property records, meet the census

England was the target (Scotland and Northern Ireland record small-area socioeconomic data under different systems, so pooling all three would mean reconciling incompatible geographies for comparatively little benefit — the great majority of the UK's internal relocators and new arrivals move to England anyway). Two datasets, joined at LSOA/MSOA level:

• HM Land Registry Price Paid Data, November 2025: residential sale transactions, each with property type, sale price and postcode.
• The English Indices of Deprivation (2024–25), plus targeted replacements for two of its seven domains. The IoD's own Income domain only measures the proportion of people on a low income — an area with one very poor resident and everyone else wealthy would register as more deprived than one where everybody earns exactly the national average, which is the wrong shape of variable for pricing a house. Actual small-area income estimates were used instead. The Education domain has the mirror problem (it measures the share lacking qualifications, not the presence of good schools), so it's replaced with Attainment 8 — the average GCSE score by Local Authority, weighted to pupil residence rather than school location, so it reflects catchment rather than which school happens to sit inside a boundary.

Land Registry doesn't record floor area, so each sale was matched to its Energy Performance Certificate for that figure instead. The code and the final merged dataset behind every table in this post are the dissertation's own:

import pandas as pd
import numpy as np
from pathlib import Path
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

CODE_DIR = Path("../socioeconomichosueingvaluator")

df = pd.read_csv(CODE_DIR / "final_data_house_sales2025.csv", encoding="latin1")
df = df.drop(columns=["LSOA"])
df = pd.get_dummies(df, columns=["Type"], drop_first=True)   # Type_F, Type_S, Type_T vs. Detached baseline

X = df.drop(columns=["Paid Price", "Postcode", "Address"])
y = df["Paid Price"]

X_train, X_test, y_train, y_test, addr_train, addr_test, pc_train, pc_test = train_test_split(
    X, y, df["Address"], df["Postcode"],
    test_size=0.2, random_state=88,
)
print(f"{len(df):,} properties -> {len(X_train):,} train / {len(X_test):,} test")
X_train.head()

16,997 properties -> 13,597 train / 3,400 test

Five models, the same dataset

Every model below is trained and tested on the identical 80/20 split of that dataset, so the comparison is clean. Two are linear (a straight line through the features, one with a regularisation penalty added); three are built to capture the kind of non-linear, interacting relationships a straight line structurally can't:

def rmse(y_true, y_pred):
    return np.sqrt(mean_squared_error(y_true, y_pred))

scaler = StandardScaler().fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Hyperparameters as tuned in the dissertation's own model.py
models = {
    "Linear Regression": (LinearRegression(), False),
    "Ridge Regression":  (Ridge(alpha=50), False),
    "Random Forest":     (RandomForestRegressor(n_estimators=100, max_depth=None,
                                                 min_samples_split=5, min_samples_leaf=2,
                                                 random_state=50), False),
    "Gradient Boosting": (GradientBoostingRegressor(n_estimators=300, learning_rate=0.1,
                                                     max_depth=4, min_samples_split=5,
                                                     min_samples_leaf=2, random_state=50), False),
    "Neural Network":    (MLPRegressor(hidden_layer_sizes=(100, 50), activation="relu",
                                        solver="adam", learning_rate_init=0.01, max_iter=500,
                                        random_state=50, alpha=0.001), True),   # scaled input
}

predictions, rows = {}, []
for name, (model, needs_scaling) in models.items():
    Xtr, Xte = (X_train_scaled, X_test_scaled) if needs_scaling else (X_train, X_test)
    model.fit(Xtr, y_train)
    y_pred = model.predict(Xte)
    predictions[name] = y_pred
    rows.append({"Model": name, "RMSE (£)": round(rmse(y_test, y_pred)),
                 "MAE (£)": round(mean_absolute_error(y_test, y_pred)),
                 "R2": round(r2_score(y_test, y_pred), 3)})

results = pd.DataFrame(rows).sort_values("RMSE (£)").reset_index(drop=True)
results

Linear and Ridge Regression land almost identically (unsurprising — Ridge is Linear Regression with large coefficients penalised, and here that penalty barely moves the needle). All three non-linear models improve on both by a wide margin. Random Forest and the Neural Network come out close to tied — Random Forest wins on MAE (a lower typical error), the Neural Network wins on RMSE and R² (fewer really large misses) — with Gradient Boosting a clear step behind both but still well ahead of the two linear models. Random Forest is the one actually wired into the project's web application, on the strength of that MAE and its faster, more robust training.

That's the full model comparison, and it's the last time this post treats the five as competitors. What's more interesting than which one wins is a failure they share.

Where the straight line breaks first: the opposite end of England

Before getting to London, it's worth seeing what the linear models get wrong, because it's the mirror image of what's coming. A straight line fitted to minimise overall error inevitably extrapolates past the point where the relationship it learned still makes sense — and pushed far enough into deprived, low-income, high-crime territory, that extrapolation crosses zero entirely:

linear_check = pd.DataFrame({
    "Address": addr_test.values,
    "Postcode": pc_test.values,
    "Actual (£)": y_test.values,
    "Linear predicted (£)": predictions["Linear Regression"],
}).sort_values("Linear predicted (£)").head(8).reset_index(drop=True)
linear_check

Every one of the most-underpredicted properties by Linear Regression is a cheap flat or terrace outside the South East, and the postcodes are a tour of England's post-industrial and seaside towns: FY1 and FY4 are Blackpool; BB9 is Nelson, in Lancashire's old cotton-mill belt; WA9 is St Helens, a former glass-and-chemicals town in Merseyside; L36 is Huyton, on Liverpool's eastern edge; further down the same list sit ST2 (Stoke-on-Trent) and HU4 (Hull). Blackpool in particular isn't a borderline case: on the 2025 Indices of Deprivation it's ranked as the single most deprived local authority in England — first nationally on the employment and health domains, second on crime (GOV.UK Indices of Deprivation 2025; Local Deprivation Explorer) — the exact dataset this model was trained on.

The model isn't wrong to associate that profile with a low house price — it's wrong about how low, because it's still trying to draw one straight line all the way from Kensington's income and school figures down to Blackpool's, and a line steep enough to capture the top of that range overshoots badly at the bottom. A model with no concept of a floor at zero, minimising squared error, will happily predict a house is worth minus a quarter of a million pounds if that's what the slope says at that end of the data — the same structural inflexibility that, at the other end of the income and price distribution entirely, is about to show up again in a different shape.

The one mistake every model makes, in the same handful of postcodes

Moving from a straight line to Random Forest, Gradient Boosting or a Neural Network fixes that specific problem: none of the three non-linear models ever predicts a negative house price, because tree-based splits and a network's learned function aren't forced through one global slope the way a straight line is. But all three independently converge on a different failure, in exactly the same postcodes — and this time it's the opposite end of the country entirely:

def most_underpredicted(name, n=8):
    resid = predictions[name] - y_test.values
    idx = np.argsort(resid)[:n]
    return set(addr_test.values[idx])

# addresses appearing in every non-linear model's own worst-underpredicted list
shared = most_underpredicted("Random Forest") & most_underpredicted("Gradient Boosting") & most_underpredicted("Neural Network")

rows = []
for addr in shared:
    mask = addr_test.values == addr
    row = {"Address": addr, "Postcode": pc_test.values[mask][0], "Actual (£)": int(y_test.values[mask][0])}
    for name in ["Random Forest", "Gradient Boosting", "Neural Network"]:
        row[name] = round(predictions[name][mask][0])
    rows.append(row)

shared_underpredicted = pd.DataFrame(rows).sort_values("Actual (£)", ascending=False).reset_index(drop=True)
shared_underpredicted

Every one of these addresses is in inner or well-to-do outer London — NW8 is St John's Wood, W11 is Notting Hill, SW7 is South Kensington, TW9 is Richmond upon Thames — and every one of the three flexible models independently placed it among its own worst underpredictions, without being told to look for London specifically. 73 Marlborough Place, NW8, sold for £8.6 million; every model's best estimate, built from that property's actual crime, income, education, employment and health figures, lands somewhere between £2.7 and £3.8 million — a multi-million-pound gap that has nothing to do with the model being unsophisticated, since these are precisely the models built to catch exactly this kind of non-linear, location-driven effect.

There's a mechanical reason a Random Forest in particular converges on this pattern: it predicts by averaging many decision trees, each trained on a bootstrap sample, which structurally pulls extreme, rare values back toward the mean of whatever similar-looking properties it has seen plenty of. But that mechanical explanation only pushes the real question back a step — why is a handful of London addresses so rare and so extreme, relative to everywhere else with a similar income, school and crime profile, that every flexible model independently treats them as the same kind of outlier?

worst = shared_underpredicted.iloc[0]
gap = worst["Actual (£)"] - worst["Random Forest"]
income_coefficient = 7.51   # £ of house price per £1 of local income, from the linear model above

print(f"{worst['Address']}: actual £{worst['Actual (£)']:,} vs. Random Forest's £{worst['Random Forest']:,}")
print(f"Shortfall: £{gap:,.0f} — equivalent to {gap / income_coefficient:,.0f} separate £1-of-local-income increases")

73 MARLBOROUGH PLACE: actual £8,600,000 vs. Random Forest's £2,695,406
Shortfall: £5,904,594 — equivalent to 786,231 separate £1-of-local-income increases

Random Forest's own shortfall on that one property is the equivalent of over three-quarters of a million separate £1-of-local-income increases, stacked on a single street. Income, schools, crime and employment explain a genuinely large share of what a house is worth almost everywhere in England — and then this specific handful of addresses carries a premium that isn't made of any of those things, at a scale no amount of model sophistication closes.

London isn't an expensive version of England. It's a different market.

Officially, in December 2025, the average house in England sold for £292,000 (GOV.UK: UK House Price Index, England, December 2025). London's own average, the same release series two months earlier, was £547,000 — nearly double the England-wide figure. That gap isn't just "London vs. everywhere else" either — it holds up inside London. Kensington and Chelsea, the most expensive local authority in the December 2025 release, averaged £1,178,497. Barking and Dagenham, London's cheapest borough, averaged £353,512 — still above the England-wide average of £292,000. The cheapest borough in the most expensive city in the country is still pricier than the national average. For comparison, Kingston upon Hull — one of the cheapest areas in England, and the same HU postcode area that showed up in Linear Regression's worst underpredictions above — averaged £131,323 in the same release: roughly a ninefold gap between Hull and Kensington and Chelsea, inside one country, one currency, one housing market on paper.

That scale of gap — England's cheapest and most expensive local authorities sitting roughly nine times apart — is exactly what sends a straight line into negative numbers at Blackpool's end of the distribution, and exactly what three structurally different flexible models still can't close at the other end, however sophisticated they get.

The dissertation's own conclusion reaches the same place from the results rather than assuming it going in: every model, however flexible, systematically underprices the same prime London properties, and the proposed fix isn't a sixth algorithm — it's treating London as its own market rather than a more-expensive tail of the national one, for instance by adding distance to central London as its own feature, or fitting a separate model for the capital entirely. Sophistication bought real accuracy almost everywhere else in the country; in St John's Wood, Notting Hill and South Kensington, it bought a smaller wrong answer, not a right one.

The Random Forest model itself is live at socioeconomichousevaluationweb.uk — enter an LSOA code, property type and floor area (or click the map) for a real prediction from the model this post is about.