Automated valuation, model governance and credit-risk transmission

The first-pass pipeline was unmeasurable, and scored worse than the sample mean

An earlier modelling pass on this dataset (three notebooks, one regularised regression, one reported accuracy figure) carried ten defects, four of which invalidated the reported results. The most consequential assigned training rows into the held-out test frame by index alignment, so every reported test metric described data the model had already seen.

None of the defects raised an error. The code ran, the diagnostics looked plausible, and the held-out prices had been withheld, so nothing could be checked against them. The De Cock (2011) publication the extract comes from still has those prices. Rescored against them, the original predictions give MdAPE 36.9% and RMSE $112,154, against $24,996 for the rebuilt pipeline. Each defect and its remediation is recorded in the model risk log.

Performance against institutional acceptance thresholds

AVMs are judged on how close most valuations land, not on variance explained. The two standard measures have conventional thresholds used by secondary-market buyers. Both are shown at their measured values on the 2009–2010 test sales.

Interval estimates, and their measured coverage

Each valuation comes with an 80% range (split-conformal, proportional to the value). The width is set separately for five price bands and re-fitted each quarter on the latest twelve months of closed sales. As served, the bands run from 21% to 30% of value.

Calibrated once at launch, the range contained the sale price 76.7% of the time against the 80% target. Re-fitted each quarter, it contains it 78.8% of the time. Conformal intervals assume the calibration sales resemble the ones being valued, and a forward-in-time test breaks that. On a shuffled split the same method reaches 80.1%, so the shortfall comes from the market changing, not from the method. The model card and every API response report the measured figure.

A single half-width also fails by segment: the constant interval covers the cheapest sale-price decile only 51.2% of the time. Mondrian conformal, calibrated separately per valuation quintile, lifts that to 62.6% for 0.6 points of added width. Conformalized quantile regression is the only method evaluated that restores the marginal promise, at 79.8%, for 37% more width. The same quantile models without the conformal step cover 57.7%.

Systematic error by price band, neighbourhood and location

The 2024 interagency rule on automated valuation models asks whether a model produces confident estimates, resists manipulation, is tested on random samples, and does not discriminate. The results:

• The price-band bias is mostly a measurement artefact. Grouped by sale price, the cheapest decile shows 10.2% mean signed error and the top -3.8%; a model unbiased by construction shows 7.6% and -4.2% under the same grouping. Grouped by the valuation a lender observes, error stays within ±2.8%. The remaining bias is about 3% at the cheap end and nil at the top; a client-report flag built on the uncorrected figure was withdrawn.
• Accuracy varies by neighbourhood across a 10.1% spread in mean signed error, so valuation quality is conditional on location.
• Errors cluster in space (Moran’s I 0.114, p = 5e-13). The neighbourhood label has not absorbed location, which is a known and unfixed weakness.

Collateral is secured against a valuation, not a price

Every loan in this constructed book is written at 80% loan-to-value on the model’s valuation. Measured against what the houses actually sold for, none is at 80%, and 10.4% are above 90%.

Under the Federal Reserve’s severely adverse scenario (a 30% fall in house prices), expected losses go from 88 bps to 1,183 bps, a 13.4× increase, and the share of borrowers underwater goes from 2.7% to 93.9%. Valuing collateral at the low end of the model’s range instead of its point estimate adds a further 75 bps at base and 250 bps under stress. This link between valuation error and loan losses is why regulators set quality-control standards for AVMs.

The market does not support rental underwriting at current prices

Underwriting was pre-registered against a 4–8% cap-rate band, the range at which residential rental acquisition is conventionally financeable. Measured across 2,412 properties, the median cap rate is 2.03%, median DSCR 0.33, and 54.4% of Monte Carlo paths terminate below zero. Only 0.17% of properties clear a 1.25 debt-service floor.

The mechanism is identifiable in the NOI bridge: property tax at the Story County consolidated levy and insurance together absorb a large share of gross rent before debt service. An acquisition priced to clear 1.25 DSCR requires an entry of $110,491, 64.3% below the current median value. Sensitivity analysis identifies HPA as the dominant driver of terminal IRR. The result is reported as measured; the pre-registered band was not met.

Observed lending outcomes in the same market

The loan book above is constructed, which bounds what it can establish about credit access. This section substitutes observed data: 32,926 HMDA mortgage applications filed in the same MSA, 2018–2024, in which denials, their stated reasons and applicant demographics are all reported.

Black applicants were denied at 24.7% against 11.5% for white applicants, 2.15× the rate. Controlling for income, loan size, loan-to-value, DTI band, loan purpose, lien status, occupancy and year, the adjusted odds of denial remain 1.98× (95% CI 1.24–3.16, p = 0.004), fitted on 17,495 applications.

HMDA records no credit score, the main missing control. Its possible effect is bounded with an E-value of 3.37 (1.78 at the confidence limit): an unmeasured confounder would have to be associated with both race and denial by that much to explain the gap away. Female applicants show 0.98×: the same model finds no gap where none is expected.

16.2% of denials (478) cite collateral: the valuation did not support the loan. That is the same problem priced above in basis points, seen here as credit that was never extended.

HMDA records no credit score. Every adjusted figure is an upper bound on unexplained disparity, not a finding of discrimination. Full tests and caveats in notebook 09.

Worked valuation reports

The one-page report returned by the service, stating the valuation, its interval and the model’s known weaknesses on the document itself.

Further analysis in the repository

• Monitoring. 6 quarters of drift and accuracy control charts. Run on the real non-arm’s-length sales as a negative control, 5 of the six controls fire.
• Transaction screening. A classifier identifying non-arm’s-length sales prior to training-set inclusion (0.80 ROC-AUC, 4.5× PR-AUC lift over the base rate).
• Macro context. Peak-to-trough drawdown of 4.3% in Ames against 13.6% for the median US metro, benchmarked across 410 FHFA metro series.
• The valuation service. FastAPI, containerised, returning a valuation, its interval and its measured coverage per request (make serve or docker run). Not hosted publicly: Hugging Face moved Docker Spaces to a paid tier.