Scraping 241 UK council planning portals – 2.6M decisions so far
A data scraping project has successfully collected 2.6 million planning decisions from 241 UK councils. The project faced challenges due to the varied configurations of council portals and restrictions on scraping. Findings reveal that the national approval rate for applications is around 88%, but many councils are missing the statutory decision timeframes.
- ▪The scraping project covered 241 councils across England, Scotland, and Wales.
- ▪36.5% of home extension applications missed the statutory 8-week target in 2025, an increase from 27.9% in 2019.
- ▪Guildford had the highest rate of delayed decisions, with 66% of applications exceeding the target timeframe.
Opening excerpt (first ~120 words) tap to expand
I've been scraping 241 UK council planning portals – 2.6M decisions so farUK planning data is technically public. In practice it's locked behind 400+ different council portals, some still running bespoke ASP.NET that looks like it dates from 2004, some behind AWS WAF, all with subtly different schemas. I've spent four months scraping them. I'm now at 241 councils and 2.6 million decisions across England, Scotland and Wales.The scraping problemMost UK councils run one of a handful of portal systems, Idox being the most common. In theory this makes things easy.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at Ycombinator.