HTTP 200 Is a Lie: A 30-Line Schema Canary for Source Drift
The article discusses the issue of silent source drift in web scraping, where a scraper may return an HTTP 200 status but still produce incorrect data. It emphasizes the importance of monitoring not just the success of the request but also the integrity of the data being scraped. A proposed solution is to implement a contract that defines the expected shape of the data to catch discrepancies early in the process.
- ▪HTTP 200 indicates that the server responded, but does not guarantee the data is correct.
- ▪Silent source drift can lead to incorrect data being fed into a database without any error signals.
- ▪Implementing a contract for data shape can help identify issues with scraped data.
Opening excerpt (first ~120 words) tap to expand
try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 3831260) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Alex Spinov Posted on May 30 • Originally published at blog.spinov.online HTTP 200 Is a Lie: A 30-Line Schema Canary for Source Drift #api #dataengineering #python #webscraping A scraper that returns HTTP 200 is not a scraper that returns good data. Those are two different claims, and almost every monitoring setup I've seen conflates them. Here's the failure mode nobody writes code for. The source you scrape quietly changes.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DEV.to (Top).
Discussion
0 commentsMore from DEV.to (Top)
