PR #252 deferred task · Webpage collectors stop parsing HTML; ContentExtractionManager (the consumer's extraction tier) owns the title too.
<title>,
publish the re-serialized HTML… and the consumer parses the exact same HTML again for content/author/date.
Title is load-bearing (the consumer requires it) so it can't be deleted — it gets relocated.
BEFORE — every article parsed twice
AFTER — parsed once, where extraction lives
fetch_webpage())response.text verbatimextract_content returns a 4-tupleTitle joins content/author/date — computed once in the manager via the existing extract_title helper. The ~150 per-site extractor classes are untouched; 6 call sites get a mechanical unpack update.
On-demand: original_request.article_title (the RSS title — already in the message). Periodic/sitemap: webpage_name. Extraction-side, "all extraction in one place".
fetch_webpage_soup + str(soup)After the switch it has zero callers → gone. Publishing response.text also kills the soup re-serialization, so the consumer sees the page byte-for-byte as served.
The periodic collector still soups its index page to discover article links. That's collection logic, not content extraction — and it's bounded: one page per source per cycle. Documented in-code.
While in the file (fix-when-touching rule): csrf["value"] KeyError risk → .get() guards, and the consent POST finally gets raise_for_status() — silent failures become loud.
Open PR #284 edits the same fetch/auth glue (async_http.py, collector cookie/reauth wiring). If it merges first → rebase this branch and re-verify _fetch_response. Task 2 has no overlap with either.
PR #280 reparented periodic webpage + sitemap under a shared AsyncPeriodicSiteCollector base — the soup/title glue lives there now, so this change lands once in the base and covers both. (The spec predates this; call sites were re-verified against current dev.)
The consumer historically juggled title (webpage) vs article_title (on-demand). Post-move it sets one normalized article_title from extraction for both paths — the dual-field dance disappears.
title field is just ignored) —
but a NEW collector message would crash an OLD consumer. So: collector + consumer deploy in the same compose rollout;
safest is a short drain of webpage-content + article-content first.