Feed fetching

Outbound requests

src/server/utils/fetch.ts uses Undici with an identified Sparkfeed User-Agent. Only public HTTP and HTTPS addresses are accepted. Every redirect is checked, and the socket’s DNS lookup rejects private addresses to prevent DNS rebinding.

Text responses have a default 5 MiB limit and a 10-second timeout. Individual callers can set smaller limits or different deadlines. HTTP charset, XML encoding declarations, and UTF-16 byte-order marks inform decoding.

host-limit.ts allows three concurrent text downloads per requested hostname, with bounded, cancellable waiting. The HTTP connection pool also limits connections per origin.

Refresh and persistence

fetchAndInsertArticles stores ETag and Last-Modified validators only after successful ingestion. A 304 response preserves stored articles and counts as a successful check. Failures preserve the previous successful state and record source health.

RSS imports consider at most 100 entries from the returned feed per refresh, without a publication-age cutoff. This is a bounded recent-feed import, not a promise to archive all publisher history. Publisher IDs and article URLs identify existing rows. Updates preserve article IDs and user favorites. Optional image enrichment runs after article persistence and does not delay refresh completion.

Website sources use extractPageLinks and Readability. A listing yields at most 40 links. All detected links are stored, while at most 12 new or retried article bodies are fetched per run. Remaining bodies can be fetched when opened.

Queues and retries

The ingestion queue runs four sources concurrently and deduplicates in-flight work by source ID. It is process-local, not a durable job queue.

The dashboard’s automatic refresh uses the one-hour freshness policy. Discover keeps successful previews for six hours and backs off failed previews for five minutes. Opening a saved reader article does not probe the publisher; Live availability is checked separately when requested.