· n8n workflows
Scrape multiple URLs in n8n: Batching or Loop Over Items
Two ways to pace a URL list in n8n so a fetch API never answers 429: the HTTP Request node's Batching option, or Loop Over Items with a Wait node.
AgentScraper blog: notes from the collection workshop
One workflow problem per post, with its sources listed at the end and a setting or script you can copy into your own n8n, Make, Zapier or cron job.
RSS feedn8n workflows
Collecting responsibly
· Collecting responsibly
Add a robots.txt check step to your n8n or cron job
A per-host robots.txt check that runs before every scheduled collection job: a Python step for cron and a Code node for n8n, with the RFC 9309 edge cases.