Guides to scheduled and batch web collection
One guide per stage of the pipeline, from the list of URLs to the model step. Each ends with what it means for your workflow and links to the shorter posts.
n8n workflows
n8n web scraping: a reliable URL-list workflow
HTTP Request node, batching, execution limits, error routing and Sheets output: the n8n settings that keep a URL-list workflow honest.
1 related post
Make and Zapier
Zapier and Make web scraping: HTTP steps and iterators
The generic HTTP step in Make and Zapier, how each tool loops over a list, and a neutral comparison with n8n for page collection.
0 related posts
URL lists and batch jobs
Scrape a list of URLs: CSV in, JSONL out, safe retries
Normalize, dedupe, size the run, retry by error type and write one JSONL line per URL, so a failed batch can resume where it stopped.
0 related posts
Scheduling
Scheduled web scraping with cron and GitHub Actions
Cron, GitHub Actions or an n8n Schedule Trigger, refresh windows for repeat runs, change alerts and checks for the failures nobody sees.
0 related posts
The AI step
The AI scraper workflow: fetch first, then ask your model
Collection and interpretation are two jobs. Fetch clean Markdown, then extract or classify with your own model, and know when an agent is worth it.
0 related posts
Collecting responsibly
Responsible web scraping: robots.txt, terms, personal data
What to check before you schedule a collection job: robots.txt, terms of service, personal data, polite rates and the acceptable-use policy.
1 related post