n8n web scraping: a reliable URL-list workflow

With a list of public URLs, n8n workflows rarely fail on hard sites. They fail at item 40, when a run hits a time limit; at a node that reports success with an empty body; and in a Google Sheet that slowly fills with blanks. This guide builds a URL-list workflow that avoids those, with node names exactly as n8n shows them. It is the fetch half of an agent scraper pipeline: the model step comes after.

The two-node baseline

Every n8n web scraping workflow is the same two moves: get an item per URL, then make one HTTP call per item.

1. A trigger. Use Manual Trigger while you build and Schedule Trigger once it works.
2. An HTTP Request node. For a raw page you would call GET https://example.com/page and parse the HTML with the HTML node and CSS selectors. That breaks every time the page layout changes, and you still have to strip menus, footers and cookie banners yourself.

The alternative is to call a fetch API that returns the main content as Markdown. With Live Web Data API the HTTP Request node is set up like this (the full table is in the no-code recipes):

{ "urls": ["{{ $json.url }}"] }

The response holds the page in results[0].markdown and the outcome in statuses[0].status and statuses[0].error.tag. Keep the key in an n8n credential, not in a plain header field, so it does not end up in exported workflow JSON.

URL lists and batching

Once the trigger produces many items, the HTTP Request node runs once per item, as fast as it can. Any API with a rate limit will start answering 429. You have two ways to pace it.

Options → Batching on the HTTP Request node. Set Items per Batch to your plan's rate limit, which counts URLs per second (one URL per item), and Batch Interval (ms) to 1000. This is the simplest fix when only one node needs pacing.

Loop Over Items (the node formerly called Split in Batches) plus a Wait node. Use it when several nodes must run per batch, for example fetch, AI step and Sheets update. Put a Wait of one second at the end of each pass and connect it back to the loop.

Batching is about the provider's limit, but also about the target sites. A polite rate is part of responsible web collection, not just a way to avoid errors.

Execution limits and sub-workflows

Long runs fail in n8n for three common reasons:

• Execution timeout. Self-hosted instances read EXECUTIONS_TIMEOUT (seconds, default -1, meaning no limit) and EXECUTIONS_TIMEOUT_MAX. n8n Cloud plans set their own limits; check your plan.
• Memory. Every item carries the full page through every later node. A few hundred pages of Markdown in one execution adds up fast.
• One failed node ends the run. By default a node error stops the workflow, so item 41 never runs.

The fix is the same for all three: make each execution small. A parent workflow reads the list and sends chunks of, say, 20 to 50 URLs to a child workflow with the Execute Workflow node (named Execute Sub-workflow in newer versions). Each child execution fetches, processes and writes its own rows, then ends. The parent only holds URLs, never page content.

Also cap the text that comes back. The API's text.maxCharacters defaults to 100,000 characters, enough for whole pages; set it lower, for example 8000, so very long pages do not bloat every item.

Error routing on per-URL status

This is the part most guides skip. A fetch API that handles a list reports per-URL failures inside a normal 200 response. If your workflow only checks the HTTP code, error pages flow into your Sheet as if they were content.

Add an IF node after the fetch with the condition {{ $json.statuses[0].status }} equals success. Then decide per error tag. The Retry? column is the same policy as the retry rules table and the batch scripts in the docs:

Whole-request errors (429, 5xx, 402, 401, 403) arrive with a real HTTP code. For those, open the node's Settings, set On Error to Continue (using error output) and send that branch to a dead-letter sheet. Retry On Fail in the same settings retries every error the node raises, 402 included, and does not read Retry-After. To match the policy exactly, route 429 and 5xx from the error output through a Wait node and back to the fetch, counting attempts in a field and giving up after 3 retries. Failed pages cost 0 credits, so retries are cheap, but retrying a restricted target can get a key suspended.

Writing to Google Sheets

Write one row per URL, not one row per success. Use the Google Sheets node with Append or Update Row and the URL as the matching column. Useful columns:

• url, status (success or the error tag), fetched_at, title
• summary or label from the AI step
• attempts, so a retry branch knows when to give up

A status column that tells the truth is the cheapest monitoring there is. Filter the sheet for anything that is not success and you see exactly what the last run missed. On the next run, read only rows whose status is empty, an error worth retrying, or older than your refresh window. That makes the whole workflow resumable.

Adding an AI step

Put the model after the IF node, on the success branch only. Send results[0].markdown, not the raw HTML: it is shorter and the model does not have to guess which part is navigation. Ask for a fixed output shape (a two-sentence summary, a label from a short list, or a JSON object with named fields) and parse it before you write it to the sheet.

The AI scraper workflow guide covers prompts, input caps and how to keep fetch costs and model costs apart.

What this means for your workflow

• One item per URL, paced with Batching or Loop Over Items plus Wait.
• Small executions: a parent with the list, a child per chunk of 20 to 50 URLs.
• Retry 429, 5xx and timeouts up to 3 times; retry nothing else in the same run; stop on 402.
• Route on statuses[0].status and the error tag, never on the HTTP code alone.
• One sheet row per URL with a status column; read only unfinished rows on the next run.
• The model goes after the fetch, on clean Markdown, on the success branch.

For a schedule that runs this every morning, continue with scheduled web scraping. For the same pattern in Make or Zapier, see Zapier and Make web scraping.