Scrape multiple URLs in n8n: Batching or Loop Over Items

An n8n workflow that reads 300 URLs from a Sheet and sends them to an HTTP Request node will try to fetch all 300 as fast as it can. Any fetch API with a rate limit answers that with 429, and the run either stops or fills your sheet with errors. There are two ways to pace it, and they suit different workflows.

The examples use Live Web Data API, whose rate limit counts URLs per second. The same settings apply to any API that limits calls per second.

Option 1: Batching on the HTTP Request node

Open the HTTP Request node, then Options → Batching:

n8n then sends that many items, waits the interval and sends the next batch. Nothing else in the workflow changes. This is the right choice when only the fetch needs pacing and the rest of the workflow is fast.

The request body sends one URL per item:

{ "urls": ["{{ $json.url }}"] }

The full node settings (Bearer Auth credential, JSON body, response format) are in the no-code recipes.

Option 2: Loop Over Items with a Wait node

Loop Over Items (formerly Split in Batches) hands a fixed number of items at a time to the nodes inside the loop. Put a Wait node at the end of each pass and connect it back to the loop:

1. Loop Over Items, Batch Size = your plan's rate limit in URLs per second.
2. HTTP Request (the fetch).
3. IF on {{ $json.statuses[0].status }} equals success.
4. Your AI step and the Google Sheets update on the success branch; an error row on the other.
5. Wait, Resume = After Time Interval, 1 second, then back to Loop Over Items.

This is the right choice when several nodes must finish for one batch before the next starts, for example when each page also goes through a model and a Sheets write. The wait then covers the whole pass, so the pace is a little slower than the limit, which is fine.

How long will the run take?

Pacing sets a floor on run time. With one URL per item, the minimum is the number of URLs divided by your rate limit:

Retries and slow pages add to that. If the number gets close to your instance's execution timeout, split the list: a parent workflow sends chunks of 20 to 50 URLs to a child workflow with Execute Sub-workflow, so each execution stays short. The n8n web scraping guide covers that setup.

When you still see 429

Pacing removes most 429s; it does not remove all of them, because other workflows may share the same key. Handle the rest with the same retry policy every guide on this site uses: retry 429, 5xx and CRAWL_LIVECRAWL_TIMEOUT up to 3 times with backoff, retry no other per-URL error in the same run, and stop the job on 402, 401 or 403. On a 429, wait the number of seconds in the Retry-After header before trying again. The full table is in the retry rules for URL lists.

If 429s show up on every run, lower Items per Batch or Batch Size instead of adding retries. A separate API key for the n8n workflows also makes it easier to see which job uses the budget.

What this means for your workflow

• One URL per item, so routing and retries work per URL.
• Only the fetch needs pacing: Options → Batching, Items per Batch = rate limit, interval 1000 ms.
• Several nodes per batch: Loop Over Items plus a one-second Wait.
• Estimate run time before you schedule; split long lists into sub-workflow executions.
• Lower the pace when 429s repeat; do not lean on retries.