Scrape a list of URLs: CSV in, JSONL out, safe retries

Scraping a list of URLs sounds like a five-line loop. It is, until the list has 3,000 entries, the run dies at row 1,800 and nobody knows which rows made it. This guide turns the loop into a batch job you can trust: a clean list in, one JSON line per URL out, retries by error type, and a run that can resume.

The examples use Python, but the same rules apply to Node, a shell script or a data pipeline tool.

Prepare the list

Most problems in a batch start in the input file.

Normalize every URL first. Lowercase the scheme and host, remove the fragment (#section), drop tracking parameters such as utm_*, and pick one rule for trailing slashes. https://Example.com/docs/ and https://example.com/docs?utm_source=x are the same page.

Dedupe after normalizing. Otherwise the same page gets fetched and billed twice and appears twice in your output.

Remove what you should not collect. Pages behind a login or paywall are out of scope and prohibited by the acceptable-use policy. Lists of profile pages about individual people raise personal data questions. The responsible web collection guide has the checklist.

Keep a stable ID. Use the normalized URL itself, or a hash of it, as the key that ties input rows to output lines.

from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode

def normalize(url: str) -> str:
    s = urlsplit(url.strip())
    query = urlencode([(k, v) for k, v in parse_qsl(s.query) if not k.startswith("utm_")])
    path = s.path.rstrip("/") or "/"
    return urlunsplit((s.scheme.lower(), s.netloc.lower(), path, query, ""))

Size the run

Before you start, answer three questions:

1. How many URLs? After dedupe, not before.
2. How many credits are left? Call GET /v1/account and compare credits.total with the URL count. Each successful page costs credits depending on how it had to be fetched (statuses[].credits shows the number per URL), and failed pages cost 0, so do a small test batch first and multiply.
3. How long will it take? URLs divided by your plan's rate limit, which counts URLs per second (rateLimit.requestsPerSecond in the same response), plus retries. A 3,000-URL list at 2 URLs per second needs at least 25 minutes.

If the balance is too low, split the list or top up first. A job that stops on 402 INSUFFICIENT_CREDITS in the middle is recoverable, but it is better not to start one you know cannot finish.

Concurrency and rate limits

Run with a concurrency equal to your plan's rate limit in URLs per second. You can send up to 10 URLs in one request, but the rate limit counts URLs, so ten URLs in one call use the same budget as ten single calls. Sending one URL per request is simplest and makes retries per URL easy.

On 429 RATE_LIMITED, read the Retry-After header, wait that many seconds and retry the same request. If you get 429s regularly, your concurrency is too high. Lower it rather than leaning on retries.

Retry rules table

Retry by error type, not blindly. This is the one policy every guide on this site uses, and the ready-made batch scripts in Automate requests implement it: retry 429, 5xx and CRAWL_LIVECRAWL_TIMEOUT up to 3 times with backoff, retry no other per-URL error in the same run, and stop the job on 402, 401 or 403.

Use exponential backoff with jitter, for example 0.5 s × 2^attempt plus a random 0 to 250 ms, so retries from parallel workers do not arrive together.

JSONL output

Write one JSON object per line, one line per input URL, success or not. JSON Lines is easy to append to, easy to stream and survives a crash: every finished line is already on disk.

{"url": "https://example.com/docs", "status": "success", "error": null, "source": "livecrawled", "fetched_at": "2026-10-27T06:00:12Z", "title": "Docs", "markdown": "# Docs\n..."}
{"url": "https://example.com/old-page", "status": "error", "error": "CRAWL_NOT_FOUND", "source": null, "fetched_at": "2026-10-27T06:00:13Z", "title": null, "markdown": null}

Keep the status and error fields even for failures. Without them a missing line and a failed page look the same, and that is how silent gaps start. At the end, print a summary: successes, failures by tag, credits used.

Resume and dedupe

Because every URL gets a line, resuming is simple: before the run, read the existing output file, collect the URLs that already have status: success (or a final error such as CRAWL_NOT_FOUND) and skip them. Only unfinished URLs and retryable errors go into the new run.

For recurring batches, keep statuses[].fetchedAt per URL and skip URLs fetched within their refresh window. Every request is a live fetch with no cache on the API side, so skipping in your own list is what makes repeat runs faster and lighter on the site. See pricing for how credits are counted.

Two more habits help:

• Dedupe the output too. If a run crashed after writing but before recording progress, the same URL can appear twice. Keep the last line per URL.
• Write to a new file per run (results-2026-10-27.jsonl) and keep the previous one. Comparing the two is the start of change detection.

What this means for your workflow

• Normalize, then dedupe, then filter out what you may not collect.
• Check the balance and the rate limit before you start, not halfway through.
• Retry 429, 5xx and timeouts up to 3 times; retry nothing else in the same run; never retry restricted targets; stop cleanly on 402.
• One JSONL line per URL, with status and error, so a crash loses nothing and a rerun skips finished work.
• When the pages feed a model, cap the text and keep the model in its own step; see the AI scraper workflow.