Retry policy for batch fetches: what to retry, what to skip

A batch job that retries everything is slow and impolite. A job that retries nothing loses pages to every slow server. The useful middle is a short policy by error type, applied the same way in every script, workflow and scheduler. This post writes that policy down once, as a table and as a function you can reuse.

The policy

This is the policy of the batch scripts in the Automate requests docs, and the one every guide on this site uses: retry 429, 5xx and CRAWL_LIVECRAWL_TIMEOUT up to 3 times with backoff, retry no other per-URL error in the same run, and stop the job on 402, 401 or 403.

The full table, with every error tag and the reason behind each row, is in the retry rules for URL lists. The short version has three groups:

• Retry now: the problem is about timing. You went too fast (429), the server had a moment (5xx, a dropped connection), or the target site was slow (CRAWL_LIVECRAWL_TIMEOUT).
• Do not retry in this run: the answer will not change in seconds. CRAWL_NOT_FOUND, CRAWL_NOT_RENDERED and UNSUPPORTED_CONTENT_TYPE are final for this URL. CRAWL_UNKNOWN_ERROR and SOURCE_NOT_AVAILABLE are left for the next run. RESTRICTED_TARGET and 451 are never retried at all.
• Stop the job: the problem is the account, not the page. 402 means the credits ran out; 401 and 403 mean the key is invalid, revoked or suspended. Every further request would fail the same way.

How long to wait

For 429, the server tells you: the Retry-After header holds the number of seconds to wait. RFC 6585, which defines status 429, says the response may include it, and this API sends it.

For 5xx and timeouts, use exponential backoff with jitter. The batch scripts wait 0.5 s × 2^attempt plus a random 0 to 250 ms:

The worst case is under 4.3 seconds of waiting per URL before it is marked as failed. The jitter matters when several workers run in parallel: without it, workers that failed together retry together and fail together again. The AWS Architecture Blog post in the sources shows this effect with simulations.

One function for every script

Put the decision in one place, so a cron script, a worker pool and a notebook all behave the same:

import random

RETRY_NOW = {"CRAWL_LIVECRAWL_TIMEOUT"}
STOP_JOB = {401, 402, 403}
MAX_RETRIES = 3

def next_step(http_status: int, tag: str | None, retries_done: int, retry_after: str | None = None):
    """Return ("retry", seconds), ("skip", None) or ("stop", None)."""
    if http_status in STOP_JOB:
        return "stop", None
    if retries_done >= MAX_RETRIES:
        return "skip", None
    if http_status == 429:
        return "retry", float(retry_after) if retry_after else 0.5 * 2 ** retries_done + random.uniform(0, 0.25)
    if http_status >= 500 or (http_status == 200 and tag in RETRY_NOW):
        return "retry", 0.5 * 2 ** retries_done + random.uniform(0, 0.25)
    return "skip", None  # 400, 451 and every other per-URL error tag

Call it after each attempt with the HTTP status, the per-URL error tag (from statuses[0].error.tag when the status is 200), the number of retries so far and the Retry-After header. Treat a dropped connection like a 5xx. Record the final outcome in your output line: success, the error tag, or skipped when the job stopped.

Make the next run do the rest

The policy only works if the output remembers what happened. Write one line per URL with its final status and error tag. At the start of the next run, skip URLs that succeeded or failed for good (CRAWL_NOT_FOUND, RESTRICTED_TARGET), and send the rest again: CRAWL_UNKNOWN_ERROR, SOURCE_NOT_AVAILABLE, timeouts that ran out of retries, and URLs skipped after a stop.

If the same host returns SOURCE_NOT_AVAILABLE run after run, stop collecting it. Repeated refusals are a message from the site; the responsible web scraping guide covers what to check.

What this means for your workflow

• Retry 429, 5xx and timeouts up to 3 times; retry nothing else in the same run.
• Wait Retry-After on 429; use backoff with jitter for the rest.
• Never retry restricted targets; stop the job on 402, 401 or 403.
• Keep the decision in one function and one line of output per URL.
• Let the next scheduled run handle what the current one could not.