Add a robots.txt check step to your n8n or cron job

Reading robots.txt once, before the first run, is a good start. It is not enough for a job that runs every morning for months: rules change, hosts get added to the list, and nobody rereads the file. The fix is a small step that runs before every fetch, checks each URL against its host's robots.txt and passes on only the allowed ones.

This post builds that step twice: as a Python script for cron, and as two nodes for n8n. It is a technical check, not a legal one. The responsible web scraping guide covers terms of service and personal data, which robots.txt does not settle.

The rules the step has to follow

RFC 9309 defines the parts that matter for a scheduled job:

Two more details from the RFC: the most specific (longest) matching rule wins, and a crawler should not use a cached robots.txt for more than 24 hours. Fetching it fresh on every daily run covers that.

Python's standard urllib.robotparser differs from the RFC in two ways that matter here. Its read() method treats 401 and 403 as "disallow everything", which is stricter than the RFC and a safe default for a job that runs unattended, so both steps below adopt it. And its matcher applies rules in file order and does not expand * inside paths, so both steps use a small matcher of their own that follows the longest-match rule instead.

A Python step for cron

The script reads a list of URLs, fetches robots.txt once per host, and writes the allowed URLs as a CSV with a url column, ready for the batch script from the docs. Skipped URLs go to standard error with the reason.

import csv, re, sys
from urllib.parse import urlsplit
import requests

HEADERS = {"User-Agent": "example-collector/1.0 (+https://example.com/contact)"}

def star_rules(text):
    """Allow/Disallow rules of the * group, as (allow, pattern) pairs."""
    rules, in_star, last_was_agent = [], False, False
    for raw in text.splitlines():
        m = re.match(r"^([A-Za-z-]+)\s*:\s*(.*)$", raw.split("#", 1)[0].strip())
        if not m:
            continue
        key, val = m.group(1).lower(), m.group(2).strip()
        if key == "user-agent":
            in_star = (in_star and last_was_agent) or val == "*"
            last_was_agent = True
        else:
            last_was_agent = False
            if in_star and key in ("allow", "disallow") and val:
                rules.append((key == "allow", val))
    return rules

def allowed(rules, path):
    best = None  # longest match wins; Allow wins a tie
    for allow, pat in rules:
        if re.match("^" + re.escape(pat).replace(r"\*", ".*").replace(r"\$", "$"), path):
            if best is None or len(pat) > len(best[1]) or (len(pat) == len(best[1]) and allow):
                best = (allow, pat)
    return best is None or best[0]

def load_rules(scheme, host):
    try:
        r = requests.get(f"{scheme}://{host}/robots.txt", headers=HEADERS, timeout=10)
    except requests.RequestException:
        return None, "robots.txt unreachable"
    if r.status_code >= 500 or r.status_code in (401, 403):
        return None, f"robots.txt HTTP {r.status_code}"
    return ([] if r.status_code >= 400 else star_rules(r.text)), None

cache, out = {}, csv.writer(sys.stdout)
out.writerow(["url"])
for line in open(sys.argv[1], encoding="utf-8"):
    url = line.strip()
    if not url:
        continue
    p = urlsplit(url)
    key = (p.scheme, p.netloc.lower())
    if key not in cache:
        cache[key] = load_rules(*key)
    rules, reason = cache[key]
    path = (p.path or "/") + (f"?{p.query}" if p.query else "")
    if rules is not None and allowed(rules, path):
        out.writerow([url])
    else:
        print(f"skip {url}: {reason or 'disallowed by robots.txt'}", file=sys.stderr)

Chain it in front of the fetch in the crontab, with the key loaded from a protected env file:

0 6 * * *  cd /opt/jobs && set -a && . ./.env && set +a && python3 robots_check.py urls.txt > allowed.csv 2>> skipped.log && python3 batch_from_csv.py allowed.csv --out results-$(date +\%F).jsonl >> collect.log 2>&1

skipped.log is your record of what the job left out and why. If one host shows up there every day, remove it from the list or ask the operator.

The same check in n8n

In n8n, put two nodes between the node that reads your list and the fetch.

1. HTTP Request, method GET, URL {{ new URL($json.url).origin }}/robots.txt. Under Options → Response, set Response Format to Text, turn on Include Response Headers and Status and Never Error, so a 404 does not stop the run.
2. Code, mode Run Once for Each Item, with the matcher below. Then an IF node on {{ $json.robotsAllowed }}.

// Rename "Read URLs" to the node that holds your list.
const url = new URL($('Read URLs').item.json.url);
const status = $json.statusCode;
const path = url.pathname + url.search;
if (status >= 500 || status === 401 || status === 403) return { json: { url: url.href, robotsAllowed: false } };
if (status >= 400) return { json: { url: url.href, robotsAllowed: true } };

const rules = [];
let inStar = false, lastWasAgent = false;
for (const raw of String($json.body || "").split(/\r?\n/)) {
  const m = raw.replace(/#.*/, "").trim().match(/^([A-Za-z-]+)\s*:\s*(.*)$/);
  if (!m) continue;
  const key = m[1].toLowerCase(), val = m[2].trim();
  if (key === "user-agent") {
    if (!lastWasAgent) inStar = false;
    if (val === "*") inStar = true;
    lastWasAgent = true;
  } else {
    lastWasAgent = false;
    if (inStar && (key === "allow" || key === "disallow") && val) rules.push({ allow: key === "allow", val });
  }
}
const toRe = (p) => new RegExp("^" + p.replace(/[.+?^{}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"));
let best = null;
for (const r of rules) {
  if (!toRe(r.val).test(path)) continue;
  if (!best || r.val.length > best.val.length || (r.val.length === best.val.length && r.allow)) best = r;
}
return { json: { url: url.href, robotsAllowed: !best || best.allow } };

Like the Python step, the matcher reads only the * group, supports * and a trailing $ in paths, and lets the longest match win, with Allow winning a tie, as RFC 9309 describes. This version fetches robots.txt once per URL. For long lists with many URLs per host, run the check once per host in a sub-workflow, or call the Python step instead.

What this means for your workflow

• Check robots.txt on every run, not once at launch.
• Fetch it once per host per run; never trust a copy older than 24 hours.
• 5xx, no connection, 401 or 403: skip the host this run. Other 4xx: no rules.
• Log every skipped URL with its reason, and review the log.
• A passing check is one item on the list. Terms of service and personal data still need a human read; this is not legal advice.