Scheduled web scraping with cron and GitHub Actions

A one-off collection run is a script. A scheduled one is a small service, and services fail in quieter ways: the schedule drifts, the job exits with code 0 after collecting nothing, or a list grows until the run no longer fits its window. This guide covers scheduled web scraping for public pages, from choosing a scheduler to the alerts that catch silent failures.

Freshness theory, such as how old a copy may be for which kind of page, is a broader topic. Here the focus is operational: making a recurring job run, finish and tell the truth.

Pick a scheduler

Any scheduler works if the job itself is a plain process that runs once and exits. Choose by where the rest of your workflow lives.

Cron and GitHub Actions

A crontab line runs a command on a fixed schedule in the server's time zone. Write output to a dated file and keep the log:

# m h dom mon dow  command
0 6 * * *  cd /opt/jobs && set -a && . ./.env && set +a && python3 batch_from_csv.py urls.csv --out results-$(date +\%F).jsonl >> collect.log 2>&1

Note the escaped %: in a crontab, an unescaped % ends the command. The key lives in /opt/jobs/.env (a line LIVE_WEB_DATA_API_KEY=...), readable only by the job's user (chmod 600 .env), not in the crontab itself. set -a exports every variable the file defines, and set +a turns that off again.

In GitHub Actions, the schedule event uses cron syntax in UTC by default:

name: Collect pages
on:
  schedule: [{ cron: "17 6 * * *" }]
  workflow_dispatch:
jobs:
  collect:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: node batch_from_csv.mjs urls.csv --out results.jsonl
        env:
          LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
      - uses: actions/upload-artifact@v4
        with: { name: results, path: results.jsonl }

Two details from GitHub's docs matter here. Scheduled runs can be delayed during high load, which is worst at the start of every hour, so pick an odd minute like 17. And in a public repository, scheduled workflows are disabled after 60 days without repository activity, so a job can stop without failing. Keep workflow_dispatch so you can run it by hand.

Both setups use the batch scripts from Automate requests, which retry 429, 5xx and timeouts up to 3 times, retry no other per-URL error in the same run, stop cleanly on 402, 401 or 403, and write results.jsonl with a summary. The full table is in the retry rules for URL lists.

Repeat runs and refresh windows

A scheduled job asks for the same pages again and again. Most of them have not changed. Live Web Data API reads every page live and keeps no cache, so the saving has to come from your own list: store statuses[].fetchedAt for each URL and only send the URLs whose last fetch is older than their refresh window.

Every URL you skip is one fetch less, less load on the site and a shorter run. Log fetchedAt next to each result so you know exactly how old every page in your output is.

Change alerts on public pages

Many scheduled jobs exist to answer one question: what changed since last time, and who needs to know? On public pages such as your own docs, public changelogs or public price pages, a simple loop works well:

1. Keep two runs. The dated output files from the cron line above already give you today's and yesterday's Markdown per URL.
2. Normalize before you compare. Trim whitespace and drop lines that change on every load, such as footer dates or "last updated" stamps. Comparing Markdown rather than HTML also ignores changing scripts, ad slots and tracking parameters; a short fingerprint per URL (any stable hash) makes the "same or not" check cheap.
3. Diff only what changed. For a changed page, compute a line diff (difflib.unified_diff in Python, diff -u in a shell) and keep the added and removed lines.
4. Set a threshold for noisy pages. Some pages change a little on every run. Alert only when more than a few lines changed, or when a line matching a watched pattern (a version number, a heading, a price line) changed.
5. Notify where people look. Post the URL and the diff to a Slack incoming webhook or an email, or write a row to the sheet the team already reads. Optionally send the diff to your own model for a two-line summary.

import difflib, json, urllib.request

def notify(webhook_url: str, url: str, old: str, new: str, min_lines: int = 3) -> None:
    diff = [l for l in difflib.unified_diff(old.splitlines(), new.splitlines(), lineterm="", n=0)
            if l[:1] in "+-" and l[:3] not in ("+++", "---")]
    if len(diff) < min_lines:
        return  # below the noise threshold
    text = f"Changed: {url}\n" + "\n".join(diff[:20])
    req = urllib.request.Request(webhook_url, data=json.dumps({"text": text}).encode(),
                                 headers={"Content-Type": "application/json"})
    urllib.request.urlopen(req, timeout=10)

Before you monitor a page that is not your own, read its robots.txt and terms. Monitoring has to stay within the acceptable-use policy: public pages only, no logged-in or personalized prices, and no personal data. The responsible web collection guide explains what to check.

Monitoring and alerts

The scheduler tells you whether the job ran. It does not tell you whether the job worked. At the end of each run, write a summary line and check it:

• No output file or summary: the job did not run or crashed early. Alert.
• Successes below a threshold, for example under 90% of the usual count: something changed. Alert.
• One error tag spiking: many CRAWL_NOT_FOUND means the list is stale; many SOURCE_NOT_AVAILABLE from one host means slow down or check that site's terms; 402 means the balance ran out.
• Run time creeping up toward the schedule interval: split the list before runs overlap.

Send the alert where someone will read it: a chat webhook, an email, or a row in the same sheet the team already uses. A "heartbeat" check (alert when no summary has arrived for a day) catches the scheduled job that was quietly disabled.

What this means for your workflow

• Make the job a plain process that runs once and exits; the scheduler only starts it.
• Choose odd minutes on GitHub Actions and remember the 60-day disable on public repos.
• Give each page type a refresh window, skip URLs fetched within it, and log fetchedAt for each page.
• Diff normalized Markdown between runs, alert above a noise threshold, and send the diff where people read it.
• Alert on counts and on silence, not just on exit codes.

The job's input is a URL list; see scrape a list of URLs for preparing it and for the retry rules.