# AgentScraper: guides and posts
> Build an agent scraper pipeline for public pages: a URL list in n8n, Make, Zapier or cron, clean Markdown out, every failure visible, your AI step after.
# n8n web scraping: a reliable URL-list workflow
Source: https://agentscraper.cc/guides/n8n-web-scraping
> n8n web scraping for public pages: HTTP Request node, batching, execution limits, error routing on per-URL status, Google Sheets output and an AI step.
With a list of public URLs, n8n workflows rarely fail on hard sites. They fail at item 40, when a run hits a time limit; at a node that reports success with an empty body; and in a Google Sheet that slowly fills with blanks. This guide builds a URL-list workflow that avoids those, with node names exactly as n8n shows them. It is the fetch half of an [agent scraper](/) pipeline: the model step comes after.
## The two-node baseline
Every n8n web scraping workflow is the same two moves: get an item per URL, then make one HTTP call per item.
1. A trigger. Use **Manual Trigger** while you build and **Schedule Trigger** once it works.
2. An **HTTP Request** node. For a raw page you would call `GET https://example.com/page` and parse the HTML with the **HTML** node and CSS selectors. That breaks every time the page layout changes, and you still have to strip menus, footers and cookie banners yourself.
The alternative is to call a fetch API that returns the main content as Markdown. With Live Web Data API the HTTP Request node is set up like this (the full table is in the [no-code recipes](/go/docs-no-code?from=guide:n8n-web-scraping)):
| Field | Value |
| --- | --- |
| Method | `POST` |
| URL | the `/v1/contents` endpoint from the docs |
| Authentication | Generic Credential Type → **Bearer Auth** |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` |
| Options → Response → Response Format | `JSON` |
```json
{ "urls": ["{{ $json.url }}"] }
```
The response holds the page in `results[0].markdown` and the outcome in `statuses[0].status` and `statuses[0].error.tag`. Keep the key in an n8n credential, not in a plain header field, so it does not end up in exported workflow JSON.
## URL lists and batching
Once the trigger produces many items, the HTTP Request node runs once per item, as fast as it can. Any API with a rate limit will start answering `429`. You have two ways to pace it.
**Options → Batching** on the HTTP Request node. Set **Items per Batch** to your plan's rate limit, which counts URLs per second (one URL per item), and **Batch Interval (ms)** to `1000`. This is the simplest fix when only one node needs pacing.
**Loop Over Items** (the node formerly called Split in Batches) plus a **Wait** node. Use it when several nodes must run per batch, for example fetch, AI step and Sheets update. Put a Wait of one second at the end of each pass and connect it back to the loop.
Batching is about the provider's limit, but also about the target sites. A polite rate is part of [responsible web collection](/guides/responsible-web-collection), not just a way to avoid errors.
## Execution limits and sub-workflows
Long runs fail in n8n for three common reasons:
- **Execution timeout.** Self-hosted instances read `EXECUTIONS_TIMEOUT` (seconds, default `-1`, meaning no limit) and `EXECUTIONS_TIMEOUT_MAX`. n8n Cloud plans set their own limits; check your plan.
- **Memory.** Every item carries the full page through every later node. A few hundred pages of Markdown in one execution adds up fast.
- **One failed node ends the run.** By default a node error stops the workflow, so item 41 never runs.
The fix is the same for all three: make each execution small. A parent workflow reads the list and sends chunks of, say, 20 to 50 URLs to a child workflow with the **Execute Workflow** node (named **Execute Sub-workflow** in newer versions). Each child execution fetches, processes and writes its own rows, then ends. The parent only holds URLs, never page content.
Also cap the text that comes back. The API's `text.maxCharacters` defaults to 100,000 characters, enough for whole pages; set it lower, for example `8000`, so very long pages do not bloat every item.
## Error routing on per-URL status
This is the part most guides skip. A fetch API that handles a list reports per-URL failures inside a normal `200` response. If your workflow only checks the HTTP code, error pages flow into your Sheet as if they were content.
Add an **IF** node after the fetch with the condition `{{ $json.statuses[0].status }}` equals `success`. Then decide per error tag. The **Retry?** column is the same policy as the [retry rules table](/guides/scrape-list-of-urls#retry-rules-table) and the batch scripts in the docs:
| Outcome | Retry? | What the workflow should do |
| --- | --- | --- |
| `429 RATE_LIMITED` | Yes, after `Retry-After`, up to 3 times | Slow the batch down |
| `5xx` or network error | Yes, up to 3 times with backoff | Temporary problem |
| `CRAWL_LIVECRAWL_TIMEOUT` | Yes, up to 3 times with backoff | The site was slow |
| `CRAWL_UNKNOWN_ERROR` | Not in the same run | Mark the row; the next run picks it up |
| `SOURCE_NOT_AVAILABLE` | Not in the same run | Mark the row; if it repeats, check the site's terms and stop collecting that host |
| `CRAWL_NOT_FOUND` | No | Mark the row; remove the URL from the list |
| `CRAWL_NOT_RENDERED` | No | Mark the row; resend later with `"renderFallback": true` or use another source |
| `UNSUPPORTED_CONTENT_TYPE` | No | Route to a file parser |
| `RESTRICTED_TARGET` / `451` | Never | Mark the row; repeated attempts can suspend a key |
| `402 INSUFFICIENT_CREDITS` | No: stop the job | Top up, then resume |
| `401` / `403` | No: stop the job | Fix or replace the key |
Whole-request errors (`429`, `5xx`, `402`, `401`, `403`) arrive with a real HTTP code. For those, open the node's **Settings**, set **On Error** to **Continue (using error output)** and send that branch to a dead-letter sheet. **Retry On Fail** in the same settings retries every error the node raises, `402` included, and does not read `Retry-After`. To match the policy exactly, route `429` and `5xx` from the error output through a **Wait** node and back to the fetch, counting attempts in a field and giving up after 3 retries. Failed pages cost 0 credits, so retries are cheap, but retrying a restricted target can get a key suspended.
## Writing to Google Sheets
Write one row per URL, not one row per success. Use the **Google Sheets** node with **Append or Update Row** and the URL as the matching column. Useful columns:
- `url`, `status` (`success` or the error tag), `fetched_at`, `title`
- `summary` or `label` from the AI step
- `attempts`, so a retry branch knows when to give up
A status column that tells the truth is the cheapest monitoring there is. Filter the sheet for anything that is not `success` and you see exactly what the last run missed. On the next run, read only rows whose status is empty, an error worth retrying, or older than your refresh window. That makes the whole workflow resumable.
## Adding an AI step
Put the model after the IF node, on the success branch only. Send `results[0].markdown`, not the raw HTML: it is shorter and the model does not have to guess which part is navigation. Ask for a fixed output shape (a two-sentence summary, a label from a short list, or a JSON object with named fields) and parse it before you write it to the sheet.
The [AI scraper workflow guide](/guides/ai-scraper-workflow) covers prompts, input caps and how to keep fetch costs and model costs apart.
## What this means for your workflow
- One item per URL, paced with **Batching** or **Loop Over Items** plus **Wait**.
- Small executions: a parent with the list, a child per chunk of 20 to 50 URLs.
- Retry `429`, `5xx` and timeouts up to 3 times; retry nothing else in the same run; stop on `402`.
- Route on `statuses[0].status` and the error tag, never on the HTTP code alone.
- One sheet row per URL with a status column; read only unfinished rows on the next run.
- The model goes after the fetch, on clean Markdown, on the success branch.
For a schedule that runs this every morning, continue with [scheduled web scraping](/guides/scheduled-web-scraping). For the same pattern in Make or Zapier, see [Zapier and Make web scraping](/guides/make-and-zapier-web-scraping).
---
# Zapier and Make web scraping: HTTP steps and iterators
Source: https://agentscraper.cc/guides/make-and-zapier-web-scraping
> Zapier and Make.com web scraping for public pages: the generic HTTP step, looping over a URL list, rate limits, failure routing and a comparison with n8n.
Make and Zapier are not scraping tools. They are glue: a trigger, a few steps, an output. That is exactly why they work well for collecting a known list of public pages, as long as the page fetch itself is one clean HTTP call. This guide covers that call in both tools, how each one loops over a list, where each one fails quietly, and how they compare with n8n.
## Make: HTTP > Make a request
Make's **HTTP** app has a generic **Make a request** module. For a fetch API that returns page content as JSON, the settings are:
| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stored in a keychain), or `No authentication` plus a header |
| URL | the `/v1/contents` endpoint from the docs |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["{{1.url}}"]}` |
| Parse response | `Yes` |
With **Parse response** on, later modules can map `statuses[].status`, `statuses[].error.tag` and `results[].markdown` directly. The exact copy-paste values are in the [no-code recipes](/go/docs-no-code?from=guide:make-and-zapier-web-scraping).
## Zapier: Webhooks by Zapier > Custom Request
Zapier's **Webhooks by Zapier** app offers GET, POST and PUT events plus **Custom Request**. It is not available on the Free plan, and Zapier caps action payloads at 5 MB.
| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | the `/v1/contents` endpoint from the docs |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/changelog"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |
The JSON response becomes fields you can use in the next step. Keep the page short with `"text": {"maxCharacters": 8000}` if the next step is a model or a Sheets cell, which has its own size limit.
## Lists and rate limits
Both tools run one module per item, and both will happily send requests faster than an API allows.
**In Make**, the URL list usually arrives as an array: rows from **Google Sheets > Search Rows**, records from Airtable, or a text field split into lines. Put an **Iterator** in front of the HTTP module. Make's docs describe it as a module that "converts an array into a series of bundles", so each URL becomes its own bundle and its own request. To stay under a rate limit, add the **Tools > Sleep** module after the HTTP module, or split a long list across several scheduled runs.
**In Zapier**, a Zap normally handles one trigger event at a time. For a list in a single run, use **Looping by Zapier** on line items or text, then the Custom Request. Pacing is coarser: **Delay by Zapier** or, more simply, one Zap run per row (a "New or Updated Spreadsheet Row" trigger) so Zapier's own queue spreads the calls out.
Either way, check your rate limit with `GET /v1/account` before a large run. Sending 10 URLs in one request is allowed and counts as 10 against the rate limit, which counts URLs, not requests, so batching URLs does not make a job faster than your plan allows.
## Routing failures
Failed pages come back inside a `200` response with `statuses[0].status` set to `error`. Both tools treat that as success unless you look.
- **Make:** add a **Filter** after the HTTP module (`statuses[].status` equal to `success`) on the main path, and a **Router** branch for errors that writes the URL and `error.tag` to an error sheet. For request-level failures (`429`, `402`), attach an error handler to the HTTP module: **Break** to retry later, or **Ignore** to skip and log.
- **Zapier:** add a **Filter** step that only continues on `success`, or use **Paths** to send errors to a separate row. Zapier's Zap history marks the run as successful either way, so the error row is your only record.
What to retry follows the same policy as every guide on this site: retry `429`, `5xx` and `CRAWL_LIVECRAWL_TIMEOUT` up to 3 times with backoff, retry no other per-URL error in the same run, and stop the scenario or Zap on `402`, `401` or `403`. Never retry `RESTRICTED_TARGET`. The full table is in [the retry rules for URL lists](/guides/scrape-list-of-urls#retry-rules-table). Failed pages cost 0 credits, but every retry still costs a Make operation or a Zapier task.
## Choosing a platform
| Question | Make | Zapier | n8n |
| --- | --- | --- | --- |
| Long lists (hundreds of URLs) | Workable with Iterator and scheduling | Awkward; one run per row is easiest | Best fit, with sub-workflows |
| Branching on status | Router and filters | Paths and filters | IF node and error output |
| What you pay per page | Operations per module run | Tasks per action step | Executions (Cloud) or your server |
| Self-hosting | No | No | Yes |
| Easiest for a non-developer | Visual, data-heavy | Simplest for linear flows | Most settings to learn |
A practical rule: if the job is "for each new row, fetch one page and summarize it", Zapier is fine. If it is "every morning, fetch 300 pages and route failures", use Make or [n8n](/guides/n8n-web-scraping). If it is thousands of pages, a plain script is cheaper and easier to debug; see [scrape a list of URLs](/guides/scrape-list-of-urls).
## What this means for your workflow
- Use the generic HTTP step in both tools; native apps are announced as coming soon.
- Turn lists into one item per URL: **Iterator** in Make, one run per row or **Looping by Zapier** in Zapier.
- Pace calls below your rate limit and count operations or tasks per page before you schedule.
- Filter on `statuses[0].status`; log every error tag to a separate sheet.
- Move to n8n or a script when the list grows past what one run handles comfortably.
---
# Scrape a list of URLs: CSV in, JSONL out, safe retries
Source: https://agentscraper.cc/guides/scrape-list-of-urls
> Scrape a list of URLs as a batch job: normalize and dedupe the list, size the run, pace requests, retry by error type, write JSONL and resume after a crash.
Scraping a list of URLs sounds like a five-line loop. It is, until the list has 3,000 entries, the run dies at row 1,800 and nobody knows which rows made it. This guide turns the loop into a batch job you can trust: a clean list in, one JSON line per URL out, retries by error type, and a run that can resume.
The examples use Python, but the same rules apply to Node, a shell script or a data pipeline tool.
## Prepare the list
Most problems in a batch start in the input file.
**Normalize every URL first.** Lowercase the scheme and host, remove the fragment (`#section`), drop tracking parameters such as `utm_*`, and pick one rule for trailing slashes. `https://Example.com/docs/` and `https://example.com/docs?utm_source=x` are the same page.
**Dedupe after normalizing.** Otherwise the same page gets fetched and billed twice and appears twice in your output.
**Remove what you should not collect.** Pages behind a login or paywall are out of scope and prohibited by the acceptable-use policy. Lists of profile pages about individual people raise personal data questions. The [responsible web collection guide](/guides/responsible-web-collection) has the checklist.
**Keep a stable ID.** Use the normalized URL itself, or a hash of it, as the key that ties input rows to output lines.
```python
from urllib.parse import urlsplit, urlunsplit, parse_qsl, urlencode
def normalize(url: str) -> str:
s = urlsplit(url.strip())
query = urlencode([(k, v) for k, v in parse_qsl(s.query) if not k.startswith("utm_")])
path = s.path.rstrip("/") or "/"
return urlunsplit((s.scheme.lower(), s.netloc.lower(), path, query, ""))
```
## Size the run
Before you start, answer three questions:
1. **How many URLs?** After dedupe, not before.
2. **How many credits are left?** Call `GET /v1/account` and compare `credits.total` with the URL count. Each successful page costs credits depending on how it had to be fetched (`statuses[].credits` shows the number per URL), and failed pages cost 0, so do a small test batch first and multiply.
3. **How long will it take?** URLs divided by your plan's rate limit, which counts URLs per second (`rateLimit.requestsPerSecond` in the same response), plus retries. A 3,000-URL list at 2 URLs per second needs at least 25 minutes.
If the balance is too low, split the list or top up first. A job that stops on `402 INSUFFICIENT_CREDITS` in the middle is recoverable, but it is better not to start one you know cannot finish.
## Concurrency and rate limits
Run with a concurrency equal to your plan's rate limit in URLs per second. You can send up to 10 URLs in one request, but the rate limit counts URLs, so ten URLs in one call use the same budget as ten single calls. Sending one URL per request is simplest and makes retries per URL easy.
On `429 RATE_LIMITED`, read the `Retry-After` header, wait that many seconds and retry the same request. If you get `429`s regularly, your concurrency is too high. Lower it rather than leaning on retries.
## Retry rules table
Retry by error type, not blindly. This is the one policy every guide on this site uses, and the ready-made batch scripts in [Automate requests](/go/docs-automation?from=guide:scrape-list-of-urls) implement it: retry `429`, `5xx` and `CRAWL_LIVECRAWL_TIMEOUT` up to 3 times with backoff, retry no other per-URL error in the same run, and stop the job on `402`, `401` or `403`.
| Outcome | Retry? | Why |
| --- | --- | --- |
| `429 RATE_LIMITED` | Yes, after `Retry-After`, up to 3 times | You went too fast |
| `5xx` or network error | Yes, up to 3 times with backoff | Temporary problem |
| `CRAWL_LIVECRAWL_TIMEOUT` | Yes, up to 3 times with backoff | The site was slow |
| `CRAWL_UNKNOWN_ERROR` | Not in the same run | Something else failed; the next run picks it up |
| `SOURCE_NOT_AVAILABLE` | Not in the same run | The site refused access; if it repeats, check its terms and stop collecting that host |
| `CRAWL_NOT_FOUND` | No | The page does not exist |
| `CRAWL_NOT_RENDERED` | No | Needs JavaScript; retry in a later run with `renderFallback: true` |
| `UNSUPPORTED_CONTENT_TYPE` | No | Not HTML or text |
| `RESTRICTED_TARGET` / `451` | Never | Excluded by policy; repeated attempts can suspend a key |
| `400 INVALID_REQUEST` | No | Fix the request first |
| `402 INSUFFICIENT_CREDITS` | No: stop the job | Top up, then resume |
| `401` / `403` | No: stop the job | Key invalid, revoked or suspended |
Use exponential backoff with jitter, for example `0.5 s × 2^attempt` plus a random 0 to 250 ms, so retries from parallel workers do not arrive together.
## JSONL output
Write one JSON object per line, one line per input URL, success or not. [JSON Lines](https://jsonlines.org/) is easy to append to, easy to stream and survives a crash: every finished line is already on disk.
```json
{"url": "https://example.com/docs", "status": "success", "error": null, "source": "livecrawled", "fetched_at": "2026-10-27T06:00:12Z", "title": "Docs", "markdown": "# Docs\n..."}
{"url": "https://example.com/old-page", "status": "error", "error": "CRAWL_NOT_FOUND", "source": null, "fetched_at": "2026-10-27T06:00:13Z", "title": null, "markdown": null}
```
Keep the `status` and `error` fields even for failures. Without them a missing line and a failed page look the same, and that is how silent gaps start. At the end, print a summary: successes, failures by tag, credits used.
## Resume and dedupe
Because every URL gets a line, resuming is simple: before the run, read the existing output file, collect the URLs that already have `status: success` (or a final error such as `CRAWL_NOT_FOUND`) and skip them. Only unfinished URLs and retryable errors go into the new run.
For recurring batches, keep `statuses[].fetchedAt` per URL and skip URLs fetched within their refresh window. Every request is a live fetch with no cache on the API side, so skipping in your own list is what makes repeat runs faster and lighter on the site. See [pricing](/go/pricing?from=guide:scrape-list-of-urls) for how credits are counted.
Two more habits help:
- **Dedupe the output too.** If a run crashed after writing but before recording progress, the same URL can appear twice. Keep the last line per URL.
- **Write to a new file per run** (`results-2026-10-27.jsonl`) and keep the previous one. Comparing the two is the start of [change detection](/guides/scheduled-web-scraping).
## What this means for your workflow
- Normalize, then dedupe, then filter out what you may not collect.
- Check the balance and the rate limit before you start, not halfway through.
- Retry `429`, `5xx` and timeouts up to 3 times; retry nothing else in the same run; never retry restricted targets; stop cleanly on `402`.
- One JSONL line per URL, with status and error, so a crash loses nothing and a rerun skips finished work.
- When the pages feed a model, cap the text and keep the model in its own step; see the [AI scraper workflow](/guides/ai-scraper-workflow).
---
# Scheduled web scraping with cron and GitHub Actions
Source: https://agentscraper.cc/guides/scheduled-web-scraping
> Scheduled web scraping that stays honest: choosing a scheduler, cron and GitHub Actions setups, refresh windows, change alerts and checks for silent failures.
A one-off collection run is a script. A scheduled one is a small service, and services fail in quieter ways: the schedule drifts, the job exits with code 0 after collecting nothing, or a list grows until the run no longer fits its window. This guide covers scheduled web scraping for public pages, from choosing a scheduler to the alerts that catch silent failures.
Freshness theory, such as how old a copy may be for which kind of page, is a broader topic. Here the focus is operational: making a recurring job run, finish and tell the truth.
## Pick a scheduler
Any scheduler works if the job itself is a plain process that runs once and exits. Choose by where the rest of your workflow lives.
| Scheduler | Good for | Watch out for |
| --- | --- | --- |
| cron on a server | Simple daily jobs you control | No retries, no history, mail on output only |
| GitHub Actions `schedule` | Jobs that live with their code | Delays under load, 5-minute minimum, 60-day auto-disable on public repos |
| n8n **Schedule Trigger** | Workflows already in n8n | Execution limits on long runs |
| Make / Zapier schedules | Small lists inside a scenario or Zap | Operations or tasks per run |
| Cloud schedulers, Kubernetes CronJob, Airflow | Teams with existing infrastructure | More setup than the job needs at first |
## Cron and GitHub Actions
A crontab line runs a command on a fixed schedule in the server's time zone. Write output to a dated file and keep the log:
```bash
# m h dom mon dow command
0 6 * * * cd /opt/jobs && set -a && . ./.env && set +a && python3 batch_from_csv.py urls.csv --out results-$(date +\%F).jsonl >> collect.log 2>&1
```
Note the escaped `%`: in a crontab, an unescaped `%` ends the command. The key lives in `/opt/jobs/.env` (a line `LIVE_WEB_DATA_API_KEY=...`), readable only by the job's user (`chmod 600 .env`), not in the crontab itself. `set -a` exports every variable the file defines, and `set +a` turns that off again.
In GitHub Actions, the `schedule` event uses cron syntax in UTC by default:
```yaml
name: Collect pages
on:
schedule: [{ cron: "17 6 * * *" }]
workflow_dispatch:
jobs:
collect:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: node batch_from_csv.mjs urls.csv --out results.jsonl
env:
LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
- uses: actions/upload-artifact@v4
with: { name: results, path: results.jsonl }
```
Two details from GitHub's docs matter here. Scheduled runs can be delayed during high load, which is worst at the start of every hour, so pick an odd minute like `17`. And in a public repository, scheduled workflows are disabled after 60 days without repository activity, so a job can stop without failing. Keep `workflow_dispatch` so you can run it by hand.
Both setups use the batch scripts from [Automate requests](/go/docs-automation?from=guide:scheduled-web-scraping), which retry `429`, `5xx` and timeouts up to 3 times, retry no other per-URL error in the same run, stop cleanly on `402`, `401` or `403`, and write `results.jsonl` with a summary. The full table is in the [retry rules for URL lists](/guides/scrape-list-of-urls#retry-rules-table).
## Repeat runs and refresh windows
A scheduled job asks for the same pages again and again. Most of them have not changed. Live Web Data API reads every page live and keeps no cache, so the saving has to come from your own list: store `statuses[].fetchedAt` for each URL and only send the URLs whose last fetch is older than their refresh window.
| Page type | Suggested refresh window | Effect |
| --- | --- | --- |
| Status or price pages | 1 hour | At most one live fetch per page per hour |
| Changelogs and docs | 24 hours | A copy from the last 24 hours is reused from your store |
| Slow-moving pages | 168 hours | Refresh once a week |
| Must be live | every run | Always fetch live; use sparingly |
Every URL you skip is one fetch less, less load on the site and a shorter run. Log `fetchedAt` next to each result so you know exactly how old every page in your output is.
## Change alerts on public pages
Many scheduled jobs exist to answer one question: what changed since last time, and who needs to know? On public pages such as your own docs, public changelogs or public price pages, a simple loop works well:
1. **Keep two runs.** The dated output files from the cron line above already give you today's and yesterday's Markdown per URL.
2. **Normalize before you compare.** Trim whitespace and drop lines that change on every load, such as footer dates or "last updated" stamps. Comparing Markdown rather than HTML also ignores changing scripts, ad slots and tracking parameters; a short fingerprint per URL (any stable hash) makes the "same or not" check cheap.
3. **Diff only what changed.** For a changed page, compute a line diff (`difflib.unified_diff` in Python, `diff -u` in a shell) and keep the added and removed lines.
4. **Set a threshold for noisy pages.** Some pages change a little on every run. Alert only when more than a few lines changed, or when a line matching a watched pattern (a version number, a heading, a price line) changed.
5. **Notify where people look.** Post the URL and the diff to a Slack incoming webhook or an email, or write a row to the sheet the team already reads. Optionally send the diff to your own model for a two-line summary.
```python
import difflib, json, urllib.request
def notify(webhook_url: str, url: str, old: str, new: str, min_lines: int = 3) -> None:
diff = [l for l in difflib.unified_diff(old.splitlines(), new.splitlines(), lineterm="", n=0)
if l[:1] in "+-" and l[:3] not in ("+++", "---")]
if len(diff) < min_lines:
return # below the noise threshold
text = f"Changed: {url}\n" + "\n".join(diff[:20])
req = urllib.request.Request(webhook_url, data=json.dumps({"text": text}).encode(),
headers={"Content-Type": "application/json"})
urllib.request.urlopen(req, timeout=10)
```
Before you monitor a page that is not your own, read its robots.txt and terms. Monitoring has to stay within the acceptable-use policy: public pages only, no logged-in or personalized prices, and no personal data. The [responsible web collection guide](/guides/responsible-web-collection#public-price-pages-and-competitor-monitoring) explains what to check.
## Monitoring and alerts
The scheduler tells you whether the job ran. It does not tell you whether the job worked. At the end of each run, write a summary line and check it:
- **No output file or summary**: the job did not run or crashed early. Alert.
- **Successes below a threshold**, for example under 90% of the usual count: something changed. Alert.
- **One error tag spiking**: many `CRAWL_NOT_FOUND` means the list is stale; many `SOURCE_NOT_AVAILABLE` from one host means slow down or check that site's terms; `402` means the balance ran out.
- **Run time creeping up** toward the schedule interval: split the list before runs overlap.
Send the alert where someone will read it: a chat webhook, an email, or a row in the same sheet the team already uses. A "heartbeat" check (alert when no summary has arrived for a day) catches the scheduled job that was quietly disabled.
## What this means for your workflow
- Make the job a plain process that runs once and exits; the scheduler only starts it.
- Choose odd minutes on GitHub Actions and remember the 60-day disable on public repos.
- Give each page type a refresh window, skip URLs fetched within it, and log `fetchedAt` for each page.
- Diff normalized Markdown between runs, alert above a noise threshold, and send the diff where people read it.
- Alert on counts and on silence, not just on exit codes.
The job's input is a URL list; see [scrape a list of URLs](/guides/scrape-list-of-urls) for preparing it and for the retry rules.
---
# The AI scraper workflow: fetch first, then ask your model
Source: https://agentscraper.cc/guides/ai-scraper-workflow
> An AI scraper workflow in two steps: collect clean Markdown first, then extract, classify or summarize with your own LLM, cap the input, keep the costs apart.
Search for "AI scraper" and most results sell one box: give it a URL and a prompt, get perfect data back. Inside that box are two different jobs. **Collection** gets the readable part of a public page, reliably, for a list of URLs. **Interpretation** asks a model what the page means. This guide pulls them apart, because a workflow that keeps them separate is cheaper to run, easier to debug and lets you change the model without rebuilding anything. It is the model half of an [agent scraper](/) pipeline; the fetch half comes first.
## Two jobs, not one
| | Collection | Interpretation |
| --- | --- | --- |
| Question | "What does the page say?" | "What does it mean for me?" |
| Fails when | The site is slow, gone, or needs JavaScript | The model misreads, invents or truncates |
| Fix | Retry, skip, use another source | Better prompt, schema, validation |
| Cost | Per page fetched | Per token sent and received |
When both happen in one step, a wrong value could come from either side, and the bill is one number. When they are separate, each step has its own status, its own log and its own cost line.
In practice the fetch step is an HTTP call that returns Markdown and a status, as in the [n8n](/guides/n8n-web-scraping) and [batch](/guides/scrape-list-of-urls) guides. Live Web Data API does exactly that and nothing more: it reads each URL (in a browser if you set `render`), but it does not crawl for new URLs or extract fields. The model step is yours.
## Extraction from Markdown
Markdown is a good input for a model: headings, lists, tables and links survive, while navigation, scripts and cookie banners are gone. To get fields out of it reliably:
1. **Define a schema.** List the fields, their types and which ones may be null. Most model APIs support structured outputs that force the response into a JSON schema; use that instead of asking politely for JSON.
2. **Allow "not found".** Tell the model to return `null` when a field is not on the page. Without that option, models fill gaps with plausible guesses.
3. **Ask for evidence.** For important fields, add a short `quote` field with the sentence the value came from. It makes spot checks fast.
4. **Validate in code.** Check types, ranges and formats after the model answers. Reject and log, do not repair silently.
```json
{
"type": "object",
"properties": {
"product_name": { "type": ["string", "null"] },
"release_date": { "type": ["string", "null"], "description": "ISO date if stated on the page" },
"breaking_changes": { "type": "array", "items": { "type": "string" } },
"quote": { "type": ["string", "null"] }
},
"required": ["product_name", "release_date", "breaking_changes", "quote"],
"additionalProperties": false
}
```
This example reads a public changelog page. The same pattern works for public documentation, job descriptions on a company's own careers page or public event listings, always within the site's terms and without personal data you have no basis to process.
### A worked example
The fetch step returns this Markdown for `https://example.com/changelog` (capped with `"text": {"maxCharacters": 8000}`):
```text
# Changelog
## 2.4.0 (2026-10-14)
- Added: webhook retries are now configurable.
- Changed: the `limit` parameter now defaults to 50.
- Removed: the deprecated `/v1/export` endpoint.
```
The prompt sent to your model, with the schema above passed as the structured-output format:
```text
Extract the latest release from this changelog.
Return null for any field the page does not state.
breaking_changes: only changes that can break existing integrations, copied as written.
quote: the heading line of the release you used.
{markdown}
```
A valid answer, which your code then checks (the date parses as ISO, every breaking change appears in the page text):
```json
{
"product_name": null,
"release_date": "2026-10-14",
"breaking_changes": [
"Changed: the `limit` parameter now defaults to 50.",
"Removed: the deprecated `/v1/export` endpoint."
],
"quote": "## 2.4.0 (2026-10-14)"
}
```
`product_name` is `null` because the page never names the product. That is the behavior you want: a guess here would look correct and be wrong.
## Classification and filtering
Often the cheapest useful AI step is not extraction at all but a yes/no or a label:
- "Is this page about topic X? Answer `relevant` or `not_relevant`."
- "Which of these five categories fits best?"
- "Did anything in this changelog affect authentication?"
Classification needs far less text than extraction. Send the title and the first 2,000 to 4,000 characters, not the whole page. Run it before more expensive steps, so only relevant pages reach the extraction prompt or your storage.
## Cap page length before the model
Every character you send is paid for, and research on long contexts ([Liu et al., 2023](https://arxiv.org/abs/2307.03172)) found that models use information buried in the middle of a long input less reliably. Cap length in two places:
- **At the fetch:** `"text": {"maxCharacters": 8000}` truncates the Markdown before it leaves the API. The default is 100,000, which keeps whole pages, so set a lower cap yourself.
- **In your code:** for classification, cut to the first sections; for extraction, cut to the sections whose headings match what you need.
Log the original length next to the capped length, so you notice when a page grew and the field you need moved past the cut.
## Fixed lists vs agent navigation
An agentic scraper lets a model decide which page to open next, click, and read again. That is useful when you do not know the pages in advance: research across unfamiliar sites, or following links to find the one page that answers a question.
It is a poor fit for a fixed list collected on a schedule. Each run repeats the same navigation decisions, pays model tokens for every step, takes longer and fails in ways that are buried in a trace. A plain loop over known URLs, followed by one model call per page, does the same work with a predictable cost. If your use case really is an agent fetching pages inside its own reasoning loop, the design questions are different (tool design, in-loop errors, prompt injection); agentfetch.live covers that side, starting with [when an agent needs a browser and when a fetch is enough](https://agentfetch.live/guides/browser-agent-vs-fetch).
## Costs: fetch vs model
Keep two cost lines and look at them separately:
- **Fetch:** credits per successful page. Failed, blocked or restricted pages cost 0 credits. Every fetch is live, so skip URLs your own store fetched within their refresh window; see [pricing](/go/pricing?from=guide:ai-scraper-workflow) for how credits are counted.
- **Model:** input tokens (driven by your length cap) plus output tokens (driven by your schema). Depending on page length and your model, this can be the larger line, which is why the cap matters.
Store both per page in your output (credits used, input tokens, output tokens). When the bill moves, you can tell which half moved.
## What this means for your workflow
- Fetch first, as its own step with its own status. Interpret second, with your own model.
- Use structured outputs with a JSON schema, allow `null`, and validate in code.
- Classify before you extract; send the model only what the task needs.
- Use an agent when the pages are unknown, a loop when the list is fixed.
- Track fetch credits and model tokens separately, per page.
Before you point the workflow at someone else's site, read the [responsible web collection guide](/guides/responsible-web-collection).
---
# Responsible web scraping: robots.txt, terms, personal data
Source: https://agentscraper.cc/guides/responsible-web-collection
> Responsible web scraping before a job is scheduled: robots.txt, terms of service, personal data, polite request rates and the usage policy. Not legal advice.
**This guide is not legal advice.** It explains the questions an automation builder should answer before scheduling a job that collects web pages. For your own project, ask a lawyer in your jurisdiction.
Many "is web scraping legal" articles come from scraping vendors, who have a reason to answer "yes, mostly". The honest answer is "it depends", and the useful part is knowing what it depends on. Responsible web scraping is less about one rule and more about a short list of checks, done before the job runs every morning.
## Check robots.txt for every host
The first item on the pre-launch checklist: fetch `https:///robots.txt` for every host on your list and compare its rules with the paths you plan to collect. The format is standardized in RFC 9309, the Robots Exclusion Protocol. A [robots.txt check step](/blog/robots-txt-check-step) shows how to run this check inside an n8n or cron job before every run.
Why the check matters:
- It states the site operator's crawling preferences, per user agent and path.
- It gives you a cheap, machine-readable first signal of whether automated access is welcome.
What robots.txt does not do:
- It is not an access control mechanism. RFC 9309 says so directly, and a December 2025 ruling in Ziff Davis v. OpenAI (S.D.N.Y.) reportedly held that robots.txt does not "effectively control" access for the purposes of the DMCA anti-circumvention rule, according to the law blog listed in the sources.
- It is not a permission slip. A path that robots.txt allows can still be covered by terms of service, copyright or data protection law.
- Ignoring it is not free of consequences. It can still be raised against you in a dispute.
Practical rule: read robots.txt for every host on your list, honor `Disallow` lines that match your paths, and treat a blanket `Disallow: /` as a strong signal to find another source or ask the operator.
## Terms of service
A site's terms may restrict automated access, limit reuse of content, or forbid commercial use. Whether those terms bind you depends on how they were presented and accepted, and on your jurisdiction. Breach of terms is generally a contract question rather than a criminal one, but it can still end in a dispute you do not want.
What to do in practice:
- Read the terms of every site you schedule, and note the date you checked.
- Look specifically for clauses about automated access, crawling, data mining and reuse.
- Prefer official channels when they exist: a public API, an RSS feed, a data export or a partnership page.
- If the terms forbid what you plan, do not schedule it.
## Personal data
Pages that mention identifiable people are where most real risk sits. In the EU and UK, the GDPR applies to personal data even when it is publicly visible: you need a lawful basis (Article 6), a clear purpose and data minimization. US state privacy laws, such as California's, add their own rules.
For collection workflows this means:
- **Do not build lists of people.** Profiles, staff pages, author pages and contact details are the wrong input for a scheduled job unless you have a clear legal basis.
- **Minimize at the source.** If a page mixes useful content with names or emails, strip them before storage, or do not store the raw text at all.
- **Set retention.** Keep raw pages only as long as the workflow needs them.
## Polite collection
Even on public pages that you are allowed to read, how you read them matters:
- Keep the request rate low per host. A daily job that needs 200 pages from one site does not need to fetch them in the same second.
- Keep the fetch time per URL and skip pages fetched within their refresh window, so repeat runs do not fetch unchanged pages again.
- Run only as often as the use case needs. Daily is enough for most public changelogs and docs.
- Stop retrying a host that refuses access. Repeated `SOURCE_NOT_AVAILABLE` errors are a message, not an obstacle.
- Have an owner and a stop switch. If an operator contacts you, you should be able to pause the job the same day.
A provider network can mean fewer blocks on public pages, but that is not a reason to treat a site as an opponent. If a site does not want to be collected, the right move is a different source.
## What the AUP prohibits
Live Web Data API's acceptable-use policy (Section 8 of the Terms) applies to every request, from any of these workflows. Among other things, it prohibits using the service or its output:
- to access content that is not publicly accessible, including pages behind a login, password, paywall, subscription or private API;
- to create or log in to accounts on third-party services, or to submit forms, purchases or bookings;
- to put an unreasonable or disproportionate load on any website;
- to surveil, track, profile or build dossiers on identifiable individuals, or to collect sensitive personal data;
- to harvest contact details for unsolicited messages;
- for automated purchasing of tickets or limited goods, or for manipulating reviews, rankings or engagement.
The policy also makes the customer responsible for checking that their use is lawful and for complying with the terms of the websites they access, to the extent those terms bind them. The provider may honor robots.txt, opt-out and takedown requests and may restrict any site. Read the full text on the shared [acceptable use](/acceptable-use) page. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. The review checks the intended use.
## Public price pages and competitor monitoring
Monitoring public price pages is a common request. Frame it narrowly: public pages, a modest schedule, robots.txt and terms checked per site, no logged-in or personalized prices, and no personal data. Do not name or target specific brands in shared workflows, and do not use the output for anything on the prohibited list above. If a site's terms forbid automated monitoring, use a licensed data source instead.
## Not legal advice
Everything above is a description of the landscape and of one provider's policy, written for people who build workflows. Laws differ by country and change, and the answer for your project depends on facts this guide cannot see. Before you schedule a job that collects other people's pages at scale, ask a lawyer.
When the checks pass, the operational guides take over: [scrape a list of URLs](/guides/scrape-list-of-urls) for the batch itself and [scheduled web scraping](/guides/scheduled-web-scraping) for the recurring run.
---
# Scrape multiple URLs in n8n: Batching or Loop Over Items
Source: https://agentscraper.cc/blog/n8n-scrape-multiple-urls
> Two ways to pace a URL list in n8n so a fetch API never answers 429: the HTTP Request node's Batching option, or Loop Over Items with a Wait node.
An n8n workflow that reads 300 URLs from a Sheet and sends them to an HTTP Request node will try to fetch all 300 as fast as it can. Any fetch API with a rate limit answers that with `429`, and the run either stops or fills your sheet with errors. There are two ways to pace it, and they suit different workflows.
The examples use Live Web Data API, whose rate limit counts URLs per second. The same settings apply to any API that limits calls per second.
## Option 1: Batching on the HTTP Request node
Open the HTTP Request node, then **Options → Batching**:
| Setting | Value |
| --- | --- |
| Items per Batch | your plan's rate limit in URLs per second (one URL per item) |
| Batch Interval (ms) | `1000` |
n8n then sends that many items, waits the interval and sends the next batch. Nothing else in the workflow changes. This is the right choice when only the fetch needs pacing and the rest of the workflow is fast.
The request body sends one URL per item:
```json
{ "urls": ["{{ $json.url }}"] }
```
The full node settings (Bearer Auth credential, JSON body, response format) are in the [no-code recipes](/go/docs-no-code?from=post:n8n-scrape-multiple-urls).
## Option 2: Loop Over Items with a Wait node
**Loop Over Items** (formerly Split in Batches) hands a fixed number of items at a time to the nodes inside the loop. Put a **Wait** node at the end of each pass and connect it back to the loop:
1. **Loop Over Items**, **Batch Size** = your plan's rate limit in URLs per second.
2. **HTTP Request** (the fetch).
3. **IF** on `{{ $json.statuses[0].status }}` equals `success`.
4. Your AI step and the **Google Sheets** update on the success branch; an error row on the other.
5. **Wait**, **Resume** = After Time Interval, 1 second, then back to **Loop Over Items**.
This is the right choice when several nodes must finish for one batch before the next starts, for example when each page also goes through a model and a Sheets write. The wait then covers the whole pass, so the pace is a little slower than the limit, which is fine.
## How long will the run take?
Pacing sets a floor on run time. With one URL per item, the minimum is the number of URLs divided by your rate limit:
| URLs | At 1 URL/s | At 2 URLs/s | At 5 URLs/s |
| --- | --- | --- | --- |
| 100 | 1 min 40 s | 50 s | 20 s |
| 500 | 8 min 20 s | 4 min 10 s | 1 min 40 s |
| 3,000 | 50 min | 25 min | 10 min |
Retries and slow pages add to that. If the number gets close to your instance's execution timeout, split the list: a parent workflow sends chunks of 20 to 50 URLs to a child workflow with **Execute Sub-workflow**, so each execution stays short. The [n8n web scraping guide](/guides/n8n-web-scraping) covers that setup.
## When you still see 429
Pacing removes most `429`s; it does not remove all of them, because other workflows may share the same key. Handle the rest with the same retry policy every guide on this site uses: retry `429`, `5xx` and `CRAWL_LIVECRAWL_TIMEOUT` up to 3 times with backoff, retry no other per-URL error in the same run, and stop the job on `402`, `401` or `403`. On a `429`, wait the number of seconds in the `Retry-After` header before trying again. The full table is in the [retry rules for URL lists](/guides/scrape-list-of-urls#retry-rules-table).
If `429`s show up on every run, lower **Items per Batch** or **Batch Size** instead of adding retries. A separate API key for the n8n workflows also makes it easier to see which job uses the budget.
## What this means for your workflow
- One URL per item, so routing and retries work per URL.
- Only the fetch needs pacing: **Options → Batching**, Items per Batch = rate limit, interval `1000` ms.
- Several nodes per batch: **Loop Over Items** plus a one-second **Wait**.
- Estimate run time before you schedule; split long lists into sub-workflow executions.
- Lower the pace when `429`s repeat; do not lean on retries.
---
# Add a robots.txt check step to your n8n or cron job
Source: https://agentscraper.cc/blog/robots-txt-check-step
> A per-host robots.txt check that runs before every scheduled collection job: a Python step for cron and a Code node for n8n, with the RFC 9309 edge cases.
Reading robots.txt once, before the first run, is a good start. It is not enough for a job that runs every morning for months: rules change, hosts get added to the list, and nobody rereads the file. The fix is a small step that runs before every fetch, checks each URL against its host's robots.txt and passes on only the allowed ones.
This post builds that step twice: as a Python script for cron, and as two nodes for n8n. It is a technical check, not a legal one. The [responsible web scraping guide](/guides/responsible-web-collection) covers terms of service and personal data, which robots.txt does not settle.
## The rules the step has to follow
RFC 9309 defines the parts that matter for a scheduled job:
| robots.txt response | What RFC 9309 says | What the step does |
| --- | --- | --- |
| `200` with rules | Follow the rules for your user agent, or the `*` group | Match each URL |
| `4xx` (for example `404`) | "Unavailable": the crawler may access any resource | Treat as no rules, except `401` and `403` |
| `5xx` or no connection | "Unreachable": assume complete disallow | Skip the host this run |
Two more details from the RFC: the most specific (longest) matching rule wins, and a crawler should not use a cached robots.txt for more than 24 hours. Fetching it fresh on every daily run covers that.
Python's standard `urllib.robotparser` differs from the RFC in two ways that matter here. Its `read()` method treats `401` and `403` as "disallow everything", which is stricter than the RFC and a safe default for a job that runs unattended, so both steps below adopt it. And its matcher applies rules in file order and does not expand `*` inside paths, so both steps use a small matcher of their own that follows the longest-match rule instead.
## A Python step for cron
The script reads a list of URLs, fetches robots.txt once per host, and writes the allowed URLs as a CSV with a `url` column, ready for the batch script from the docs. Skipped URLs go to standard error with the reason.
```python
import csv, re, sys
from urllib.parse import urlsplit
import requests
HEADERS = {"User-Agent": "example-collector/1.0 (+https://example.com/contact)"}
def star_rules(text):
"""Allow/Disallow rules of the * group, as (allow, pattern) pairs."""
rules, in_star, last_was_agent = [], False, False
for raw in text.splitlines():
m = re.match(r"^([A-Za-z-]+)\s*:\s*(.*)$", raw.split("#", 1)[0].strip())
if not m:
continue
key, val = m.group(1).lower(), m.group(2).strip()
if key == "user-agent":
in_star = (in_star and last_was_agent) or val == "*"
last_was_agent = True
else:
last_was_agent = False
if in_star and key in ("allow", "disallow") and val:
rules.append((key == "allow", val))
return rules
def allowed(rules, path):
best = None # longest match wins; Allow wins a tie
for allow, pat in rules:
if re.match("^" + re.escape(pat).replace(r"\*", ".*").replace(r"\$", "$"), path):
if best is None or len(pat) > len(best[1]) or (len(pat) == len(best[1]) and allow):
best = (allow, pat)
return best is None or best[0]
def load_rules(scheme, host):
try:
r = requests.get(f"{scheme}://{host}/robots.txt", headers=HEADERS, timeout=10)
except requests.RequestException:
return None, "robots.txt unreachable"
if r.status_code >= 500 or r.status_code in (401, 403):
return None, f"robots.txt HTTP {r.status_code}"
return ([] if r.status_code >= 400 else star_rules(r.text)), None
cache, out = {}, csv.writer(sys.stdout)
out.writerow(["url"])
for line in open(sys.argv[1], encoding="utf-8"):
url = line.strip()
if not url:
continue
p = urlsplit(url)
key = (p.scheme, p.netloc.lower())
if key not in cache:
cache[key] = load_rules(*key)
rules, reason = cache[key]
path = (p.path or "/") + (f"?{p.query}" if p.query else "")
if rules is not None and allowed(rules, path):
out.writerow([url])
else:
print(f"skip {url}: {reason or 'disallowed by robots.txt'}", file=sys.stderr)
```
Chain it in front of the fetch in the crontab, with the key loaded from a protected env file:
```bash
0 6 * * * cd /opt/jobs && set -a && . ./.env && set +a && python3 robots_check.py urls.txt > allowed.csv 2>> skipped.log && python3 batch_from_csv.py allowed.csv --out results-$(date +\%F).jsonl >> collect.log 2>&1
```
`skipped.log` is your record of what the job left out and why. If one host shows up there every day, remove it from the list or ask the operator.
## The same check in n8n
In n8n, put two nodes between the node that reads your list and the fetch.
1. **HTTP Request**, method `GET`, URL `{{ new URL($json.url).origin }}/robots.txt`. Under **Options → Response**, set **Response Format** to `Text`, turn on **Include Response Headers and Status** and **Never Error**, so a `404` does not stop the run.
2. **Code**, mode **Run Once for Each Item**, with the matcher below. Then an **IF** node on `{{ $json.robotsAllowed }}`.
```javascript
// Rename "Read URLs" to the node that holds your list.
const url = new URL($('Read URLs').item.json.url);
const status = $json.statusCode;
const path = url.pathname + url.search;
if (status >= 500 || status === 401 || status === 403) return { json: { url: url.href, robotsAllowed: false } };
if (status >= 400) return { json: { url: url.href, robotsAllowed: true } };
const rules = [];
let inStar = false, lastWasAgent = false;
for (const raw of String($json.body || "").split(/\r?\n/)) {
const m = raw.replace(/#.*/, "").trim().match(/^([A-Za-z-]+)\s*:\s*(.*)$/);
if (!m) continue;
const key = m[1].toLowerCase(), val = m[2].trim();
if (key === "user-agent") {
if (!lastWasAgent) inStar = false;
if (val === "*") inStar = true;
lastWasAgent = true;
} else {
lastWasAgent = false;
if (inStar && (key === "allow" || key === "disallow") && val) rules.push({ allow: key === "allow", val });
}
}
const toRe = (p) => new RegExp("^" + p.replace(/[.+?^{}()|[\]\\]/g, "\\$&").replace(/\*/g, ".*"));
let best = null;
for (const r of rules) {
if (!toRe(r.val).test(path)) continue;
if (!best || r.val.length > best.val.length || (r.val.length === best.val.length && r.allow)) best = r;
}
return { json: { url: url.href, robotsAllowed: !best || best.allow } };
```
Like the Python step, the matcher reads only the `*` group, supports `*` and a trailing `$` in paths, and lets the longest match win, with `Allow` winning a tie, as RFC 9309 describes. This version fetches robots.txt once per URL. For long lists with many URLs per host, run the check once per host in a sub-workflow, or call the Python step instead.
## What this means for your workflow
- Check robots.txt on every run, not once at launch.
- Fetch it once per host per run; never trust a copy older than 24 hours.
- `5xx`, no connection, `401` or `403`: skip the host this run. Other `4xx`: no rules.
- Log every skipped URL with its reason, and review the log.
- A passing check is one item on the list. Terms of service and personal data still need a human read; this is not legal advice.
---
# Live Web Data API — full documentation
# Overview
Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.
## What it does
You send up to 10 URLs (and optionally a country). We fetch each page through a residential or mobile connection, extract the main content and return it in the format your agent or pipeline needs. Successful pages cost 1 credit; failures cost nothing.
| | Value |
| --- | --- |
| Base URL | `https://livewebdata.com/api` |
| Endpoints | `POST /v1/contents`, `GET /v1/account` |
| Formats | markdown and text by default; html with `html: true` |
| OpenAPI | [`/openapi.json`](https://agentscraper.cc/openapi.json) · [`/openapi.yaml`](https://agentscraper.cc/openapi.yaml) |
| Fetch strategy (v1) | static fetch (`fast`), a retry through another connection (`retry`), or a real browser (`browser`) with `render` or `renderFallback` |
## Authentication
Create a key in the dashboard under [API keys](https://agentscraper.cc/dashboard/keys) and send it as a bearer token. Keys start with `1337_live_` (test keys with `1337_test_`) and are shown only once. Keep them in an environment variable such as `LIVE_WEB_DATA_API_KEY` — never in client-side code or repositories.
```bash title="Headers"
Authorization: Bearer 1337_live_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json
```
## Quickstart
Fetch a page as Markdown:
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/article"]}'
```
The response contains one entry in `results` per successful page and one entry in `statuses` per requested URL. The page HTML is included only when you send `"html": true`:
```json title="Response (abridged)"
{
"requestId": "req_01J…",
"results": [{
"id": "https://example.com/article",
"url": "https://example.com/article",
"title": "Article",
"markdown": "# Article\n\n…",
"text": "Article\n\n…"
}],
"statuses": [{
"id": "https://example.com/article",
"status": "success",
"source": "livecrawled",
"fetchedAt": "2026-10-23T09:15:02.118Z",
"httpStatusCode": 200,
"strategy": "fast",
"credits": 1,
"error": null
}],
"searchTime": 1240
}
```
When `text` or `markdown` was cut at `text.maxCharacters`, `results[].textTruncated` is `true`. `results[].warnings` (absent when empty) flags pages that could only be partly retrieved (`PAGE_TRUNCATED`), probably need `render: true` (`THIN_CONTENT`), were converted as plain text (`EXTRACTION_FALLBACK`) or lost their `html` (and maybe part of their text) to the response's 24 MiB size limit (`RESPONSE_SIZE_LIMIT`).
See the [API reference](https://agentscraper.cc/docs/api-reference) for every parameter, or try it without code in the dashboard [playground](https://agentscraper.cc/dashboard/playground).
## SDKs
Thin, typed clients for JavaScript/TypeScript and Python with built-in retries and typed errors. See [SDKs](https://agentscraper.cc/docs/sdks).
```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";
const client = new Teneo({
apiKey: process.env.LIVE_WEB_DATA_API_KEY,
baseUrl: "https://livewebdata.com/api",
});
const res = await client.contents({
urls: ["https://example.com/article"]
});
for (const s of res.statuses) console.log(s.id, s.status, s.error?.tag ?? "");
console.log(res.results[0]?.markdown);
```
```python title="Python SDK"
import os
from teneo import Teneo
client = Teneo(
api_key=os.environ["LIVE_WEB_DATA_API_KEY"],
base_url="https://livewebdata.com/api",
)
res = client.contents(
urls=["https://example.com/article"],
)
for s in res["statuses"]:
print(s["id"], s["status"], (s["error"] or {}).get("tag", ""))
print(res["results"][0]["markdown"] if res["results"] else "")
```
## Live fetching
Every request is fetched live; there is no cache.
## Geotargeting
Set `countryCode` to an ISO-3166-1 alpha-2 code (for example `de`, `jp`, `br`) to fetch the page as seen from that country. The default is `us`.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/pricing"
],
"countryCode": "de"
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/pricing"],
"countryCode": "de"
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/pricing"
],
"countryCode": "de"
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Multiple URLs
Send up to 10 URLs in one request. They are processed in parallel; `statuses` has one entry per URL (matched by `id`) and credits are charged per successful URL. A restricted URL gets a per-URL `RESTRICTED_TARGET` status instead of failing the whole request. More than 10 URLs return `400 TOO_MANY_URLS`. Rate limits count URLs, not HTTP requests.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Restricted targets
Some domain categories are blocked for legal and regulatory reasons. Requests are rejected with `451` and cost no credits. Repeated attempts lead to automatic key suspension. Report abuse to abuse@livewebdata.com.
## MCP for agents
Agents can use the API as a tool over MCP with your existing key — two tools, `get_page_contents` and `get_account_status`. Setup for Claude Code, Cursor, VS Code and Claude Desktop: [MCP for agents](https://agentscraper.cc/docs/mcp).
## For AI coding agents
Every docs page is also available as plain Markdown by appending `.md` to its URL (e.g. [`/docs/api-reference.md`](https://agentscraper.cc/docs/api-reference.md)). An index for LLMs lives at [`/llms.txt`](https://agentscraper.cc/llms.txt) and the full docs in one file at [`/llms-full.txt`](https://agentscraper.cc/llms-full.txt).
---
# API reference
Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.
## Endpoint
```bash title="Request"
POST https://livewebdata.com/api/v1/contents
Authorization: Bearer
Content-Type: application/json
```
Machine-readable specs:
- OpenAPI 3.1 (JSON): https://agentscraper.cc/openapi.json
- OpenAPI 3.1 (YAML): https://agentscraper.cc/openapi.yaml
- Postman collection: https://agentscraper.cc/live-web-data-api.postman_collection.json
- Interactive reference: https://agentscraper.cc/docs/api-reference/interactive
## Request parameters
| Parameter | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | `string[]` | yes | — | The pages to fetch, 1–10 http or https URLs of at most 8,192 characters each. Processed in parallel; one entry per URL in results/statuses. Duplicates (same URL, ignoring #fragment) are fetched once; repeated positions cost 0 credits. |
| `countryCode` | `string` | no | "us" | ISO-3166-1 alpha-2 code. The page is fetched through a connection in this country. |
| `text` | `boolean \| object` | no | true | Content options. Pass an object to set maxCharacters and verbosity. |
| `text.maxCharacters` | `integer` | no | 100000 | Truncates text and markdown to this many characters (max 1,000,000); `results[].textTruncated` tells you when it did. The MCP tool defaults to 8,000 (`max_characters`). |
| `text.verbosity` | `"compact" \| "full"` | no | "compact" | compact = main content only. full = whole body incl. navigation and footer. |
| `html` | `boolean` | no | false | Include the page HTML as served in results[].html. A response carries at most 24 MiB of JSON: past that, the remaining results come without html (warning RESPONSE_SIZE_LIMIT). |
| `livecrawlTimeout` | `integer (ms)` | no | 30000 (60000 with render) | Maximum time for a live fetch, 1–90000 ms. A lower value is raised to 30000, the time the browser fallback needs, unless you send renderFallback: false (then any value, default 10000). With render or renderFallback: true at least 30000. |
| `render` | `boolean` | no | false | Render every URL in a real browser, as a normal visitor sees it: 3 credits, 5 when the page needed a retry through another connection. Cannot be combined with renderFallback. |
| `renderFallback` | `boolean` | no | on | Fetch normally first and render in a real browser only when the first attempt did not return the page, shows no or very little visible text, or was blocked. On by default: omit it. Send false for a plain fetch only (never a browser). A normal success costs its normal 1 or 2 credits, a rendered one only the render credits; statuses[].fallbackUsed tells you when it rendered. Cannot be combined with render: true. |
```json title="Example body"
{
"urls": ["https://example.com/article"],
"countryCode": "de",
"text": { "maxCharacters": 50000, "verbosity": "compact" },
"livecrawlTimeout": 45000
}
```
Unknown fields are rejected with `400 INVALID_REQUEST`; the message names the field. So is a URL longer than 8,192 characters (`urls.0: Each URL may be at most 8,192 characters.`).
## Response
| Field | Type | Description |
| --- | --- | --- |
| `requestId` | `string` | Id of this request, the same for every repeat with the same Idempotency-Key. Also returned as X-Request-Id. |
| `results[]` | `object[]` | One entry per successfully fetched URL. |
| `results[].url` | `string` | Final URL after redirects. |
| `results[].title` | `string \| null` | Page title. |
| `results[].author / publishedDate / image` | `string \| null` | Metadata when the page exposes it. |
| `results[].favicon` | `string \| null` | Favicon URL. |
| `results[].markdown` | `string` | Main content as Markdown. |
| `results[].text` | `string` | Plain text, tags removed, whitespace normalized. |
| `results[].html` | `string` | Page HTML as served. Only with html: true, and only while the response stays within its 24 MiB size limit (else warning RESPONSE_SIZE_LIMIT). |
| `results[].textTruncated` | `boolean` | true when text or markdown was cut at text.maxCharacters, or at the response size limit (warning RESPONSE_SIZE_LIMIT). Present whenever text is enabled. |
| `results[].warnings` | `{ code, message }[]` | Absent when empty. "PAGE_TRUNCATED" = only part of the page could be retrieved; content after that point is missing. "THIN_CONTENT" = very little text, the page probably needs render: true. "EXTRACTION_FALLBACK" = the page was too complex to convert in time; text is plain and markdown has no structure. "RESPONSE_SIZE_LIMIT" = the response reached its 24 MiB size limit: this result has no html, or no html and cut text/markdown. |
| `statuses[]` | `object[]` | One entry per requested URL, success or error. |
| `statuses[].status` | `"success" \| "error"` | Outcome for this URL. |
| `statuses[].source` | `"livecrawled" \| null` | Always "livecrawled" on success: every request is fetched live. null on error. |
| `statuses[].fetchedAt` | `string \| null` | When the page was fetched (ISO 8601). null on error. |
| `statuses[].strategy` | `"fast" \| "retry" \| "browser"` | Fetch strategy. "fast" = standard page (1 credit); "retry" = the first attempt did not return the page, a retry through another connection did (2 credits); "browser" = rendered in a real browser, as a normal visitor sees it (3 credits, 5 when it needed a retry through another connection). |
| `statuses[].credits` | `integer` | Credits charged for this URL: 1 standard, 2 retried, 3 rendered, 5 rendered + retried, 0 on error and for a short page (thin). |
| `statuses[].fallbackUsed` | `boolean` | Present (true) only when the browser fallback ran for this URL. With strategy "browser" the content is that render (fast → render); otherwise rendering was tried and did not help. |
| `statuses[].thin` | `boolean` | Present (true) only for a short page: fewer than 500 characters of visible text even after rendering. Delivered as a success at 0 credits, with only that visible text. |
| `statuses[].error` | `{ tag, message } \| null` | Per-URL error, see crawl errors. |
| `searchTime` | `integer (ms)` | Total processing time. |
Browser fallback is on by default: a page the fast fetch cannot read (no or very little visible text, blocked, 401/403, a timeout) is rendered in a real browser and charged only the render credits. `statuses[].fallbackUsed` is `true` when that happened (`strategy: "browser"` = the content is the render). Send `renderFallback: false` for a fast fetch only. A page that still shows fewer than 500 characters of visible text after rendering is delivered as a success with `statuses[].thin: true` at 0 credits, holding only that visible text — it is not an error.
`text.maxCharacters` (default 100,000, max 1,000,000) cuts `text` and `markdown`; `results[].textTruncated` is `true` when it did. A response carries at most 24 MiB of JSON: from the result that would pass it on, results come without `html`, and if that is not enough, `text` and `markdown` are cut (`textTruncated: true`), each such result flagged `RESPONSE_SIZE_LIMIT`. The request never fails because of it. `title` and `author` are at most 2,000 characters each; an `image` or `favicon` URL longer than 4,096 characters is left out (`favicon` then falls back to `/favicon.ico`). The MCP tool `get_page_contents` has its own, smaller default of 8,000 (`max_characters`), because its result goes into an agent's context. `results[].warnings` is present only when there is something to know about a page:
| Code | Meaning | What to do |
| --- | --- | --- |
| `PAGE_TRUNCATED` | Only part of the page could be retrieved (size limit or an interrupted transfer); content after that point is missing. | Nothing to retry; the result is the part that arrived. |
| `THIN_CONTENT` | Very little text next to scripts or an empty app root: the page probably loads its content with JavaScript. The URL is still charged as a success. | Retry with `render: true`. The default browser fallback renders a page with fewer than 500 characters of visible text by itself, not one with more. |
| `EXTRACTION_FALLBACK` | The page structure was too complex to convert in time; `text` is plain and `markdown` carries no structure. | Use `text`. |
| `RESPONSE_SIZE_LIMIT` | The response reached its 24 MiB size limit. This result and every later one come without `html`; when that was not enough, `text` and `markdown` were cut too (`textTruncated: true`). The URL is still a success and is charged. | Request fewer URLs at once, or without `html`. |
## Response headers
| Header | Description |
| --- | --- |
| `X-Request-Id` | Same as requestId in the body. |
| `X-Credits-Remaining` | Credit balance after this request. |
| `X-RateLimit-Limit` | URLs you can send at once (bucket size, = rateLimit.burst in GET /v1/account). The bucket refills at rateLimit.requestsPerSecond URLs per second. |
| `X-RateLimit-Remaining` | URLs you can send right now. |
| `Retry-After` | Seconds to wait: before retrying a 429, or before repeating a 202 with the same Idempotency-Key. |
| `Idempotency-Key` | The request's Idempotency-Key: yours, or one the API created when you sent none. On every answer (200, 202, errors) except a 400 for a malformed key and a 413 for a body over 10 MiB (refused unread). Repeat a 202 with it. |
## Errors
### Per-URL errors
Returned inside `statuses[].error` with HTTP 200. These never cost credits.
| Tag | Meaning |
| --- | --- |
| `CRAWL_NOT_FOUND` | The page returned 404/410. |
| `SOURCE_NOT_AVAILABLE` | The target refused access (e.g. 403 or 429) after retries. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The live fetch did not finish within livecrawlTimeout. |
| `CRAWL_NOT_RENDERED` | No usable content, e.g. the page needs JavaScript rendering (retry without renderFallback: false, or with render: true), or a rendered page shows no visible text. 0 credits. |
| `UNSUPPORTED_CONTENT_TYPE` | The URL does not serve HTML or text. |
| `CRAWL_UNKNOWN_ERROR` | Something else went wrong. Safe to retry. |
| `SOURCE_BLOCKED` | Access denied by the site, or it answered with an interstitial page (HTTP 202) instead of the content. 0 credits. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable; no credits were charged. |
| `RATE_LIMITED` | Rendering limit for your plan reached; no credits were charged. |
| `RESTRICTED_TARGET` | This URL is restricted. Other URLs in the same request are still processed. 0 credits. |
### Request errors
Returned with a non-200 status and no results.
| HTTP | Tag | Meaning |
| --- | --- | --- |
| 400 | `INVALID_REQUEST` | Invalid parameters. |
| 400 | `TOO_MANY_URLS` | More URLs than allowed per request (10). |
| 401 | `UNAUTHORIZED` | Missing or invalid API key. |
| 402 | `INSUFFICIENT_CREDITS` | Not enough credits. Top up or upgrade. |
| 403 | `KEY_SUSPENDED` | The key or account is suspended or not yet approved. |
| 429 | `RATE_LIMITED` | Rate limit exceeded. Retry after the Retry-After header. |
| 409 | `IDEMPOTENCY_CONFLICT` | The Idempotency-Key was already used with a different request, or its result is no longer available (results are kept 15 minutes). Use a new key. |
| 413 | `PAYLOAD_TOO_LARGE` | The request body is larger than 10 MiB. Nothing was read or charged. |
| 451 | `RESTRICTED_TARGET` | All requested URLs are restricted. No credits charged. |
| 503 | `SERVICE_UNAVAILABLE` | Temporarily unavailable. Retry shortly. |
```json title="Error body"
{
"error": {
"code": 402,
"tag": "INSUFFICIENT_CREDITS",
"message": "Not enough credits. Top up or upgrade your plan."
},
"requestId": "req_01J…"
}
```
## Long requests and Idempotency-Key
Most requests answer within seconds. A request still fetching after about 100 seconds (slow sites, rendering, a long `livecrawlTimeout`) answers `202` instead of making you wait on an open connection:
```json title="202 Accepted"
{
"status": "pending",
"requestId": "req_01J…",
"idempotencyKey": "lwda_3f1c2b7e-0d4a-4c1e-9a55-2b8f0e6c7d10",
"retryAfter": 1,
"message": "Still fetching. Send the same request again with the same Idempotency-Key header to get the result; it is not charged again."
}
```
Send the **same request body** again with the header `Idempotency-Key: ` after `Retry-After` seconds. The repeat waits for the fetch and answers `200` with the result, or `202` again if it is still running. A repeat is never charged again. The SDKs and the MCP server do this for you.
**Always send your own `Idempotency-Key`** (8–128 characters of `A–Z a–z 0–9 _ - : .`, for example a UUID) on the first request, and keep it until you have the answer. A connection can break after the API received the request: a network error, a timeout on your side, a deploy. Send the same request again with the same key, and you get that fetch (its result, or `202` while it still runs) instead of a new one, so you pay once. Without your own key, a cut connection is a lost answer: the retry has no key, so it is a new request and charged again.
Every answer of `POST /v1/contents` (`200`, `202` and errors) carries the request's key in the `Idempotency-Key` response header: yours, or one the API created when you sent none. The only exceptions are the `400` for a malformed key and the `413` for a request body over 10 MiB, which is refused before it is read (nothing was charged, there is nothing to repeat). The SDKs create a key for each `contents()` call when you pass none, send it on every retry and put it on the error they raise (`idempotencyKey` / `idempotency_key`).
- Results are kept for **15 minutes**. A result nobody collects stays charged; after 15 minutes a repeat answers `409 IDEMPOTENCY_CONFLICT`, and a new key fetches the pages again.
- A key is bound to its request body for 24 hours: reusing it with a different body answers `409 IDEMPOTENCY_CONFLICT`. Use a new key for every new request.
## Rate limits
Limits are per account and count **URLs per second** (a request with 3 URLs uses 3). When exceeded, the API returns `429` with a `Retry-After` header (seconds). Limits work as a bucket: `X-RateLimit-Limit` is how many URLs you can send at once (`rateLimit.burst` in `GET /v1/account`), `X-RateLimit-Remaining` how many right now, and the bucket refills at your plan's URLs per second (`rateLimit.requestsPerSecond`).
| Plan | Price / month | Rate limit (URLs/s) |
| --- | --- | --- |
| Lite | $10 | 1 |
| Starter | $100 | 2 |
| Developer | $500 | 5 |
| Business | $900 | 10 |
| Enterprise | $2,250 | 20 |
| Custom | Custom | Custom |
## Credits
| Request | Credits |
| --- | --- |
| Standard page (static / server-rendered) | 1 |
| Retried page (succeeded on a retry through another connection) | 2 |
| JS-rendered page | 3 |
| Rendered + retried page | 5 |
| Short page (under 500 characters of visible text; delivered as that text, a success) | 0 |
| Failed request (incl. access denied by the site) | 0 |
- Plan credits reset monthly.
- Top-up credits never expire.
When your balance is too low for the URLs in a request, the API returns `402 INSUFFICIENT_CREDITS`. The current balance is returned in `X-Credits-Remaining` on every response.
## Account
`GET https://livewebdata.com/api/v1/account` returns your plan, remaining credits and rate limit. It is free — call it before large jobs.
```json title="GET /v1/account"
{
"plan": "starter",
"status": "active",
"credits": { "plan": 81234, "topup": 10000, "total": 91234 },
"rateLimit": { "requestsPerSecond": 2, "burst": 4 },
"periodEndsAt": "2026-11-01T00:00:00Z"
}
```
---
# Examples
Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.
## Basic fetch
The smallest possible request — one URL, defaults for everything else.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/article"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/article"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/article"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Geotargeting
Fetch a page as seen from Japan.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/pricing"
],
"countryCode": "jp"
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/pricing"],
"countryCode": "jp"
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/pricing"
],
"countryCode": "jp"
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Text limits
Return at most 5,000 characters of the full page body, including navigation and footer.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/docs/getting-started"
],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/docs/getting-started"],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/docs/getting-started"
],
"text": {
"maxCharacters": 5000,
"verbosity": "full"
}
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Several URLs
Fetch up to 10 pages in one request. Each URL gets its own status; only successful pages cost credits.
```bash title="cURL"
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}'
```
```python title="Python"
import requests
response = requests.post(
"https://livewebdata.com/api/v1/contents",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"urls": ["https://example.com/a", "https://example.com/b", "https://example.com/blocked"]
},
timeout=60,
)
data = response.json()
print(data["results"][0]["markdown"])
```
```javascript title="JavaScript"
const response = await fetch("https://livewebdata.com/api/v1/contents", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
"urls": [
"https://example.com/a",
"https://example.com/b",
"https://example.com/blocked"
]
}),
});
const data = await response.json();
console.log(data.results[0]?.markdown);
```
## Whole jobs
For URL lists, retries and scheduling see [Automate requests](https://agentscraper.cc/docs/automation) — including ready-made Python and Node scripts.
---
# Automate requests
Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.
## From playground to code
Every run in the [playground](https://agentscraper.cc/dashboard/playground) shows a **Use it in code** panel with cURL, Python, JavaScript and SDK snippets for exactly the request you just made. Start there, then move to the scripts below.
Download all example scripts: https://agentscraper.cc/live-web-data-api-examples.zip
| File | What it does |
| --- | --- |
| `examples/python/quickstart.py` | One request, prints title + first 500 chars of markdown |
| `examples/python/batch_from_csv.py` | Reads `urls.csv` (`url`, optional `country`), fetches with concurrency = rate limit, retries with backoff, writes `results.jsonl` and prints a summary |
| `examples/node/quickstart.mjs` | Same as the Python quickstart, native `fetch`, no dependencies |
| `examples/node/batch_from_csv.mjs` | Same as the Python batch script, no dependencies |
| `examples/urls.csv` | 10 sample URLs incl. a restricted, a missing and a timing-out URL to show error handling |
```bash title="Run the batch script"
export LIVE_WEB_DATA_API_KEY=1337_live_… # never hard-code keys
python examples/python/batch_from_csv.py examples/urls.csv
node examples/node/batch_from_csv.mjs examples/urls.csv
```
## Process a list of URLs
The simplest loop: one request per URL, one after the other.
```python title="Python"
import os, requests
API = os.environ.get("LIVE_WEB_DATA_API_BASE_URL", "https://livewebdata.com/api")
headers = {"Authorization": f"Bearer {os.environ['LIVE_WEB_DATA_API_KEY']}"}
for url in ["https://example.com/a", "https://example.com/b"]:
r = requests.post(f"{API}/v1/contents", json={"urls": [url]}, headers=headers, timeout=60)
status = r.json()["statuses"][0]
print(url, status["status"], status["error"] and status["error"]["tag"])
```
```javascript title="Node"
const API = process.env.LIVE_WEB_DATA_API_BASE_URL ?? "https://livewebdata.com/api";
const headers = { Authorization: `Bearer ${process.env.LIVE_WEB_DATA_API_KEY}`, "Content-Type": "application/json" };
for (const url of ["https://example.com/a", "https://example.com/b"]) {
const res = await fetch(`${API}/v1/contents`, { method: "POST", headers, body: JSON.stringify({ urls: [url] }) });
const { statuses } = await res.json();
console.log(url, statuses[0].status, statuses[0].error?.tag ?? "");
}
```
You can also send up to 10 URLs per request — see [Several URLs](https://agentscraper.cc/docs/examples#several-urls).
## Respect rate limits
- Keep a steady pace of **your plan's URLs per second** (from `GET /v1/account` → `rateLimit.requestsPerSecond`). Bursts up to `rateLimit.burst` URLs (the bucket size, also sent as `X-RateLimit-Limit`) are fine; the bucket refills at that rate.
- Watch `X-RateLimit-Remaining` on each response: it is how many URLs you can send right now.
- On `429`, wait `Retry-After` seconds and retry the same request.
## Retries
Retry `429`, `5xx` and per-URL `CRAWL_LIVECRAWL_TIMEOUT` up to **3 times** with exponential backoff and jitter (e.g. `0.5s · 2^n + random(0–0.25s)`). Do **not** retry:
| Outcome | Why |
| --- | --- |
| `451 RESTRICTED_TARGET` | The domain is blocked; repeated attempts can suspend your key |
| `400 INVALID_REQUEST` / `TOO_MANY_URLS` | Fix the request first |
| `401` / `403` | Key invalid, revoked or suspended |
| `402 INSUFFICIENT_CREDITS` | Stop the job cleanly, top up, resume |
| `CRAWL_NOT_FOUND` | The page doesn't exist |
**Send your own `Idempotency-Key`** with every `POST /v1/contents` (one per logical request, for example per URL of your list) and send the same key on every retry of that request. A connection cut after the API received the request (a network error, a timeout, a deploy) is then retried without a second charge: the repeat returns that fetch's result. A `202` (still fetching) is repeated the same way, after `Retry-After`. Use a **new** key for a new attempt at a URL that already has an answer, such as a per-URL `CRAWL_LIVECRAWL_TIMEOUT`: the same key returns the same answer. See [Long requests and Idempotency-Key](https://agentscraper.cc/docs/api-reference#long-requests-and-idempotency-key).
The [SDKs](https://agentscraper.cc/docs/sdks) implement this policy for you.
## Check your balance first
Call `GET /v1/account` before a large job and compare `credits.total` with the number of URLs. Stop cleanly when a request returns `402`.
```bash title="cURL"
curl https://livewebdata.com/api/v1/account -H "Authorization: Bearer $LIVE_WEB_DATA_API_KEY"
```
## Scheduling
Any scheduler works — the scripts are plain processes that exit when done.
```bash title="cron (daily at 06:00)"
0 6 * * * cd /opt/jobs && LIVE_WEB_DATA_API_KEY=… /usr/bin/python3 batch_from_csv.py urls.csv --out results.jsonl
```
```yaml title=".github/workflows/fetch.yml"
name: Fetch pages
on:
schedule: [{ cron: "0 6 * * *" }]
workflow_dispatch:
jobs:
fetch:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: node examples/node/batch_from_csv.mjs examples/urls.csv --out results.jsonl
env:
LIVE_WEB_DATA_API_KEY: ${{ secrets.LIVE_WEB_DATA_API_KEY }}
```
Other schedulers (Cloud Scheduler, Kubernetes CronJob, n8n, Airflow) run the same command.
## Keep your key safe
- Store it in an environment variable (`LIVE_WEB_DATA_API_KEY`) or your platform's secret manager.
- Never put it in client-side code, mobile apps, browser extensions or repositories.
- Use one key per environment (e.g. production, staging, n8n) — you can label keys in the dashboard and see the label in the usage log.
- Rotate a key immediately if it leaks: create a new one, then revoke the old one.
---
# SDKs
Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.
## Install
The SDKs live in the repository under `packages/` and are not published yet.
```bash title="JavaScript / TypeScript"
cd packages/sdk-js && npm install && npm run build
npm install ./packages/sdk-js # in your project
```
```bash title="Python"
pip install ./packages/sdk-python # requires httpx
```
## JavaScript / TypeScript
```javascript title="JS SDK"
import { Teneo } from "@teneo/contents";
const client = new Teneo({ apiKey: process.env.LIVE_WEB_DATA_API_KEY, baseUrl: "https://livewebdata.com/api" });
const page = await client.getPage("https://example.com/article", { country: "de", format: "markdown" });
console.log(page.title, page.credits);
const res = await client.contents({ urls: ["https://example.com/a", "https://example.com/b"] });
const account = await client.account();
```
## Python
```python title="Python SDK"
import os
from teneo import Teneo
client = Teneo(api_key=os.environ["LIVE_WEB_DATA_API_KEY"], base_url="https://livewebdata.com/api")
page = client.get_page("https://example.com/article", country="de", format="markdown")
print(page.title, page.credits)
res = client.contents(urls=["https://example.com/a", "https://example.com/b"])
account = client.account()
```
Both clients read the key from `LIVE_WEB_DATA_API_KEY` and the base URL from `LIVE_WEB_DATA_API_BASE_URL` when you don't pass them.
## Methods
| JS | Python | Description |
| --- | --- | --- |
| `contents(request, { idempotencyKey })` | `contents(urls=…, country_code=…, text=…, html=…, livecrawl_timeout=…, render=…, render_fallback=…, idempotency_key=…)` | Raw `POST /v1/contents` (1–10 URLs). `render_fallback=False` / `renderFallback: false` turns the default browser fallback off |
| `getPage(url, opts)` | `get_page(url, country=…, format=…)` | One URL → `{ title, url, content, credits, source }`; raises if that URL failed. `format: "html"` requests the HTML (`html: true`) |
| `account()` | `account()` | `GET /v1/account` |
## Errors
| Error | When |
| --- | --- |
| `TeneoAuthError` | 401 / 403 — key invalid, revoked or suspended |
| `TeneoInsufficientCreditsError` | 402 |
| `TeneoRateLimitError` | 429 after all retries (`retryAfter` / `retry_after`) |
| `TeneoIdempotencyError` | 409 — the idempotency key was used with a different request, or its result expired (15 minutes) |
| `TeneoRestrictedTargetError` | 451 |
| `TeneoConnectionError` | Network error or timeout after all retries, or a `2xx` answer cut while it was read. The API may have received the request: repeat it with the error's `idempotencyKey` / `idempotency_key` |
| `TeneoPendingError` | Still `202` when `maxWaitMs` / `max_wait` ran out; `idempotencyKey` / `idempotency_key` collects the result later |
| `TeneoPageError` | `getPage` / `get_page`: the URL failed (`tag`, e.g. `CRAWL_NOT_FOUND`) |
| `TeneoAPIError` | Base class (status, tag, requestId, and `idempotencyKey` / `idempotency_key`: the key of a `contents()` call, else `null` / `None`) |
## Retries
Both SDKs retry `429` (respecting `Retry-After`), `5xx`, network errors, a `2xx` answer cut while it is read and — for `getPage` — `CRAWL_LIVECRAWL_TIMEOUT`, up to 3 times with exponential backoff and jitter. `400`, `401`, `402`, `403`, `409` and `451` are never retried. Configure with `maxRetries` / `max_retries`.
Every attempt of one `contents()` call sends the same `Idempotency-Key`, so a retry picks up the first fetch and is never charged twice (render requests included). A fetch that runs longer than about 100 seconds answers `202`; the SDKs repeat it after `Retry-After` until the result arrives, for up to 5 minutes (`maxWaitMs` / `max_wait`). These repeats do not count as retries. See [Long requests](https://agentscraper.cc/docs/api-reference#long-requests-and-idempotency-key).
Without your own key, the SDKs create one per `contents()` call. When the retries run out, the error carries that key: call `contents()` again with the same request and `{ idempotencyKey: e.idempotencyKey }` / `idempotency_key=e.idempotency_key` within 15 minutes, and a fetch the API already received returns its result instead of being charged again. **Better: always pass your own key** (one per logical request, for example per row of your input), so a request cut by a network error, a timeout or a deploy can be retried without a second charge even after your process restarts.
---
# MCP for agents
Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.
## What is MCP
The Model Context Protocol (MCP) is an open standard that lets AI agents discover and call external tools. Our MCP server exposes the Contents API as two tools, so an agent can read any public web page on its own.
## Server & auth
| | Value |
| --- | --- |
| Server URL | `https://agentscraper.cc/mcp` |
| Transport | Streamable HTTP (stateless): JSON-RPC over `POST`; `GET` and `DELETE` answer `405` |
| Auth | `Authorization: Bearer ` |
| Keys | Same keys as the REST API — create one under [API keys](https://agentscraper.cc/dashboard/keys) |
Missing or invalid keys are rejected with HTTP 401 (suspended accounts: 403, "API access for this account is suspended. Contact support.") before any tool runs.
## Client setup
Replace `YOUR_KEY` with your API key. In the dashboard, the [Integrations](https://agentscraper.cc/dashboard/integrations) page fills in your key for you.
### Claude Code
Run this in your terminal. The server is then available in every Claude Code session for this project.
```bash
claude mcp add --transport http live-web-data https://agentscraper.cc/mcp \
--header "Authorization: Bearer YOUR_KEY"
```
Add `--scope user` to make it available in all projects. Check with `claude mcp list`.
### Cursor
Add this to your Cursor MCP config, globally or per project.
File: `~/.cursor/mcp.json or .cursor/mcp.json`
```json
{
"mcpServers": {
"live-web-data": {
"url": "https://agentscraper.cc/mcp",
"headers": {
"Authorization": "Bearer YOUR_KEY"
}
}
}
}
```
Prefer an env var: `"Bearer ${env:LIVE_WEB_DATA_API_KEY}"`.
### VS Code
Add this to your workspace. VS Code asks for the key the first time the server starts and stores it securely.
File: `.vscode/mcp.json`
```json
{
"inputs": [
{
"type": "promptString",
"id": "live-web-data-key",
"description": "Live Web Data API key",
"password": true
}
],
"servers": {
"live-web-data": {
"type": "http",
"url": "https://agentscraper.cc/mcp",
"headers": {
"Authorization": "Bearer ${input:live-web-data-key}"
}
}
}
}
```
Uses the top-level key `servers` (not `mcpServers`). Paste `YOUR_KEY` when prompted.
### Claude Desktop
Claude Desktop connects to remote servers through the mcp-remote bridge. Add this to your Claude Desktop config and restart the app.
File: `claude_desktop_config.json`
```json
{
"mcpServers": {
"live-web-data": {
"command": "npx",
"args": [
"-y",
"mcp-remote",
"https://agentscraper.cc/mcp",
"--header",
"Authorization:${AUTH_HEADER}",
"--transport",
"http-only"
],
"env": {
"AUTH_HEADER": "Bearer YOUR_KEY"
}
}
}
}
```
Requires Node.js. Alternative on paid plans: Settings → Connectors → Add custom connector with request header `authorization` = `Bearer …` (where available; the server must be publicly reachable).
### Other MCP clients
Any MCP client that supports Streamable HTTP can connect directly with the URL and the Authorization header. For stdio-only clients, use the mcp-remote bridge:
```bash
npx -y mcp-remote https://agentscraper.cc/mcp \
--header "Authorization: Bearer YOUR_KEY" --transport http-only
```
Server URL: `https://agentscraper.cc/mcp` · Transport: Streamable HTTP · Auth header: `Authorization: Bearer `
### REST API
Prefer plain HTTP? The same API key works with the REST endpoint.
```bash
curl -X POST https://livewebdata.com/api/v1/contents \
-H "Authorization: Bearer YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/article"]}'
```
## get_page_contents
Fetch a public web page live and return its main content as clean Markdown, plain text or HTML. Read-only, idempotent. Single URL per call.
| Parameter | Type | Default | Description |
| --- | --- | --- | --- |
| `url` | string (http/https, at most 8,192 characters) | required | The public URL to fetch. |
| `country` | ISO-3166 alpha-2 | `"us"` | Fetch the page as seen from this country, e.g. 'de', 'gb', 'us'. |
| `format` | `"markdown"` \| `"text"` \| `"html"` | `"markdown"` | Output format. Prefer markdown. The page HTML is fetched and returned only for `"html"`. |
| `max_characters` | integer 500–100000 | `8000` | Maximum characters of content to return. Smaller than the REST API's default (`text.maxCharacters`, 100,000) because the result goes into the agent's context; raise it when the content is cut. |
| `verbosity` | `"compact"` \| `"full"` | `"compact"` | compact = main content only, full = whole page body. |
The result is one text item with a short header, followed by the content. The full REST-style JSON is also returned as `structuredContent` (with `results[].html` only for `format: "html"`).
```markdown title="Tool result"
# Example article
Source: https://example.com/article · country: us · live · credits used: 1 · credits left: 8,411
# Example article
Main content as clean Markdown…
```
## get_account_status
Check the remaining credits, plan and rate limit for the connected account. Agents should call it before large batches. No parameters, free. Returns the same numbers as `GET /v1/account`.
```text title="Tool result"
Plan: Starter · Credits left: 8,412 · Rate limit: 2 URL/s (burst 4) · Status: active
```
## Errors
Errors are returned with `isError: true` and one actionable sentence the agent can follow. When nothing was charged, the message ends with “No credits were charged.”
| Case | Message to the agent |
| --- | --- |
| 451 restricted target | This website is restricted and cannot be retrieved. Do not retry; use another source. |
| `CRAWL_NOT_RENDERED` | The page has no usable text: it needs JavaScript rendering, or shows no text even when rendered. Try another source or a different URL for the same information. |
| `CRAWL_NOT_FOUND` | Page not found (404/410). |
| `SOURCE_NOT_AVAILABLE` | The website refused this request. Retry once with a different country, otherwise use another source. |
| `SOURCE_BLOCKED` | Access denied by the website. Retry later or with a different country, otherwise use another source. |
| `CRAWL_LIVECRAWL_TIMEOUT` | The page timed out. Retry once with a longer timeout, otherwise use another source. |
| `BROWSER_UNAVAILABLE` | Browser rendering is temporarily unavailable. Retry in a few minutes, otherwise use another source. |
| `RATE_LIMITED` | The rendering limit for the user's plan is reached. Do not retry rendering now; use another source. |
| 402 insufficient credits | The account is out of credits. Tell the user to top up at …/dashboard/billing. |
| 429 rate limited | Rate limit reached. Wait N seconds before retrying. |
| 401 / 403 | The API key is invalid or suspended. Tell the user to check their key at …/dashboard/keys. |
| Still fetching after 5 minutes | The page did not finish loading within 5 minutes, and the fetch may still be charged. Use another source. |
## Credits
MCP calls are billed exactly like REST calls: 1 credit per successful page, failed requests (including access denied by the site) are free. They appear in Usage with channel `mcp`. See [Credits](https://agentscraper.cc/docs/api-reference#credits).
---
# No-code recipes
Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.
## Before you start
You need an API key from [API keys](https://agentscraper.cc/dashboard/keys). Every recipe sends the same request:
| Setting | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Header | `Authorization: Bearer YOUR_KEY` |
| Body (JSON) | `{"urls": ["https://example.com/article"], "countryCode": "us"}` |
The response is JSON: page content in `results[0].markdown`, the outcome in `statuses[0].status` and `statuses[0].error.tag`.
## n8n
> **Native integration coming soon.** Use the built-in **HTTP Request** node for now.
_Screenshot coming soon: n8n HTTP Request node_
| Field (HTTP Request node) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Authentication | Generic Credential Type → **Bearer Auth** (token = your key) — or **Header Auth** with Name `Authorization`, Value `Bearer YOUR_KEY` |
| Send Body | on |
| Body Content Type | `JSON` |
| Specify Body | `Using JSON` → `{"urls": ["{{ $json.url }}"]}` |
| Options → Response → Response Format | `JSON` (default: autodetect) |
**URL lists:** feed one item per URL into the node. Use the node's **Options → Batching** (*Items per Batch* = your rate limit, *Batch Interval (ms)* = 1000) — or a **Loop Over Items** node plus a **Wait** node (1 second) to stay within your rate limit.
## Make
> **Native integration coming soon.** Use the **HTTP** app's **Make a request** module for now.
_Screenshot coming soon: Make HTTP "Make a request" module_
| Field (HTTP → Make a request) | Value |
| --- | --- |
| Authentication type | `API key` (stores the key in a keychain) — or `No authentication` plus the header below |
| URL | `https://livewebdata.com/api/v1/contents` |
| Method | `POST` |
| Headers | `Authorization` = `Bearer YOUR_KEY` |
| Body content type | `application/JSON` |
| Body input method | `JSON string` |
| Body content | `{"urls": ["https://example.com/article"]}` |
| Parse response | `Yes` |
**URL lists:** use an **Iterator** over your URLs before the HTTP module and set the scenario's scheduling so the number of operations per second stays within your rate limit.
## Zapier
> **Native integration coming soon.** Use **Webhooks by Zapier** → **Custom Request** for now (not available on the Free plan).
_Screenshot coming soon: Zapier Webhooks by Zapier – Custom Request_
| Field (Custom Request) | Value |
| --- | --- |
| Method | `POST` |
| URL | `https://livewebdata.com/api/v1/contents` |
| Data Pass-Through? | `False` |
| Data | `{"urls": ["https://example.com/article"]}` |
| Headers | `Authorization` = `Bearer YOUR_KEY`, `Content-Type` = `application/json` |
The JSON response is available as fields in later steps.
## Tips
- Failed pages return HTTP 200 with `statuses[0].status = "error"` — add a filter/router on that field.
- Keep the key in the tool's credential store, not in plain step fields, wherever possible.
---