The AI scraper workflow: fetch first, then ask your model

Search for "AI scraper" and most results sell one box: give it a URL and a prompt, get perfect data back. Inside that box are two different jobs. Collection gets the readable part of a public page, reliably, for a list of URLs. Interpretation asks a model what the page means. This guide pulls them apart, because a workflow that keeps them separate is cheaper to run, easier to debug and lets you change the model without rebuilding anything. It is the model half of an agent scraper pipeline; the fetch half comes first.

Two jobs, not one

When both happen in one step, a wrong value could come from either side, and the bill is one number. When they are separate, each step has its own status, its own log and its own cost line.

In practice the fetch step is an HTTP call that returns Markdown and a status, as in the n8n and batch guides. Live Web Data API does exactly that and nothing more: it reads each URL (in a browser if you set render), but it does not crawl for new URLs or extract fields. The model step is yours.

Extraction from Markdown

Markdown is a good input for a model: headings, lists, tables and links survive, while navigation, scripts and cookie banners are gone. To get fields out of it reliably:

1. Define a schema. List the fields, their types and which ones may be null. Most model APIs support structured outputs that force the response into a JSON schema; use that instead of asking politely for JSON.
2. Allow "not found". Tell the model to return null when a field is not on the page. Without that option, models fill gaps with plausible guesses.
3. Ask for evidence. For important fields, add a short quote field with the sentence the value came from. It makes spot checks fast.
4. Validate in code. Check types, ranges and formats after the model answers. Reject and log, do not repair silently.

{
  "type": "object",
  "properties": {
    "product_name": { "type": ["string", "null"] },
    "release_date": { "type": ["string", "null"], "description": "ISO date if stated on the page" },
    "breaking_changes": { "type": "array", "items": { "type": "string" } },
    "quote": { "type": ["string", "null"] }
  },
  "required": ["product_name", "release_date", "breaking_changes", "quote"],
  "additionalProperties": false
}

This example reads a public changelog page. The same pattern works for public documentation, job descriptions on a company's own careers page or public event listings, always within the site's terms and without personal data you have no basis to process.

A worked example

The fetch step returns this Markdown for https://example.com/changelog (capped with "text": {"maxCharacters": 8000}):

# Changelog

## 2.4.0 (2026-10-14)

- Added: webhook retries are now configurable.
- Changed: the `limit` parameter now defaults to 50.
- Removed: the deprecated `/v1/export` endpoint.

The prompt sent to your model, with the schema above passed as the structured-output format:

Extract the latest release from this changelog.
Return null for any field the page does not state.
breaking_changes: only changes that can break existing integrations, copied as written.
quote: the heading line of the release you used.

<page url="https://example.com/changelog">
{markdown}
</page>

A valid answer, which your code then checks (the date parses as ISO, every breaking change appears in the page text):

{
  "product_name": null,
  "release_date": "2026-10-14",
  "breaking_changes": [
    "Changed: the `limit` parameter now defaults to 50.",
    "Removed: the deprecated `/v1/export` endpoint."
  ],
  "quote": "## 2.4.0 (2026-10-14)"
}

product_name is null because the page never names the product. That is the behavior you want: a guess here would look correct and be wrong.

Classification and filtering

Often the cheapest useful AI step is not extraction at all but a yes/no or a label:

• "Is this page about topic X? Answer relevant or not_relevant."
• "Which of these five categories fits best?"
• "Did anything in this changelog affect authentication?"

Classification needs far less text than extraction. Send the title and the first 2,000 to 4,000 characters, not the whole page. Run it before more expensive steps, so only relevant pages reach the extraction prompt or your storage.

Cap page length before the model

Every character you send is paid for, and research on long contexts (Liu et al., 2023) found that models use information buried in the middle of a long input less reliably. Cap length in two places:

• At the fetch: "text": {"maxCharacters": 8000} truncates the Markdown before it leaves the API. The default is 100,000, which keeps whole pages, so set a lower cap yourself.
• In your code: for classification, cut to the first sections; for extraction, cut to the sections whose headings match what you need.

Log the original length next to the capped length, so you notice when a page grew and the field you need moved past the cut.

Fixed lists vs agent navigation

An agentic scraper lets a model decide which page to open next, click, and read again. That is useful when you do not know the pages in advance: research across unfamiliar sites, or following links to find the one page that answers a question.

It is a poor fit for a fixed list collected on a schedule. Each run repeats the same navigation decisions, pays model tokens for every step, takes longer and fails in ways that are buried in a trace. A plain loop over known URLs, followed by one model call per page, does the same work with a predictable cost. If your use case really is an agent fetching pages inside its own reasoning loop, the design questions are different (tool design, in-loop errors, prompt injection); agentfetch.live covers that side, starting with when an agent needs a browser and when a fetch is enough.

Costs: fetch vs model

Keep two cost lines and look at them separately:

• Fetch: credits per successful page. Failed, blocked or restricted pages cost 0 credits. Every fetch is live, so skip URLs your own store fetched within their refresh window; see pricing for how credits are counted.
• Model: input tokens (driven by your length cap) plus output tokens (driven by your schema). Depending on page length and your model, this can be the larger line, which is why the cap matters.

Store both per page in your output (credits used, input tokens, output tokens). When the bill moves, you can tell which half moved.

What this means for your workflow

• Fetch first, as its own step with its own status. Interpret second, with your own model.
• Use structured outputs with a JSON schema, allow null, and validate in code.
• Classify before you extract; send the model only what the task needs.
• Use an agent when the pages are unknown, a loop when the list is fixed.
• Track fetch credits and model tokens separately, per page.

Before you point the workflow at someone else's site, read the responsible web collection guide.