# AgentScraper

> Build an agent scraper pipeline for public pages: a URL list in n8n, Make, Zapier or cron, clean Markdown out, every failure visible, your AI step after.

Scheduled web collection for AI workflows, built to be boring. A Teneo Labs project · Built on Live Web Data API.

## Guides

- [n8n web scraping: a reliable URL-list workflow](https://agentscraper.cc/guides/n8n-web-scraping): n8n web scraping for public pages: HTTP Request node, batching, execution limits, error routing on per-URL status, Google Sheets output and an AI step.
- [Zapier and Make web scraping: HTTP steps and iterators](https://agentscraper.cc/guides/make-and-zapier-web-scraping): Zapier and Make.com web scraping for public pages: the generic HTTP step, looping over a URL list, rate limits, failure routing and a comparison with n8n.
- [Scrape a list of URLs: CSV in, JSONL out, safe retries](https://agentscraper.cc/guides/scrape-list-of-urls): Scrape a list of URLs as a batch job: normalize and dedupe the list, size the run, pace requests, retry by error type, write JSONL and resume after a crash.
- [Scheduled web scraping with cron and GitHub Actions](https://agentscraper.cc/guides/scheduled-web-scraping): Scheduled web scraping that stays honest: choosing a scheduler, cron and GitHub Actions setups, refresh windows, change alerts and checks for silent failures.
- [The AI scraper workflow: fetch first, then ask your model](https://agentscraper.cc/guides/ai-scraper-workflow): An AI scraper workflow in two steps: collect clean Markdown first, then extract, classify or summarize with your own LLM, cap the input, keep the costs apart.
- [Responsible web scraping: robots.txt, terms, personal data](https://agentscraper.cc/guides/responsible-web-collection): Responsible web scraping before a job is scheduled: robots.txt, terms of service, personal data, polite request rates and the usage policy. Not legal advice.

## Posts

- [Scrape multiple URLs in n8n: Batching or Loop Over Items](https://agentscraper.cc/blog/n8n-scrape-multiple-urls): Two ways to pace a URL list in n8n so a fetch API never answers 429: the HTTP Request node's Batching option, or Loop Over Items with a Wait node.
- [Add a robots.txt check step to your n8n or cron job](https://agentscraper.cc/blog/robots-txt-check-step): A per-host robots.txt check that runs before every scheduled collection job: a Python step for cron and a Code node for n8n, with the RFC 9309 edge cases.

## Built on Live Web Data API

> Live web pages as clean Markdown, text or HTML for AI agents. Send up to 10 URLs per request to POST /v1/contents; failed pages cost nothing.

- API base URL: https://livewebdata.com/api
- Auth: `Authorization: Bearer <api_key>` (keys start with 1337_live_)
- OpenAPI: https://agentscraper.cc/openapi.json
- MCP server (Streamable HTTP, same bearer key): https://agentscraper.cc/mcp

### Docs

- [Overview](https://agentscraper.cc/docs.md): Live Web Data API turns any public URL into clean Markdown, text or HTML — fetched live through our fetch network.
- [API reference](https://agentscraper.cc/docs/api-reference.md): Two endpoints. HTTP 200 is returned even when individual pages failed — check statuses[] for the per-URL outcome.
- [Examples](https://agentscraper.cc/docs/examples.md): Copy-paste examples in cURL, Python and JavaScript. Replace YOUR_API_KEY with a key from the dashboard.
- [Automate requests](https://agentscraper.cc/docs/automation.md): Move from clicking Run in the playground to fetching URL lists from your own code — with rate limits, retries and scheduling handled.
- [SDKs](https://agentscraper.cc/docs/sdks.md): Thin, typed clients for JavaScript/TypeScript and Python — built-in retries, typed errors, one-line page fetches.
- [MCP for agents](https://agentscraper.cc/docs/mcp.md): Use the Live Web Data API as a tool in Claude, Cursor, VS Code and any other MCP client — with one URL and your existing API key.
- [No-code recipes](https://agentscraper.cc/docs/no-code.md): Call the API from n8n, Make or Zapier with each tool's generic HTTP step. Native integrations are on the roadmap.

### Optional

- [Full docs in one file](https://agentscraper.cc/llms-full.txt)
- [Postman collection](https://agentscraper.cc/live-web-data-api.postman_collection.json)
- [Example scripts (.zip)](https://agentscraper.cc/live-web-data-api-examples.zip)