Responsible web scraping: robots.txt, terms, personal data

This guide is not legal advice. It explains the questions an automation builder should answer before scheduling a job that collects web pages. For your own project, ask a lawyer in your jurisdiction.

Many "is web scraping legal" articles come from scraping vendors, who have a reason to answer "yes, mostly". The honest answer is "it depends", and the useful part is knowing what it depends on. Responsible web scraping is less about one rule and more about a short list of checks, done before the job runs every morning.

Check robots.txt for every host

The first item on the pre-launch checklist: fetch https://<host>/robots.txt for every host on your list and compare its rules with the paths you plan to collect. The format is standardized in RFC 9309, the Robots Exclusion Protocol. A robots.txt check step shows how to run this check inside an n8n or cron job before every run.

Why the check matters:

• It states the site operator's crawling preferences, per user agent and path.
• It gives you a cheap, machine-readable first signal of whether automated access is welcome.

What robots.txt does not do:

• It is not an access control mechanism. RFC 9309 says so directly, and a December 2025 ruling in Ziff Davis v. OpenAI (S.D.N.Y.) reportedly held that robots.txt does not "effectively control" access for the purposes of the DMCA anti-circumvention rule, according to the law blog listed in the sources.
• It is not a permission slip. A path that robots.txt allows can still be covered by terms of service, copyright or data protection law.
• Ignoring it is not free of consequences. It can still be raised against you in a dispute.

Practical rule: read robots.txt for every host on your list, honor Disallow lines that match your paths, and treat a blanket Disallow: / as a strong signal to find another source or ask the operator.

Terms of service

A site's terms may restrict automated access, limit reuse of content, or forbid commercial use. Whether those terms bind you depends on how they were presented and accepted, and on your jurisdiction. Breach of terms is generally a contract question rather than a criminal one, but it can still end in a dispute you do not want.

What to do in practice:

• Read the terms of every site you schedule, and note the date you checked.
• Look specifically for clauses about automated access, crawling, data mining and reuse.
• Prefer official channels when they exist: a public API, an RSS feed, a data export or a partnership page.
• If the terms forbid what you plan, do not schedule it.

Personal data

Pages that mention identifiable people are where most real risk sits. In the EU and UK, the GDPR applies to personal data even when it is publicly visible: you need a lawful basis (Article 6), a clear purpose and data minimization. US state privacy laws, such as California's, add their own rules.

For collection workflows this means:

• Do not build lists of people. Profiles, staff pages, author pages and contact details are the wrong input for a scheduled job unless you have a clear legal basis.
• Minimize at the source. If a page mixes useful content with names or emails, strip them before storage, or do not store the raw text at all.
• Set retention. Keep raw pages only as long as the workflow needs them.

Polite collection

Even on public pages that you are allowed to read, how you read them matters:

• Keep the request rate low per host. A daily job that needs 200 pages from one site does not need to fetch them in the same second.
• Keep the fetch time per URL and skip pages fetched within their refresh window, so repeat runs do not fetch unchanged pages again.
• Run only as often as the use case needs. Daily is enough for most public changelogs and docs.
• Stop retrying a host that refuses access. Repeated SOURCE_NOT_AVAILABLE errors are a message, not an obstacle.
• Have an owner and a stop switch. If an operator contacts you, you should be able to pause the job the same day.

A provider network can mean fewer blocks on public pages, but that is not a reason to treat a site as an opponent. If a site does not want to be collected, the right move is a different source.

What the AUP prohibits

Live Web Data API's acceptable-use policy (Section 8 of the Terms) applies to every request, from any of these workflows. Among other things, it prohibits using the service or its output:

• to access content that is not publicly accessible, including pages behind a login, password, paywall, subscription or private API;
• to create or log in to accounts on third-party services, or to submit forms, purchases or bookings;
• to put an unreasonable or disproportionate load on any website;
• to surveil, track, profile or build dossiers on identifiable individuals, or to collect sensitive personal data;
• to harvest contact details for unsolicited messages;
• for automated purchasing of tickets or limited goods, or for manipulating reviews, rankings or engagement.

The policy also makes the customer responsible for checking that their use is lawful and for complying with the terms of the websites they access, to the extent those terms bind them. The provider may honor robots.txt, opt-out and takedown requests and may restrict any site. Read the full text on the shared acceptable use page. Sign-up is for businesses only. Only Business, Enterprise and Custom sign-ups are reviewed before checkout, usually within 1 business day. The review checks the intended use.

Public price pages and competitor monitoring

Monitoring public price pages is a common request. Frame it narrowly: public pages, a modest schedule, robots.txt and terms checked per site, no logged-in or personalized prices, and no personal data. Do not name or target specific brands in shared workflows, and do not use the output for anything on the prohibited list above. If a site's terms forbid automated monitoring, use a licensed data source instead.

Not legal advice

Everything above is a description of the landscape and of one provider's policy, written for people who build workflows. Laws differ by country and change, and the answer for your project depends on facts this guide cannot see. Before you schedule a job that collects other people's pages at scale, ask a lawyer.

When the checks pass, the operational guides take over: scrape a list of URLs for the batch itself and scheduled web scraping for the recurring run.