The AI step
Classify scraped pages in n8n without losing rows to Discard Item
Label collected pages with n8n's Text Classifier node so every URL still gets a sheet row: route failed fetches first, use the Other branch, catch parse errors.
By Caitlin Sanders, AgentScraper team · Published
This article was drafted with AI assistance from the sources listed below and edited and published by the AgentScraper team. Disclosure: the AgentScraper API, which this article mentions, is this site's own product.
- Route on fetch status first: only pages with Markdown reach the classifier
- When No Clear Match: Output on Extra, 'Other' Branch
- Allow Multiple Classes To Be True: off for one label per row
- On Error: Continue, then check error on the first output
- One label node per output, then Merge in Append mode
Picture a run like this one (the numbers are an example, not data): you collect 200 public pages, send the Markdown to n8n's Text Classifier, and write the label to a sheet. The sheet has 171 rows. Nothing failed, the execution is green, and 29 URLs are simply gone.
That is not a model problem. It is a routing problem, and the node's defaults cause most of it. This walkthrough wires the classifier so every URL leaves the run with a row: a label, other, or a status that says why there is no label.
Where rows disappear
Four behaviors cost you rows. The first is in the docs, the rest are in the node source.
| Cause | What happens | Fix |
|---|---|---|
| When No Clear Match = "Discard Item" (default) | An item that fits no category is dropped and goes to no output | Set "Output on Extra, 'Other' Branch" |
| Empty text to classify | The node throws "Text to classify for item N is not defined" | Send only successful fetches with Markdown |
| Model answer cannot be parsed | With continue-on-fail, the item goes to the first category's output with an error field | Check error on that branch |
| Allow Multiple Classes To Be True on | One item is copied to every category it matched | Leave it off for one label per row |
The first row explains the 29 missing URLs. The second explains the run that stops at the first 404 page. The third is the quiet one: a page that broke the parser looks like a page of your first category.
Build the classifier branch
The fetch part is the same as in n8n scraping to Google Sheets: one URL per item, HTTP Request with Options → Batching. The node settings for AgentScraper API are in the no-code recipes.
Step 1. After the fetch, add an Edit Fields (Set) node with five fields:
| Field | Value |
|---|---|
url | the URL from your list item |
status | {{ $json.statuses[0].status }} |
error_tag | {{ $json.statuses[0].error ? $json.statuses[0].error.tag : '' }} |
fetched_at | {{ $json.statuses[0].fetchedAt }} |
markdown | {{ $json.results[0].markdown }} |
What you see: one flat item per URL, success or not.
Step 2. Add an IF node. Condition one: status equals success. Condition two: markdown is not empty. Combine them with AND.
What you see: pages with text on the true branch, everything else on the false branch.
The second condition matters. The node source reads the text with no fallback and throws when it is empty. With On Error set to "Stop Workflow", one empty page halts the whole workflow.
Step 3. On the false branch, add an Edit Fields (Set) node named "Label: no page". Set label to fetch_failed and keep the other fields.
What you see: failed URLs carry status, error_tag and a label. Failed fetches cost 0 credits, and this row is how you find them.
Step 4. On the true branch, add a Text Classifier node. Attach a chat model sub-node. Set Input Prompt to:
{{ $json.markdown.slice(0, 8000) }}
What you see: the node asks for at least one category.
The slice keeps a very long page from filling the model's context. Pick a cap that fits your model.
Step 5. Add your categories. Give each a name and a description. The description tells the model what the name means:
| Category | Description |
|---|---|
changelog | Release notes or a list of dated product changes |
pricing | A page that lists plans, prices or billing terms |
docs | Reference or how-to documentation for a product |
What you see: one output per category, labeled with its name.
Step 6. Open Options. Set When No Clear Match to "Output on Extra, 'Other' Branch". Leave Allow Multiple Classes To Be True off. What you see: a fourth output named Other, always the last one.
The default prompt in the source asks the model to classify the text into one of the categories and to answer only with JSON. With the Other branch on, the model may also pick none, and that item leaves on Other.
Step 7. Open Settings. Set On Error to "Continue". n8n's docs describe it as going on to the next node despite the error. What you see: a page that breaks the parser no longer stops the run.
Enable Auto-Fixing is on by default in the source. It sends a broken answer back to the model with the parse error, so a bad answer can cost a second model call. If that also fails, the item still arrives, on the first category's output, with an error field.
Label every branch
The classifier routes items. It does not write the category into the item. Each output needs its own label node.
Step 8. On each category output, add an Edit Fields (Set) node that sets label to the category name. On the first output only, use an expression instead:
{{ $json.error ? 'classify_failed' : 'changelog' }}
What you see: parse failures labeled classify_failed, not changelog.
Step 9. On the Other output, set label to other.
Step 10. Add a Merge node. Set Mode to "Append" and Number of Inputs to the number of label nodes, here 5: three categories, Other and "Label: no page". More than two inputs need n8n 1.49.0 or later. Connect each label node to its own input. What you see: one list of items, one per URL.
The Merge docs say Append waits for all connected inputs. Test once with a list where one category gets no pages, and check that the sheet still gets every row.
Step 11. Drop the markdown field, then write once with Append or Update Row, matched on url.
Status and label table
status | label | Meaning | Next run |
|---|---|---|---|
success | a category | Classified | Skip until the refresh window passes |
success | other | Fetched, fits no category | Review the categories, not the page |
success | classify_failed | Model answer unusable | Classify again; the page is already fetched |
success | fetch_failed | Fetched, but no text | Check the page; a model cannot label it |
error | fetch_failed | Fetch failed, see error_tag | Follow the retry rules |
A rising other count means your categories no longer match the pages. A rising classify_failed count points to the model or the prompt. Neither is visible if the rows are discarded.
The fetch bills per page and the model bills per call, on separate invoices. The AI scraper workflow guide covers keeping the two apart.
What this means for your workflow
- Route on
statuses[0].statusand on empty Markdown before the classifier. Empty text throws. - Set When No Clear Match to "Output on Extra, 'Other' Branch". "Discard Item" drops rows silently.
- Set On Error to "Continue" and check
erroron the first category's output. - Keep Allow Multiple Classes To Be True off when one row holds one label.
- Set a
labelper output, merge with Append, write once. - Count rows: the sheet should hold exactly as many rows as the list has URLs.
Frequently asked questions
Why does the n8n Text Classifier output fewer items than it receives?
The When No Clear Match option defaults to Discard Item. An item that fits none of your categories is dropped and leaves on no output. Set the option to Output on Extra, 'Other' Branch, and connect the new Other output.
Why does the Text Classifier fail on some scraped pages?
The node throws an error when the text to classify is empty. A failed fetch has no Markdown, so it stops the node. Route on the fetch status with an IF node first and send only successful pages with text to the classifier.
Does the Text Classifier add the chosen category as a field?
Not by default. It routes each item to the output named after the category and keeps the item's fields. Add a small Edit Fields node on each output that sets a label field, so the sheet can show the result.
What happens when the model's answer cannot be parsed?
Enable Auto-Fixing is on by default in the node source and sends the parse error back to the model, which can cost an extra model call. If it still fails and the node is set to continue on error, the source puts the item with an error field on the first category's output.
Sources
- Text Classifier node (n8n docs) (Documentation)
- Text Classifier node (n8n source) (Documentation)
- Text Classifier item processing (n8n source) (Documentation)
- Node settings (n8n docs) (Documentation)
- Merge node (n8n docs) (Documentation)