Agents
By job

Uncensored AI for web scraping code

The best uncensored AI for web scraping code is DeepSeek V4.1 Flash 1017 in a coding agent, because scrapers are long, repetitive code that a low-cost model with a 1M-token context handles well. Keep the scraper polite by respecting robots.txt, rate limits and the site's terms.

Updated

Work that hosted assistants often decline

  • A scraper for public listings or prices
  • A crawler that archives your own docs
  • Tables in public PDFs turned into CSV

All of it is legitimate on systems you own or may work on, as why coding agents refuse explains.

Which model

ModelUse it forPrice per 1M, in / out
DeepSeek V4.1 FlashScrapers and parsers, day to day$0.24 / $0.90
GLM-5.3 FlashMany parsers for many sites$0.24 / $1.00
Kimi K3.1 ProMessy sites, read from screenshots as well as HTML$3.00 / $8.00

Silk Compute prices, read November 18, 2026.

Workflow

  1. Check the rules first

    Read the site's robots.txt (RFC 9309) and terms, and look for an official API first.

  2. Give the agent real samples

    Save a few pages as HTML fixtures in the project.

  3. Parser first, with tests

    Ask for the parser and its tests before any network code.

  4. Then fetch politely

    Add rate limiting, caching, backoff and a User-Agent that identifies you, and run small first.

Notes

This page covers scrapers for public data and sites you run, not getting past logins, CAPTCHAs or anti-bot systems.

A prompt to start from

Prompt
fixtures/ has five saved listing pages from a public directory. Write a parser that extracts name, address and category into a dataclass, with pytest tests on the fixtures. Then write a fetcher that reads robots.txt, waits two seconds between requests, caches responses on disk and sets a User-Agent with my contact address.

Paste it into Claude Code, Codex or OpenCode once it runs on an abliterated model.