Scan a site's URLs

POSThttps://api.crawl4ai.com/scan

Every URL of one site, without crawling it. /scan asks the sources that already know the site: its sitemaps (the ones robots.txt names, else /sitemap.xml and /sitemap_index.xml; a sitemap index is followed), the links on its home page, and the two newest Common Crawl indexes. mode: "full" also reads the Wayback Machine's index. It answers in one call, in 1 to 3 seconds for most sites. Use it to plan a crawl: pick the URLs, then send them to /scrape/jobs, or crawl the site with /site.

1 to 3 s most sites1 unit for the call, plus each page it reads50000 URLs at most7 days a repeat comes from the cache

curl https://api.crawl4ai.com/scan \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "domain": "docs.crawl4ai.com", "max_urls": 500, "include": "/core" }'

Body

`domain`required string	The site's host, e.g. `docs.crawl4ai.com`. A leading `https://` is dropped. `www.` and the bare host count as one site.
`mode` string	Default `fast` `fast` (default) · `full` — `full` adds the Wayback Machine's index, up to 20 seconds more.
`max_urls` int	The most rows returned. Default 5000, max 50000; a larger value is cut to 50000.
`include` string	Keep only matching URLs: a path prefix (`/docs`), or a glob with `` over the path (`/blog/`, `.html`; over the whole URL when it starts with `http`). Default: every URL.
`include_subdomains` bool	Also list URLs on subdomains of `domain`. Default false.
`bypass_cache` bool	Scan again even when a scan with the same inputs is in the cache (7 days).

Response

{
  "recipe": "site-scan", "version": 1,
  "rows": [
    { "url": "https://docs.crawl4ai.com/", "sources": ["sitemap", "links", "commoncrawl"], "last_seen": "2026-06-18", "depth": 0 },
    { "url": "https://docs.crawl4ai.com/core/installation", "sources": ["sitemap", "links"], "last_seen": "2026-06-18", "depth": 2 }
  ],
  "usage": { "units": 8, "boxes": ["crawl"], "lines": [ { "call": "scrape", "url": "https://docs.crawl4ai.com/robots.txt", "units": 1 }, ... ] },
  "meta": { "pages_fetched": 7, "elapsed_ms": 1709, "from_cache": false, "warnings": ["module: site-scan: commoncrawl 37 · links 61 · sitemap 87 · union 92"] }
}

Field	Description
rows[].url	One page of the site, in one normal form: https, no fragment, no tracking keys (`utm_*`, `fbclid`, …), a trailing slash dropped except on the home page. Files that are not pages (images, scripts, sitemaps) are left out. URLs that more sources know come first, then by URL.
rows[].sources	The sources that knew the URL: `sitemap`, `links` (the home page), `commoncrawl`, `wayback`.
rows[].last_seen	The sitemap's `lastmod` as the site wrote it, else the index's capture time (`YYYYMMDDhhmmss`), else `null`.
rows[].depth	The number of path segments: `/` is 0, `/docs/api` is 2.
usage	What the scan cost: `units`, and one line in `lines` per page it read.
meta	`pages_fetched`, `elapsed_ms`, `from_cache`, and `warnings`: one line with the count from each source, and a line when a source did not answer or the list was cut at `max_urls`.

Credits and errors

A scan costs 1 unit for the call, plus each page it reads (robots.txt, the home page, each sitemap, each index reply), priced as a /scrape of that page. Most scans read 4 to 26 pages; mode: "full" reads one more. usage.lines lists every page. A repeat with the same inputs within 7 days comes from the cache: usage.cache is "hit" and it costs the call's 1 unit only. No flat recipe price applies to a scan.

Status	error	Meaning
400	bad_input	The body is not a JSON object, or an input has the wrong type.
402	no_credit	Your balance is empty. Start a plan, or buy extra credits on your plan, and send it again.
404	not_available	The service is not open to your account yet.