# Scan a site's URLs

`POST https://api.crawl4ai.com/scan`

Every URL of one site, without crawling it. `/scan` asks the sources that already know the site: its sitemaps (the ones `robots.txt` names, else `/sitemap.xml` and `/sitemap_index.xml`; a sitemap index is followed), the links on its home page, and the two newest Common Crawl indexes. `mode: "full"` also reads the Wayback Machine's index. It answers in one call, in 1 to 3 seconds for most sites. Use it to plan a crawl: pick the URLs, then send them to [/scrape/jobs](https://crawl4ai.com/docs/scrape-jobs/), or crawl the site with [/site](https://crawl4ai.com/docs/site/).


curl:

```bash
curl https://api.crawl4ai.com/scan \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "domain": "docs.crawl4ai.com", "max_urls": 500, "include": "/core" }'
```

## Body

| Name | Type | Default | Description |
|---|---|---|---|
| domain* | string | - | The site's host, e.g. `docs.crawl4ai.com`. A leading `https://` is dropped. `www.` and the bare host count as one site. |
| mode | string | fast | `fast` (default) · `full` — `full` adds the Wayback Machine's index, up to 20 seconds more. |
| max_urls | int | - | The most rows returned. Default 5000, max 50000; a larger value is cut to 50000. |
| include | string | - | Keep only matching URLs: a path prefix (`/docs`), or a glob with `*` over the path (`*/blog/*`, `*.html`; over the whole URL when it starts with `http`). Default: every URL. |
| include_subdomains | bool | - | Also list URLs on subdomains of `domain`. Default false. |
| bypass_cache | bool | - | Scan again even when a scan with the same inputs is in the cache (7 days). |

## Response

json · shortened:

```json
{
  "recipe": "site-scan", "version": 1,
  "rows": [
    { "url": "https://docs.crawl4ai.com/", "sources": ["sitemap", "links", "commoncrawl"], "last_seen": "2026-06-18", "depth": 0 },
    { "url": "https://docs.crawl4ai.com/core/installation", "sources": ["sitemap", "links"], "last_seen": "2026-06-18", "depth": 2 }
  ],
  "usage": { "units": 8, "boxes": ["crawl"], "lines": [ { "call": "scrape", "url": "https://docs.crawl4ai.com/robots.txt", "units": 1 }, ... ] },
  "meta": { "pages_fetched": 7, "elapsed_ms": 1709, "from_cache": false, "warnings": ["module: site-scan: commoncrawl 37 · links 61 · sitemap 87 · union 92"] }
}
```

| Field | Description |
|---|---|
| rows[].url | One page of the site, in one normal form: https, no fragment, no tracking keys (`utm_*`, `fbclid`, …), a trailing slash dropped except on the home page. Files that are not pages (images, scripts, sitemaps) are left out. URLs that more sources know come first, then by URL. |
| rows[].sources | The sources that knew the URL: `sitemap`, `links` (the home page), `commoncrawl`, `wayback`. |
| rows[].last_seen | The sitemap's `lastmod` as the site wrote it, else the index's capture time (`YYYYMMDDhhmmss`), else `null`. |
| rows[].depth | The number of path segments: `/` is 0, `/docs/api` is 2. |
| usage | What the scan cost: `units`, and one line in `lines` per page it read. |
| meta | `pages_fetched`, `elapsed_ms`, `from_cache`, and `warnings`: one line with the count from each source, and a line when a source did not answer or the list was cut at `max_urls`. |

## Credits and errors

A scan costs 1 unit for the call, plus each page it reads (`robots.txt`, the home page, each sitemap, each index reply), priced as a [/scrape](https://crawl4ai.com/docs/scrape/) of that page. Most scans read 4 to 26 pages; `mode: "full"` reads one more. `usage.lines` lists every page. A repeat with the same inputs within 7 days comes from the cache: `usage.cache` is `"hit"` and it costs the call's 1 unit only. No flat recipe price applies to a scan.

| Status | error | Meaning |
|---|---|---|
| 400 | bad_input | The body is not a JSON object, or an input has the wrong type. |
| 402 | no_credit | Your balance is empty. Start a plan, or buy extra credits on your plan, and send it again. |
| 404 | not_available | The service is not open to your account yet. |

> **Note**
>
> A `domain` that is not a host returns empty `rows` and a line in `meta.warnings`.



**Try it with your key**

Run this call in the playground and see the real answer, the time and the units.
[Open the playground](https://crawl4ai.com/#playground)



---
Source: https://crawl4ai.com/docs/scan/