# Crawl a whole site

`POST https://api.crawl4ai.com/site`

Every page under a URL, as Markdown, level by level: the start page (level 0), the pages it links to (level 1), and so on to `depth`. It follows only links inside the site, through your `include` and `exclude` filters, and stops at `max_pages`. A site crawl runs only as a job: submit it, follow it, then read the rows. A job runs for at most one hour. To list a site's URLs without fetching them, use [/scan](https://crawl4ai.com/docs/scan/).


curl:

```bash
# 1. submit -> job id
JOB=$(curl -s https://api.crawl4ai.com/site \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.crawl4ai.com/", "depth": 2, "max_pages": 50, "include": "/core" }' | jq -r .job_id)

# 2. poll until it ends (done, or paused with a reason)
while true; do
  S=$(curl -s https://api.crawl4ai.com/site/jobs/$JOB -H "Authorization: Bearer sk_live_..." | jq -r .status)
  case "$S" in done|paused) break;; esac; sleep 5
done

# 3. read the rows (NDJSON, one line per page)
curl -s https://api.crawl4ai.com/site/jobs/$JOB/results \
  -H "Authorization: Bearer sk_live_..."
```

## Body

| Name | Type | Default | Description |
|---|---|---|---|
| url* | string | - | The start page, a public http/https URL. |
| depth | int | - | How many levels of links to follow below the start page. Default 2, max 5. |
| max_pages | int | - | The most pages fetched. Default 100, max 500; a larger value is cut to 500. |
| include | string | - | Follow only matching URLs: a path prefix (`/docs`), or a glob with `*` over the path (over the whole URL when it starts with `http`). Default: every URL. |
| exclude | string | - | Never follow matching URLs, in the same form as `include`. Default: none. |
| include_subdomains | bool | - | Also follow links to subdomains of the start URL's host. Default false. |
| content | string | md | `md` (default) · `html` · `none` — what each row carries: `markdown`, `html`, or neither (only the URL list with titles and status codes). |

## The job

json · POST /site → 202:

```json
{ "job_id": "j_...", "urls": 1, "status": "pending", "completes_by_ms": 1791334800000, "usage": { "units": 1, "boxes": ["jobs"] } }
```

| Route | Does |
|---|---|
| POST /site | Submit the body above. Returns `202` with the `job_id`. |
| GET /site/jobs/{id} | The status (`pending`, `running`, `done`, `paused` with a `paused_reason`), the counts, `completes_by_ms` and the job's `usage`. |
| GET /site/jobs/{id}/results | The rows, one JSON object per line, each with a `seq` number. Page with `?after=N` (500 per page). |
| POST /site/jobs/{id}/cancel | Stop a running job. It ends as `paused`. |

The same job also reads at `/scrape/jobs/{id}` and `/scrape/jobs/{id}/results`.

## Each row

json · one line of the results:

```json
{ "url": "https://docs.crawl4ai.com/core/installation", "level": 1, "title": "Installation", "status": 200, "links": 61, "markdown": "# Installation\n\n...", "seq": 3 }
```

| Field | Description |
|---|---|
| url | The page. |
| level | 0 for the start page, 1 for the pages it links to, and so on. |
| title | The page's `<title>`, else its first `# ` heading. |
| status | The site's HTTP status for the page. |
| links | How many internal links the page had. |
| markdown · html | The page's content, by `content`: `markdown` for `md`, `html` for `html`, neither for `none`. |
| error | Only on a page that failed: why. |

## Credits and errors

A site crawl costs 1 job unit when the job is accepted (the `usage` of the `202`), plus each page it fetches, priced as a [/scrape](https://crawl4ai.com/docs/scrape/) of that page. Each page is its own receipt in your usage history. No flat recipe price applies to a site crawl.

| Status | error | Meaning |
|---|---|---|
| 400 | bad_input | The body is not a JSON object, or an input has the wrong type. |
| 402 | no_credit | Your balance is empty. Start a plan, or buy extra credits on your plan, and submit again. |
| 404 | not_available | The service is not open to your account yet. |
| 405 | POST only | A method other than `POST` on `/site`. |

> **Note**
>
> A `url` that is not an http/https page gives a job with no rows.



**Try it with your key**

Run this call in the playground and see the real answer, the time and the units.
[Open the playground](https://crawl4ai.com/#playground)



---
Source: https://crawl4ai.com/docs/site/