Crawl4AI Cloud

Crawl a whole site

POSThttps://api.crawl4ai.com/site

Every page under a URL, as Markdown, level by level: the start page (level 0), the pages it links to (level 1), and so on to depth. It follows only links inside the site, through your include and exclude filters, and stops at max_pages. A site crawl runs only as a job: submit it, follow it, then read the rows. A job runs for at most one hour. To list a site's URLs without fetching them, use /scan.

1 job unit when the job is accepted500 pages at most5 levels at most1 hour a job runs at most
# 1. submit -> job id
JOB=$(curl -s https://api.crawl4ai.com/site \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{ "url": "https://docs.crawl4ai.com/", "depth": 2, "max_pages": 50, "include": "/core" }' | jq -r .job_id)

# 2. poll until it ends (done, or paused with a reason)
while true; do
  S=$(curl -s https://api.crawl4ai.com/site/jobs/$JOB -H "Authorization: Bearer sk_live_..." | jq -r .status)
  case "$S" in done|paused) break;; esac; sleep 5
done

# 3. read the rows (NDJSON, one line per page)
curl -s https://api.crawl4ai.com/site/jobs/$JOB/results \
  -H "Authorization: Bearer sk_live_..."

Body

urlrequired
string

The start page, a public http/https URL.

depth
int

How many levels of links to follow below the start page. Default 2, max 5.

max_pages
int

The most pages fetched. Default 100, max 500; a larger value is cut to 500.

include
string

Follow only matching URLs: a path prefix (/docs), or a glob with * over the path (over the whole URL when it starts with http). Default: every URL.

exclude
string

Never follow matching URLs, in the same form as include. Default: none.

include_subdomains
bool

Also follow links to subdomains of the start URL's host. Default false.

content
string
Default md

md (default) · html · none — what each row carries: markdown, html, or neither (only the URL list with titles and status codes).

The job

{ "job_id": "j_...", "urls": 1, "status": "pending", "completes_by_ms": 1791334800000, "usage": { "units": 1, "boxes": ["jobs"] } }
Route Does
POST /site Submit the body above. Returns 202 with the job_id.
GET /site/jobs/{id} The status (pending, running, done, paused with a paused_reason), the counts, completes_by_ms and the job's usage.
GET /site/jobs/{id}/results The rows, one JSON object per line, each with a seq number. Page with ?after=N (500 per page).
POST /site/jobs/{id}/cancel Stop a running job. It ends as paused.

The same job also reads at /scrape/jobs/{id} and /scrape/jobs/{id}/results.

Each row

{ "url": "https://docs.crawl4ai.com/core/installation", "level": 1, "title": "Installation", "status": 200, "links": 61, "markdown": "# Installation\n\n...", "seq": 3 }
Field Description
url The page.
level 0 for the start page, 1 for the pages it links to, and so on.
title The page's <title>, else its first # heading.
status The site's HTTP status for the page.
links How many internal links the page had.
markdown · html The page's content, by content: markdown for md, html for html, neither for none.
error Only on a page that failed: why.

Credits and errors

A site crawl costs 1 job unit when the job is accepted (the usage of the 202), plus each page it fetches, priced as a /scrape of that page. Each page is its own receipt in your usage history. No flat recipe price applies to a site crawl.

Status error Meaning
400 bad_input The body is not a JSON object, or an input has the wrong type.
402 no_credit Your balance is empty. Start a plan, or buy extra credits on your plan, and submit again.
404 not_available The service is not open to your account yet.
405 POST only A method other than POST on /site.
Note

A url that is not an http/https page gives a job with no rows.

Try it with your key

Run this call in the playground and see the real answer, the time and the units.

Open the playground

Next

Scan a site's URLsPOST /scan

Every URL of one site, without crawling it.

Scrape a pagePOST /scrape

Fetch a page and return clean Markdown (and/or HTML).

PreviousExtractNextScan a site

Something unclear or wrong on this page? Tell us.