Recipes

POSThttps://api.crawl4ai.com/recipes/{name}

Ready-made scrapers. A recipe knows one site: you send its inputs (a URL, an id, a search word), you get rows back, every row with the same named columns. Find a recipe in the catalog, run it with one call. Browse and try them at /recipes/.

3 credits per record from a second source300 s a sync run at most1 hour a job runs at most6 hours between health checks

Browse every recipe: /recipes/

Find a recipe `GET /recipes`

curl https://api.crawl4ai.com/recipes -H "Authorization: Bearer sk_live_..."

Free (no credit). Returns {"recipes": [...]}, one object per recipe. A recipe's output lists its columns in order, and output_about gives one sentence per column: what it holds and in which form (an id as a string, a time in ISO 8601, an object with its keys).

{
  "name": "tiktok-post", "version": 1, "title": "One TikTok post",
  "kind": "module", "stage": "published", "run": "both", "tags": ["social"],
  "description": "One public TikTok video post as one row: its text and hashtags, ...",
  "inputs": { "post": { "type": "string", "default": null, "required": true, "example": "7654036719242726670", "secret": false } },
  "output": ["post_id", "url", "description", "hashtags", "created_at", "author", "stats", ...],
  "output_about": {
    "post_id": "TikTok's id of the post (digits, as a string).",
    "created_at": "When it was posted, ISO 8601 UTC (2026-06-22T02:11:46Z).",
    "stats": "{plays, likes, comments, shares, saves}, integers, as TikTok rounds them on the page.",
    ...
  },
  "cost": "1 unit; 1 record when it falls back", "fresh_for_s": 900,
  "health": { "status": "ok", "rows": 1, "ms": 1240, "checked_at": 1791331200 }
}

Field	Description
name	The recipe's name, the `{name}` in the run URL.
title · description · tags	What it reads, in a line and in two or three sentences; `tags` holds its one category (news, social, video, shopping, jobs, dev tools, knowledge, finance, reviews).
inputs	One entry per input: `type` (`string`, `int`, `bool`, `date`), `default`, `required`, an `example` value, `secret`. A `session` input also has `need`: `required` or `optional`.
output	The column names of every row, in order.
output_about	One sentence per column, keyed by the column name.
cost	A hint in words: the units a run usually takes, and "record" when the recipe can take rows from a second source (see the cost below).
run	`sync` (one call, the rows in the answer), `job` (submit, then read the rows) or `both`.
fresh_for_s	How long a result stays in the cache (see below).
stage · health · version · kind	See "Staged recipes and health". `kind` is `json`, `module` or `script`.

Run one

curl https://api.crawl4ai.com/recipes/hn-hiring \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "thread": 49522897, "keyword": "Rust" }'

`<input>` string · int · bool · date	One field per recipe input, as the catalog lists them. A missing input takes the recipe's default; a required one without a default fails with `bad_input`.
`bypass_cache` bool	Run again even when a fresh result is cached (see below).

{
  "recipe": "hn-hiring", "version": 1,
  "rows": [ { "company": "Senzing", "author": "samlk", "text": "Senzing | Platform Engineer | Remote (USA) ..." } ],
  "usage": {
    "units": 1, "boxes": ["crawl"], "cache": "miss", "engine": "recipe",
    "lines": [ { "call": "scrape", "url": "https://news.ycombinator.com/item?id=49522897", "units": 1, "cache": "archive", "engine": "archive" } ]
  },
  "meta": { "pages_fetched": 1, "from_cache": false, "elapsed_ms": 812, "llm_calls": 0, "warnings": [] }
}

Field	Description
rows	One object per row, exactly the recipe's `output` columns, in order. A column the page did not have is `null`.
usage	What this run used, the same object every endpoint returns: `units`, `llm_tokens` (only when an LLM ran), `boxes`, `remaining_before` on capped tiers. `lines` lists every inner call in order: each page (`scrape`, its `url`), each LLM call (`extract`, its `step`) and, for some recipes, one `records` line with the count of rows taken from the second source. Each line is one receipt in your usage history.
meta	`pages_fetched`, `llm_calls`, `elapsed_ms`, `from_cache`, and `warnings` (columns that came back empty, unmatched joins, a module's log lines).

Job recipes

A recipe with run: job reads many pages (a long list, every comment of a video) and can take minutes. Submit it, follow it, then read the rows. A recipe with run: both takes either call. A job recipe sent to the sync URL answers 400 use_jobs. A job runs for at most one hour.

curl https://api.crawl4ai.com/recipes/youtube-comments/jobs \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{ "video": "dQw4w9WgXcQ", "limit": 500 }'
# → 202 {"job_id": "j_...", "status": "pending"}

Route	Does
POST /recipes/{name}/jobs	Submit the recipe's inputs, the same body as a sync run. Returns a `job_id`.
GET /recipes/jobs/{id}	The status (`pending`, `running`, `done`, `paused` with a `paused_reason`), the counts and the job's `usage`.
GET /recipes/jobs/{id}/results	The rows, one JSON object per line, each with a `seq` number.
POST /recipes/jobs/{id}/cancel	Stop a running job. It ends as `paused`.

The cost

A run costs the call itself (the recipes line in the pricing table), plus each page it fetched and each LLM step, priced as a /scrape or an /extract you send yourself. Each of them is a line in usage.lines and its own receipt in your usage history; usage.units adds them up. The catalog's cost gives the usual figure before you call.

Some recipes have a second source: when the site's own page fails, they take the rows from it. Each row taken that way is one record, and a record costs 3 credits: $3 per 1,000 records, at $0.001 a credit. The catalog's cost says "record" for these recipes, and the receipt's records line gives the count. Records are charged only on a successful run, never on a cache hit. Each account has a daily limit of records; past it the run answers 503 vendor_off.

The result cache

Every recipe declares fresh_for_s. A second call with the same inputs inside that window is served from the cache: usage.cache is "hit", usage.units is 1 (the call only: no page, no record), meta.from_cache is true. Send "bypass_cache": true to run again. Recipes with a secret input are never cached.

Your session

Some recipes read a site as you: they take a session input, your own login cookies for that site as one Cookie header line. The Crawl4AI Session extension copies it in one click. Crawl4AI uses it for this run only and does not store it: no archive, no cache, no log. Your account and the site's terms are your responsibility. The catalog says per input whether it is a secret and whether the session is required or optional; without an optional session the recipe reads what a logged-out visitor sees. A wrong or expired session costs the run's unit: rows is empty, meta.warnings carries bad_session, and meta.blocked names the site.

Staged recipes and health

Every recipe in the catalog carries a stage: published, or staged for a new recipe that runs but is not yet promoted. Each region checks every recipe every six hours and the catalog carries the result as health (ok, fail or unknown, with the last check's rows, time and error; ours for a recipe that reads our own crawler and needs no check). GET /recipes/health returns the same, without a key.

Errors

Status	error	Meaning
400	bad_input	An unknown input, a required one missing, or a wrong type.
400	use_jobs	A `job` recipe sent to the sync URL: submit it to `/recipes/{name}/jobs`.
404	unknown_recipe	No recipe with that name.
403	host_refused	The recipe tried a host outside its allowed list.
422	page_budget · module_limit · bad_module	The run needed more pages than the recipe allows, or its module hit a limit.
499	cancelled	The job was stopped before it finished.
502 · 504	fetch_failed · timeout	A page could not be fetched, or the run passed 300 s (a job: one hour).
503	vendor_off	The site's page failed and the recipe's second source reached today's limit, or is paused. Try again later; the limit resets each day (UTC).

Note

A failed run still returns usage: the pages fetched before the failure were billed, and it says which. A partial result is never returned as success.

Try it with your key

Run this call in the playground and see the real answer, the time and the units.

Open the playground

Watch a page for changesPOST /monitors

Watch a page and get told when it changes in the way you care about.

Prices from pricing pagesPOST /price

Give it websites, get their prices.

Something unclear or wrong on this page? Tell us.