Web scraping for RAG: any website to LLM-ready Markdown.

1What does a RAG crawl need?

A RAG pipeline (retrieval-augmented generation: the model reads pages you give it before it answers) is only as good as the text you put in. From the web, that text must be clean: no menus, no cookie banners, no scripts. It must also keep the headings, because your chunker uses them to cut the page.

The work has three steps. Find the URLs. Fetch each page as Markdown. On the next run, find what changed. The calls below do each step. Every output on this page is a real answer from the live API on 10 November 2026, shortened where it says so.

Step 1: find the URLs

Most sites list their pages in /sitemap.xml. Fetch it with /scrape and read the <loc> lines. A page we already hold in our archive costs half.

curl https://api.crawl4ai.com/scrape \
  -H "Authorization: Bearer $CRAWL4AI_KEY" -H "Content-Type: application/json" \
  -d '{"url":"https://docs.crawl4ai.com/sitemap.xml","format":"html"}'

{"ok":true,"engine":"archive","cache":"archive","ms":17,"text_len":5219,
 "html":"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset ...>\n  <url>\n    <loc>https://docs.crawl4ai.com/</loc>\n    <lastmod>2026-10-26</lastmod> ..."}

Real answer, shortened: the sitemap held 89 <loc> lines.

No sitemap? Fetch the home page with "parse":{"links":true} and keep the links on the same host.

Step 2: fetch the pages as Markdown

Send up to 50 URLs in one /scrape/batch call. The answer streams back as NDJSON, one line per URL as it finishes. For a long list (up to 10,000 URLs), use /scrape/jobs: it runs in the background and nothing waits on a slow page.

curl -N https://api.crawl4ai.com/scrape/batch \
  -H "Authorization: Bearer $CRAWL4AI_KEY" -H "Content-Type: application/json" \
  -d '{"urls":["https://docs.crawl4ai.com/core/cache-modes",
               "https://docs.crawl4ai.com/core/ask-ai"],"format":"md"}'

{"url":"https://docs.crawl4ai.com/core/cache-modes","ok":true,"result":{"engine":"fetch","ms":952,"text_len":4120,"content_hash":"65d80f7e9f8a...","markdown":"..."}}
{"url":"https://docs.crawl4ai.com/core/ask-ai","ok":true,"result":{"engine":"browser-light","ms":10837,"text_len":2411,"content_hash":"50f45758e980...","markdown":"..."}}

Real answer, shortened. The first page needed only a plain fetch. The second needed a browser, so it took 10.8 s.

One page alone is a /scrape call. This is its real answer for the library's quick-start page:

{"ok":true,"engine":"http","cache":"miss","ms":440,"text_len":24163,
 "final_url":"https://docs.crawl4ai.com/core/quickstart/",
 "content_hash":"7759a795ab401dca600d0d52082cf0c49c0c78e103fce1e2a01da9d9dea42024",
 "ladder":[{"rung":"http","outcome":"sounder:http","ms":117}],
 "markdown":"... # Getting Started with Crawl4AI\n\nWelcome to **Crawl4AI**, an open-source LLM-friendly Web Crawler & Scraper. In this tutorial, you’ll:\n\n1.  Run your **first crawl** using minimal configuration.\n2.  Generate **Markdown** output ..."}

Real answer, shortened. ladder lists the steps the service tried, in order.

2How do I re-embed only the pages that changed?

Every /scrape answer carries a content_hash of the page content. Store it next to the URL. On the next run, fetch the pages again and compare. The same hash means the same content: skip the page. A new hash means new content: re-chunk and re-embed that page only. A URL that left the sitemap is a page to delete from the index.

This keeps the embedding bill small, because most pages of a docs site or a knowledge base do not change from one week to the next.

For a few pages that must be fresh the same day (a pricing page, a status page, a policy), Watch (in beta) checks the page for you and calls your webhook when it changes. See the Watch docs.

3Can my agent search the web and read the results?

Yes. GET /search returns ranked results with title, URL and snippet. Your agent then reads the pages it needs with /scrape. A search costs 1 credit.

curl "https://api.crawl4ai.com/search?q=retrieval+augmented+generation+chunking" \
  -H "Authorization: Bearer $CRAWL4AI_KEY"

{"n":10,"ms":1501,"results":[
 {"title":"Chunking Methods on Retrieval-Augmented Generation","url":"https://arxiv.org/html/2606.00881v1","source":"gg","snippet":"May 30, 2026 — A standard RAG pipeline initially splits documents ..."},
 {"title":"Develop a RAG Solution on Azure - Chunking Phase","url":"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-chunking-phase","source":"gg","snippet":"Jun 18, 2026 — Chunking is breaking documents into appropriately sized chunks ..."}, ...]}

Real answer, shortened to 2 of 10 results.

In Claude Code, Cursor, Codex or any MCP client, add the hosted MCP server once. The agent then has scrape, search, batch and jobs_submit as tools:

claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp \
  --header "Authorization: Bearer $CRAWL4AI_KEY"

Agents can read the whole API from one file: crawl4ai.com/llms.txt. More in the MCP docs.

4What does a RAG crawl cost?

You pay for the effort each page needed. A failed call costs nothing. One credit is $0.001 at list price.

From GET /v1/prices (by_level.scrape), read 10 November 2026. A search is 1 credit.

Example, not a measurement: a docs site of 2,000 pages, 80% plain and 20% that need a browser, costs 1,600 × 0.1 + 400 × 0.5 = 360 credits, so $0.36 at list price. The free plan gives 1,000 credits every month. Paid plans: Starter $10 a month with 10,000 credits, Builder $99 with 125,000, Scale $499 with 750,000. See pricing.

5How do I load the pages into LangChain or LlamaIndex?

Read the NDJSON lines and turn each page into a document with its URL and hash as metadata. Your splitter and vector store stay as they are. This short Python example uses only the standard library and requests:

import json, os, requests

KEY = os.environ["CRAWL4AI_KEY"]
urls = ["https://docs.crawl4ai.com/core/cache-modes", "https://docs.crawl4ai.com/core/ask-ai"]

r = requests.post("https://api.crawl4ai.com/scrape/batch",
                  headers={"Authorization": f"Bearer {KEY}"},
                  json={"urls": urls, "format": "md"}, stream=True, timeout=120)
docs = []
for line in r.iter_lines():
    row = json.loads(line)
    if "summary" in row or not row.get("ok"):
        continue
    res = row["result"]
    docs.append({"text": res["markdown"],
                 "metadata": {"url": row["url"], "content_hash": res["content_hash"]}})
# docs -> your splitter (for example MarkdownHeaderTextSplitter) -> your vector store

6Should I run the open-source library or use the Cloud?

Crawl4AI Cloud runs the same engine as the open-source Crawl4AI library. The library gives the same Markdown on your own machine, for free. You then run the browsers, buy the proxies and handle the sites that block you. The Cloud does that part and charges only for pages that worked.

For a small docs site with no blocks, the library is often enough. For many sites, pages that need a browser, or sites that block plain requests, compare the numbers on build vs buy.