Build vs buy

Build vs buy a web scraper: the real cost, line by line.

Build when the job is narrow and stable: a few sites, plain HTML, no blocks, low volume. Then the free, open-source Crawl4AI library on one small server is often the cheapest answer. Buy when pages need a real browser, sites block plain requests, or many sites must run every day: then proxy traffic, browser time and engineer hours cost more than an API.

Last checked 10 November 2026 · every outside price links to its source · our prices from GET /v1/prices

Measure it on your own URLs

One prompt, and your agent measures it

Copy this, paste it into your coding agent (Claude Code, Cursor, Codex), add your API key, run it. It measures your real mix of pages and writes the cost report.

Get a free API key · 1,000 free credits every month, no card.

Measure what my crawling costs: self-hosted versus Crawl4AI Cloud.

GOAL
Take a sample of my real URLs, fetch them through Crawl4AI Cloud, measure how many pages are plain,
need a browser, are hard (the site fought back) or fail, and write a monthly cost report for both options.

CRAWL4AI CLOUD
- Read the API reference for agents first: https://crawl4ai.com/llms.txt
- Base URL: https://api.crawl4ai.com
- Header on every call: Authorization: Bearer $CRAWL4AI_KEY
- My API key: YOUR_CRAWL4AI_API_KEY
  (no key yet: get a free key at https://crawl4ai.com/signin/)
- Put the key in the environment variable CRAWL4AI_KEY. Never write it into a file in the repository.
- Optional, use the MCP tools instead of HTTP:
  claude mcp add --transport http crawl4ai https://api.crawl4ai.com/mcp --header "Authorization: Bearer $CRAWL4AI_KEY"

INPUTS (ask me for any that are missing)
- URLS: a file with my URLs, one per line
- SAMPLE: how many to test, default 100 (pick them at random, spread over all the sites in the file)
- MONTHLY_PAGES: how many pages I fetch in a month
- ENGINEER_HOURS: hours a month my team spends on the crawler (fixes, blocked sites, on-call)
- HOURLY_RATE: what one engineer hour costs us

STEPS
1. Fetch the sample.
   POST /scrape/batch  {"urls": [up to 50 URLs], "format": "md"}  (repeat for the rest of the sample)
   The answer is NDJSON: one line per URL, then one {"summary": ...} line.
   From each line keep: url, ok, cost (credits), effort, result.engine, result.ms, and the error reason when ok is false.
   Set the HTTP timeout to 120 seconds; do not cancel early. A failed URL is not charged.
2. Group the sample. Use effort when the line has it (levels[] in GET /v1/prices: 0 = archive, 1 and 2 = plain,
   3 = real browser, 4 = hard). Otherwise use result.engine: archive, cache, http and fetch = plain;
   browser-light and browser = browser; browser-heavy = hard. ok false = failed.
   Report the share of each group, the median ms of each, and the credits per page of each (from cost).
3. Read the live prices: GET https://api.crawl4ai.com/v1/prices
   by_level.scrape = dollars per 1,000 pages by effort; plans[] = label, usd, credits, extra_usd_per_1k;
   limits..allowance_per_month = the most credits a plan can use in a month.
4. Crawl4AI Cloud cost for MONTHLY_PAGES: average credits per page from the sample x MONTHLY_PAGES.
   For each plan: usd + max(0, credits - plan credits) / 1000 x extra_usd_per_1k, if credits fit the allowance.
   Pick the cheapest plan. Above the Scale allowance, write "Enterprise, ask for a quote".
5. Self-hosted cost for MONTHLY_PAGES, with these inputs (show each one in the report; I may change them):
   - browser time: (browser pages + hard pages / (1 - block rate)) x seconds per browser page / 3600 x $ per browser hour
     defaults: seconds = the measured median ms of browser pages / 1000; $0.10 per browser hour (Browserbase Startup plan, browserbase.com/pricing)
   - residential proxy traffic: hard pages / (1 - block rate) x MB per page / 1000 x $ per GB
     defaults: block rate 30%; 1.87 MB per page (HTTP Archive Web Almanac 2025 median inner page); $5.51 per GB (IPRoyal, 10 GB pay as you go)
   - servers: ask me, default $50 a month
   - engineer time: ENGINEER_HOURS x HOURLY_RATE
6. Write cost_report.md: the sample table (group, share, median ms, credits per page), the failed URLs with reasons,
   the Cloud cost and plan, the self-hosted cost line by line, and every assumption with its source.

EXPECTED OUTPUT
sample.csv (one row per URL) and cost_report.md.
Run SAMPLE=20 first, show me the table, then ask before the full sample.

1What does "build" mean for a web scraper?

Fetching one page is easy. A crawler that runs every day needs more parts: a browser pool for pages that need JavaScript, proxies for sites that block datacenter addresses, retries and fallbacks, a queue, storage, and someone who fixes it when a site changes. We maintain both the open-source Crawl4AI library and Crawl4AI Cloud, so here is what each one leaves to your team.

Part	Self-hosted library	Crawl4AI Cloud
The engine (fetch, browser, Markdown)	included, Apache-2.0	the same engine
Proxies	you buy and rotate them	included
Real browsers at scale	you run them	included
Handling sites that block requests	your team	ours
Failed calls	your cost	free
Regions	where you host	EU, Singapore, US
Support	GitHub and Discord	email, reply in 4 business hours to 2 business days by plan

2What are the cost inputs of a self-hosted crawler?

These are public prices and public numbers. Each one links to its source and says when we read it. Put your own numbers in the calculator below when you have them.

Input	Public number	Source, read 10 November 2026
Residential proxy traffic	$8 per GB pay as you go; $7, $6 and $5 per GB on the $499, $999 and $1,999 monthly plans	Bright Data pricing (the page also shows a 50% coupon for 3 months)
Residential proxy traffic	$7.35 per GB for 1 GB, $5.51 per GB for 10 GB, pay as you go; $1.84 per GB from 10 TB	IPRoyal pricing
Hosted browser time	$0.12 per browser hour after 100 hours ($20 plan); $0.10 after 500 hours ($99 plan)	Browserbase pricing
Servers	shared vCPU from €6.49 a month; dedicated vCPU from €43.49 a month, without VAT	Hetzner Cloud
Page weight (what a browser downloads)	median inner page 1,866 KB; median home page 2,710 KB (July 2025)	HTTP Archive Web Almanac 2025
Engineer pay	US software developer median $135,980 a year (May 2025), about $65 an hour before overhead	US Bureau of Labor Statistics
Blocked requests	at a 30% block rate, each good page takes 1 / 0.7 = 1.43 tries	arithmetic

Our side, from GET /v1/prices on the same day: a page already in our archive $0.05, a plain page $0.10, a real browser page $0.50 and a hard site $2.00 per 1,000 pages. A failed page is free. Plans: Free with 1,000 credits a month, Starter $10 with 10,000, Builder $99 with 125,000, Scale $499 with 750,000. Extra credits on a plan cost $1.00, $0.79 and $0.67 per 1,000 credits. One credit is $0.001 at list price.

3What would my crawl cost each way?

Change any number. The self-hosted side adds browser time, residential proxy traffic for the hard pages, servers and engineer time. The Crawl4AI side counts credits by the effort each page needed and picks the cheapest plan.

Pages a month Share that needs a browser (%) Share of hard sites (%) Block rate on hard sites (%) Seconds per browser page MB per hard page $ per GB of residential traffic $ per browser hour Servers, $ a month Engineer hours a month $ per engineer hour

Self-hosted, a month$2,147

Crawl4AI Cloud, a month$201.70255,000 credits a monthon Builder, extra credits at the plan's rate

The formula. Hard tries = pages × hard share / (1 − block rate). Browser hours = (pages × browser share + hard tries) × seconds per page / 3,600. Proxy GB = hard tries × MB per page / 1,000. Self-hosted = browser hours × $ per hour + proxy GB × $ per GB + servers + engineer hours × rate. Crawl4AI credits = plain pages × 0.1 + browser pages × 0.5 + hard pages × 2; a failed page is 0.

The defaults are an example, not a measurement. Plain pages on your own servers cost only server time, so they sit in the servers line. The browser hours use a hosted browser price; your own servers can be cheaper per hour, so put your number in.

4A worked example: one million pages a month

This is the calculator's default case. It is an example mix, not a measurement of any customer: 1,000,000 pages a month, 80% plain, 15% need a browser, 5% are hard sites that block 30% of tries.

Line	Self-hosted	Crawl4AI Cloud
Browser time	615 h × $0.10 = $61.51	included
Residential traffic	71,429 tries × 1.87 MB = 133.6 GB × $5.51 = $735.98	included
Servers	$50 (example)	included
Engineer time	20 h × $65 = $1,300 (example)	none for the crawling
Credits	-	800,000 × 0.1 + 150,000 × 0.5 + 50,000 × 2 = 255,000
A month	$2,147.49	$201.70 on Builder: $99 + 130,000 extra credits × $0.79 per 1,000

The largest line on the self-hosted side is engineer time, then residential traffic. Take both away: with no hard sites and no engineer time, the same million pages cost about $92 self-hosted (browser time $41.67 and servers $50) against $126.65 on Builder (160,000 credits). That is the case where building wins.

A small case: a 10,000-page docs site, all plain pages, fetched once a month, is 1,000 credits. That fits the free plan. It also runs well on the library on one small server.

5Which costs do not show on a bill?

Blocked tries. A blocked request still uses proxy traffic and browser time. At a 30% block rate you pay for 1.43 tries per good page; at 50%, for 2.
Failed pages. On your own stack, a failed page costs the same as a good one. On Crawl4AI Cloud a failed page is not charged.
Engineer time that moves. A site changes its layout or its block rules, and someone stops other work to fix the crawler. Count those hours for one month before you trust any estimate, including ours.
Waiting. A page that needs a real browser often takes 10 to 60 seconds. Your queue and timeouts must allow for it.

6Can I build part of it and buy the rest?

Yes. Many teams run the library for the easy sites and send only the hard pages to the Cloud. The output is the same Markdown from the same engine, so the rest of the pipeline does not change. You pay the Cloud only for the pages that needed it, and nothing for the ones that failed.

7What about a dedicated fleet or our own network?

On Enterprise we run the same engine as a dedicated fleet, inside your network, or as a finished data pipeline. A dedicated fleet is quoted on your volume, billed yearly. If you already pay another provider, bring your invoice: we compare cost per 1,000 pages and the success rate on your hardest URLs in a one-week pilot.

8Questions

Is Crawl4AI Cloud the same engine as the open-source library?

Yes. Crawl4AI Cloud runs the open-source Crawl4AI engine, written by the same team. Nothing is forked. The Cloud adds the browsers, the proxies, the handling of sites that block requests, and the three regions.

Can I move back to self-hosting later?

Yes. The library is Apache-2.0 and stays open source. It produces the same Markdown, so the rest of your pipeline does not change.

How much proxy traffic does one page use?

A browser that loads every file of a page downloads the whole page weight. The HTTP Archive Web Almanac 2025 gives a median of 1,866 KB for an inner page and 2,710 KB for a home page. A plain fetch of the HTML alone is much smaller.

Do I pay for pages that fail on Crawl4AI Cloud?

No. A blocked page, a login wall, a site that does not answer and our own errors are not charged. In a job, URLs that never got an answer are refunded at the end.

When should I build it myself?

When the job is narrow and stable: a few sites, plain HTML, no blocks, low volume, and nobody loses money when it stops for a day. Then the free library on one small server is often the cheapest answer.

Next: RAG and knowledge for LLMs · market intelligence · pricing · credits docs