Guide
Crawl4AI: self-hosting, limits, and when to switch
What Crawl4AI does, how to self-host it, where it breaks on defended sites, and when a managed extraction API is the better fit.
Crawl4AI is the default open-source answer to "I need clean Markdown from a web page for a RAG pipeline." It is an Apache-2.0 Python library with 87.7k GitHub stars and a v0.9.6 release on 20 November 2026. It drives a real browser, extracts the main content, and returns Markdown, JSON, or HTML that drops into an LLM context window.
The library is good at that job. It works on blogs and documentation sites. The harder question is what you own when you run it yourself, and where it stops.
Facts below come from docs.crawl4ai.com and the unclecode/crawl4ai GitHub repository, read on 22 November 2026. Version numbers, limits, and cloud terms change; re-check them before pinning a production pipeline.
What Crawl4AI does
Crawl4AI is a Python crawler and scraper built for LLM input. It loads a page in a browser, waits for the content to settle, runs a readability-style extraction, and converts the result to Markdown. It can also return structured JSON, raw HTML, screenshots, PDFs, links, media, and page metadata.
The feature set goes beyond a single fetch:
- Extraction with CSS selectors, XPath, regex, or an LLM, including schema generation.
- Crawling in single-page, deep, adaptive, and concurrent modes.
- Browser control through profiles, CDP, sessions, proxies, and a stealth adapter.
- Caching, hooks, JavaScript execution, and full-page scrolling.
- Interfaces as a Python library, CLI, Docker REST API, and MCP server.
The open-source library is Apache-2.0 and needs no API key. Crawl4AI Cloud is a separate hosted product from the same project; it adds /search, /serp, and /answer, and manages browsers and proxies for you. The cloud has a free tier of 1,000 credits per month at 10 requests per minute and 2 concurrent requests, with paid plans from $10 per month at 120 requests per minute and 10 concurrent. The library and the cloud are different products with different terms.
Install and run it
The documented install is two commands:
pip install -U crawl4ai
crawl4ai-setup
crawl4ai-setup installs the browser once. crawl4ai-doctor checks the installation if something does not start. The README's sample is labeled Python 3; the pages checked do not state a numeric minimum version.
For Docker, the image is unclecode/crawl4ai:latest. The Docker server expects CRAWL4AI_API_TOKEN; without it, access is limited to the container.
A minimal crawl looks like this:
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
async def main():
browser_conf = BrowserConfig(headless=True)
run_conf = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
async with AsyncWebCrawler(config=browser_conf) as crawler:
result = await crawler.arun(
url="https://example.com",
config=run_conf,
)
print(result.markdown)
asyncio.run(main())
CacheMode.BYPASS is the default. For scheduled re-crawls of stable pages, CacheMode.ENABLED is the setting that avoids re-fetching content that has not changed.
Where Crawl4AI stops
The architecture that makes Crawl4AI good is also what makes it detectable. It drives a browser launched by an automation framework, usually from a datacenter IP. Modern bot defences score the TLS fingerprint, the HTTP/2 signature, the IP's ASN reputation, the browser fingerprint, and the artefacts of the automation connection. A stock browser from a cloud address does not look like a person on a residential connection.
Crawl4AI ships enable_stealth and an undetected-browser adapter. Those help against defences that only check JavaScript-visible properties. They do not guarantee a pass against a defence that also scores transport fingerprints and IP reputation, and the project does not claim a guaranteed bypass.
The failure mode is familiar: a challenge page, a 403, or an empty body with a 200 status. The Crawl4AI Cloudflare guide covers the signals in detail and the four practical fixes: stealth plugins, residential or mobile IPs, running the full stack yourself, and moving the transport problem to a managed API.
Before any of that, check whether you need the request. Documentation, product catalogues, and reference material rarely change between runs. Caching removes a large share of blocked requests in most pipelines.
Self-hosted versus managed: what you own
Self-hosting costs more than compute. You also own the layer that breaks when a target stops returning content.
| Layer | Self-hosted Crawl4AI | Managed extraction API |
|---|---|---|
| Browser | You install, pin, and update the browser and its dependencies | Vendor |
| IP and proxy | You supply, rotate, and pay for the network | Vendor routes across transport tiers |
| Anti-bot handling | Stealth adapter plus your own tuning; no guarantee | Vendor's stealth tier |
| Scaling | Your concurrency, memory, and queue | Vendor |
| Output contract | Extraction output from your code; validate downstream | Schema pass available |
| Cost shape | Compute, proxies, and engineering time | Published credits per page |
datamint publishes the managed side as credits: 1 credit for a static fetch, 5 for a headless render, 15 for stealth routing, and 2 more for a schema pass, at $1 per 1,000 credits. A failed request costs nothing. Those are the numbers to put against your own compute, proxy, and maintenance bill.
Self-hosting can be cheaper at scale and gives you control that a managed API does not. A managed API removes the browser, proxy, and maintenance work.
When to switch
Four questions, in order:
- Is the page public and static? If it is documentation, a blog, or a catalogue that rarely changes, Crawl4AI plus caching is a good fit. There is no transport problem to solve.
- Is the block IP-based or fingerprint-based? Test the same code from a residential connection. If it passes, the ASN is the problem and a proxy tier is the fix. If it still fails, you are being fingerprinted, and a proxy alone will not save you.
- Does the output need a contract? Crawl4AI can generate a schema and extract against it. If the next system must reject a response that does not match, add validation downstream or use a service that makes the schema pass part of the request.
- Is fetching pages core to your product? If yes, owning the stack is defensible. If fetching is incidental, the maintenance cost compounds silently, and paying per page is usually cheaper than paying an engineer.
Alternatives
Firecrawl is the managed option with the broadest surface: crawl, map, search, agent, and monitoring, plus an official MCP server. See the Firecrawl alternatives page and the Firecrawl MCP guide.
Jina Reader is the shortest path from one URL to Markdown: prefix https://r.jina.ai/. It does not crawl and does not bypass anti-bot protection. The Jina Reader guide covers its endpoints, rate limits, and licensing.
datamint takes a URL and returns Markdown or JSON validated against a schema. It routes each request across a static fetch, a headless render, and stealth routing, with the credit costs above. PII stripping at ingestion and configurable robots.txt compliance are part of the product. See the URL to Markdown API, pricing, or the MCP server docs.
Crawl4AI Cloud is the hosted version of the project. It is the option if you want Crawl4AI's interface without running browsers, and it adds search and answer endpoints that the open-source library does not have.
FAQ
Is Crawl4AI free?
Yes. The library is Apache-2.0 and has no API key or paywall. You pay for the compute, browsers, and proxies you run it on. Crawl4AI Cloud is a separate hosted product with a free tier of 1,000 credits per month and paid plans from $10 per month.
Does Crawl4AI bypass Cloudflare?
It can pass some defences, but it does not guarantee a bypass. The library ships a stealth adapter and an undetected-browser adapter; the project does not claim a guaranteed pass. Against a target that scores TLS fingerprints and IP reputation, the transport layer usually needs more than browser patching. The Cloudflare guide covers the options.
Does Crawl4AI return JSON?
Yes. Outputs include Markdown, structured JSON, raw HTML, screenshots, and PDFs. Extraction can use CSS, XPath, regex, or an LLM, and the library can generate a schema. If your pipeline needs a hard validation guarantee, validate the object downstream.
Can I self-host Crawl4AI with Docker?
Yes. The image is unclecode/crawl4ai:latest, and the Docker server expects CRAWL4AI_API_TOKEN. Without the token, access is limited to the container.
What is the difference between Crawl4AI and Firecrawl?
Crawl4AI is an open-source library you run yourself. Firecrawl is a managed API with crawl, map, search, agent, and monitoring operations. Crawl4AI gives you control over the browser and pipeline; Firecrawl gives you a hosted surface and an official MCP server. The Firecrawl alternatives page compares the pricing.
Is Crawl4AI Cloud the same as the open-source library?
No. The library is Apache-2.0 and runs locally. Crawl4AI Cloud is a hosted product from the same project; it manages browsers and proxies and adds /search, /serp, and /answer. The two have separate terms and pricing.
Sources
- Crawl4AI documentation: https://docs.crawl4ai.com/
- Crawl4AI repository and README: https://github.com/unclecode/crawl4ai
- Crawl4AI quickstart: https://docs.crawl4ai.com/core/quickstart/
The repository showed v0.9.6, released 20 November 2026, and 87.7k stars when read on 22 November 2026. The documentation was labeled v0.9.x. Limits and cloud terms can change; verify them against the current pages before relying on a number in production.
This guide covers retrieving publicly accessible data. Check the target's robots.txt and terms before crawling it, and do not use Crawl4AI to access authenticated or paywalled content.
datamint returns schema-validated JSON and LLM-ready Markdown from bot-defended sites through one API call, with routing across transport tiers handled on our side.