Why Crawl4AI gets blocked by Cloudflare, and what to do about it

Crawl4AI has become the default open-source answer to "I need clean Markdown from a web page for my RAG pipeline." It is well built, it is fast, and it produces output that drops straight into an LLM context window without a parsing layer. If you have used it, you know why it spread.

You also probably know the failure mode. It works beautifully on documentation sites, blogs, and most marketing pages. Then you point it at a target that sits behind Cloudflare, and instead of Markdown you get a challenge page, a 403, or an empty body with a 200 status.

Crawl4AI is working as designed. Its architecture produces exactly this failure.

What Crawl4AI does

Underneath, Crawl4AI drives a browser. Depending on configuration that is Playwright with Chromium, or a lighter HTTP fetch for simple cases. It navigates to the URL, waits for the page to settle, runs a readability-style extraction, and converts the result to Markdown.

That architecture is what makes it good: a real browser renders JavaScript, follows client-side routing, and produces a DOM that reflects what a user would see. And it is exactly what makes it detectable, because a browser launched by an automation framework does not look identical to a browser a person opened.

What a modern bot defence is looking at

It helps to separate the signals, because the fixes are different for each.

Network-layer identity. Every TLS handshake carries a fingerprint: the set of ciphers, extensions, and their ordering. That is commonly hashed as JA3 or JA4. The same is true one layer up for HTTP/2: frame settings, header ordering, and priority hints form a signature. A client that does not produce the fingerprint of a real browser build is identifiable before a single byte of HTML is exchanged.

IP reputation. Datacenter ranges are catalogued. Cloud providers publish their own IP ranges, and the major bot-management vendors maintain ASN reputation data. A request from an AWS or Hetzner address is scored differently from one from a residential connection, and no amount of browser patching changes that.

Browser fingerprint. Navigator properties, WebGL renderer strings, font lists, screen metrics, timezone, and the presence of automation markers. navigator.webdriver is the famous one, and also the easiest to patch. That is why it stopped being the thing that mattered.

Protocol-level tells. A browser driven over the Chrome DevTools Protocol exposes artefacts of that connection. Detecting CDP is a well-established technique and it survives most JavaScript patching.

Behaviour. Request cadence, mouse movement, navigation timing, and whether a session accumulates the messy history of a real user.

The challenge itself. Turnstile and its equivalents are not passive checks. They present an interactive challenge and score the interaction. Passing one requires either solving it or being convincing enough that it issues a token without friction.

The four options

1. Stealth plugins

playwright-extra combined with a stealth plugin patches the JavaScript-visible layer: it removes navigator.webdriver, fabricates a plugin list, normalises WebGL vendor strings, and fixes a handful of other properties.

Against a defence that only checks JavaScript-visible properties, this is often enough. Against a defence that also scores TLS fingerprints and IP reputation, it does not help, because neither of those lives in JavaScript. Teams frequently report exactly this: stealth plugins fix one target and do nothing for the next one.

It is cheap to try and worth trying first.

2. Route through residential or mobile IPs

This addresses the IP reputation signal, which is often the single largest contributor to a block. It is also the most predictable cost in the stack, typically billed per gigabyte, and the traffic is real.

Two things surprise people. First, static ISP proxies are often a better fit than rotating residential: a stable session on a clean ASN looks more like a returning visitor than a new IP on every request. Second, rotating per request is frequently worse than holding a session, because a defence that sees a session appear from twenty ASNs in a minute has learned something.

3. Run the whole stack yourself

Stealth plugin, residential proxy, a browser build whose TLS fingerprint matches a real Chrome release, CDP detection avoidance, and per-target tuning. This works. It is also a system rather than a script: the fingerprinting changes, the challenge vendors update, and each target needs its own configuration.

The build is the cheap part. Maintenance is what compounds. Most teams who move off a self-managed stack do it because keeping it working became a recurring cost against a roadmap that had nothing to do with scraping.

4. Use a managed extraction API

If the goal is clean data rather than running a crawler, an API that returns validated output shifts the fingerprinting problem to someone whose product it is. The tradeoff is real: you lose the ability to tune the browser, you pay per page rather than for your own compute, and you take a dependency.

When a page fetch is incidental to your product rather than the product itself, this is usually the correct trade.

A note on the cache

Before any of this: check whether you need to make the request at all. Crawl4AI supports caching, and documentation sites, product catalogs, and reference material rarely change between runs. A large share of blocked requests in most pipelines are re-fetches of pages that have not changed.

Deciding

Three questions, in order:

1. Does the page change between runs? If not, cache it and the problem mostly disappears.
2. Is the block IP-based or fingerprint-based? Test from a residential connection. If the same code passes, the problem is the ASN and a proxy tier is the fix. If it still fails, you are being fingerprinted and no proxy will save you.
3. Is fetching pages core to your product? If yes, owning the stack is defensible. If no, the maintenance cost compounds silently, and paying per page is usually cheaper than paying an engineer.

Treat it as a transport problem, because transport problems have transport-shaped solutions.