POST /v1/crawl

Crawl a site

Asynchronous deep crawling across a domain or path hierarchy. The request returns immediately with a crawl_id, and pages are delivered via webhook or polled as they complete.

Request

curl -X POST https://api.datamint.xyz/v1/crawl \
  -H "Authorization: Bearer mint_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/docs",
    "max_depth": 2,
    "max_pages": 50,
    "path_include": "^/docs/",
    "schema": "article",
    "webhook_url": "https://api.yourdomain.com/webhooks/datamint"
  }'

Parameters

urlstringrequired

The starting URL. Must be an absolute http or https URL.

max_depthinteger

How many link hops to follow from the starting URL. Integer from 0 to 10. Default: 2.

max_pagesinteger

Total page budget limit for this crawl job. Integer from 1 to 1,000. Default: 25.

path_includestring

Regex pattern for URLs that must be crawled (e.g. ^/blog/ or ^/docs/).

path_excludestring

Regex pattern for URLs that should be skipped (e.g. /(login|cart|checkout)/).

stealthboolean

Route all requests through residential proxies with fingerprint mutation across the crawl.

schemastring

A registered schema name (e.g. article, documentation). Every crawled page passes through extraction.

webhook_urlstring

HTTPS URL notified with the completed payload when the crawl job finishes.

max_depth is capped at 10 and max_pages defaults to 25 (up to 1,000 per job). Frontier expansion multiplies quickly, so bounds are strictly validated up front to prevent accidental runaway crawls.

Immediate Response & Delivery Modes

Because crawling is asynchronous, POST /v1/crawl returns 202 Accepted within 200ms with your job identifier:

{
  "crawl_id": "crl_9f8a1b2c3d4e",
  "status": "in_progress",
  "url": "https://example.com/docs",
  "max_pages": 50,
  "created_at": "2026-09-21T05:00:00Z"
}

You receive results in two ways:

1. Webhook Notification (Recommended)

If you supply a webhook_url, Datamint posts an application/json payload containing the full batch of extracted pages, per-page latency, and credit tallies once the crawl halts.

2. Status Polling & MCP Streaming

Poll job progress via GET https://api.datamint.xyz/v1/crawl?crawl_id=crl_... or query the MCP tool datamint_crawl_status and streaming resource datamint://crawls/{id}.

Cost

A deep crawl is billed as a dynamic headless render: it fans out across pages, and the per-page total is only known once it has run. Adding stealth moves the tier to stealth armor, and supplying a schema adds the extraction pass.

The quota check covers the crawl as a whole before it starts, so an account that cannot cover the work is refused up front instead of discovering it partway through. See Credits.

Errors

400 for a malformed body, a missing url, or a max_depth outside 0–10. 405 if you use GET. The rest are shared across the API. See Errors.

Need volume pricing or a custom parser?

Start freeTalk to sales