--- name: datamint description: Web data ingestion, anti-bot bypass, structured schema extraction, and deep asynchronous crawling via the Datamint API (https://api.datamint.xyz) and Model Context Protocol (MCP) server. Use when fetching web content, extracting structured data, or crawling bot-defended sites. --- # Datamint Agent Skill Datamint provides high-reliability web data ingestion and extraction infrastructure designed specifically for autonomous AI agents, LLM pipelines, and automated research workflows. It handles TLS/JA4 fingerprinting, residential proxy rotation, CAPTCHA click-solving, and schema validation automatically. -------------------------------------------------------------------------------- ## 1. Quick Reference & Endpoints | Protocol / Interface | Endpoint / Command | Primary Use Case | | :--- | :--- | :--- | | **REST: Mint** | `POST https://api.datamint.xyz/v1/mint` | Synchronous clean Markdown or schema JSON (~800ms–2.5s) | | **REST: Extract** | `POST https://api.datamint.xyz/v1/extract` | Strict schema validation (canonical or custom JSON Schema) | | **REST: Crawl** | `POST https://api.datamint.xyz/v1/crawl` | Asynchronous multi-page frontier crawl with webhook/polling | | **MCP (Hosted SSE)** | `https://mcp.datamint.xyz/sse` | Zero-install agent connection for Cursor, Claude, Windsurf | | **MCP (Local Stdio)** | `npx -y @datamint/mcp` | Local stdio transport for Claude Desktop or CLI agents | | **MCP (Container)** | `ghcr.io/dataminthq/datamint-mcp:latest` | Isolated Docker / Kubernetes agent sandboxes | -------------------------------------------------------------------------------- ## 2. Authentication & Headers Every request to the Datamint REST API authenticates using a Bearer token: ```http Authorization: Bearer mint_live_xxxxxxxxxxxxxxxxxxxxxxxx Content-Type: application/json ``` - **Production Keys**: Prefixed with `mint_live_` (metered against active credit balance). - **Test Keys**: Prefixed with `mint_test_` (sandbox mode; requests succeed against sample targets without deducting live credits). - **Environment Variable**: Set `DATAMINT_API_KEY="mint_live_..."` in your runtime environment. -------------------------------------------------------------------------------- ## 3. Cost & Credit Multipliers Credits are billed at **$1.00 USD = 1,000 credits**. Every account receives 1,000 free credits monthly. Requests that fail before extraction (`400`, `401`, `402`, `502`) consume **0 credits**. The transport cost ladders automatically based on target difficulty: | Tier | Workload Flag | Credits / Page | Latency | Capabilities | | :--- | :--- | :--- | :--- | :--- | | **Tier 1** | Default (`js: false, stealth: false`) | **1 credit** | ~800ms | Direct HTTP/2 with BoringSSL Chrome 133 TLS/JA4 fingerprinting | | **Tier 2** | `js: true` | **5 credits** | ~2.5s | Headless Chromium, DOM hydration, wait states, canvas render | | **Tier 3** | `stealth: true` | **15 credits** | ~4.5s | Rotating residential IP pool, CAPTCHA click-solver, Bezier kinematics | | **Schema** | `schema: "..."` or `schema: { ... }` | **+2 credits** | — | Additive JSON Schema validation pass on top of fetch tier | *Rule*: Setting both `js: true` and `stealth: true` costs **15 credits** (Stealth supersedes JS; tiers are a ladder, not a sum). -------------------------------------------------------------------------------- ## 4. API Usage & Request Examples ### Single-Page Mint (`POST /v1/mint`) Extracts clean, noise-free Markdown with navigation, scripts, and cookie banners stripped: ```bash curl -X POST https://api.datamint.xyz/v1/mint \ -H "Authorization: Bearer $DATAMINT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/article/123", "js": false, "stealth": false, "no_cache": false }' ``` **Optional Parameters**: - `js` (*boolean*): Execute JavaScript in headless browser (Tier 2 · 5 credits). - `stealth` (*boolean*): Route through rotating residential proxy pool and solve CAPTCHAs (Tier 3 · 15 credits). - `schema` (*string | object*): Registered name or inline JSON Schema (+2 credits). - `no_cache` (*boolean*): Bypass cache and force fresh upstream re-fetch. - `wait_for_selector` (*string*): CSS selector to wait for before extracting DOM (e.g. `"#content"`). - `timeout` (*number*): Execution timeout in seconds (5–60, default: 30). **Response Structure**: ```json { "markdown": "# Article Headline\n\nFull clean article content...", "status": 200, "_mint_meta": { "tier_used": "tier_1", "credits_billed": 1, "latency_ms": 342, "confidence": 0.98 } } ``` --- ### Structured Schema Extraction (`POST /v1/extract`) Extracts zero-code typed JSON conforming strictly to a canonical or custom schema: ```bash curl -X POST https://api.datamint.xyz/v1/extract \ -H "Authorization: Bearer $DATAMINT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/products/laptop", "schema": "ecommerce_item", "stealth": true }' ``` #### Pre-mapped Canonical Schemas: - `ecommerce_item`: `product_name`, `price`, `currency`, `in_stock`, `description`, `sku`, `brand`, `images`, `rating`, `review_count` - `article`: `headline`, `author`, `date_published`, `summary`, `content_markdown`, `lead_image`, `tags` - `job_posting`: `title`, `company_name`, `location`, `remote_policy`, `employment_type`, `salary_range`, `requirements`, `posted_at` - `company_profile`: `name`, `domain`, `industry`, `headquarters`, `founded_year`, `employee_count`, `description`, `social_profiles` - `real_estate_listing`: `address`, `price`, `property_type`, `bedrooms`, `bathrooms`, `square_feet`, `description`, `amenities`, `mls_id` - `faq`: `questions: Array<{ question: string, answer: string }>` **Custom JSON Schema Example**: ```json { "url": "https://example.com/company/about", "schema": { "type": "object", "properties": { "founded": { "type": "number" }, "founders": { "type": "array", "items": { "type": "string" } }, "mission": { "type": "string" } }, "required": ["founded", "founders"] } } ``` --- ### Asynchronous Deep Crawl (`POST /v1/crawl`) Walks an entire domain or URL sub-tree asynchronously: ```bash curl -X POST https://api.datamint.xyz/v1/crawl \ -H "Authorization: Bearer $DATAMINT_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/docs", "max_depth": 2, "max_pages": 50, "path_include": "^/docs/", "schema": "article", "webhook_url": "https://your-agent.com/webhooks/datamint" }' ``` **Parameters**: - `url` (*string, required*): Starting seed URL. - `max_depth` (*integer*): Link hop limit from seed (0–10, default: 2). - `max_pages` (*integer*): Page budget ceiling for the job (1–1,000, default: 25). - `path_include` (*string*): Regex filter for allowed URLs. - `path_exclude` (*string*): Regex filter for skipped URLs. - `stealth` (*boolean*): Residential proxy routing across the entire crawl. - `schema` (*string*): Registered schema name applied to all crawled pages. - `webhook_url` (*string*): HTTPS endpoint notified upon job completion. **Immediate Response (202 Accepted)**: ```json { "crawl_id": "crl_8f1a2b3c4d5e", "status": "in_progress", "url": "https://example.com/docs", "max_pages": 50, "created_at": "2026-09-21T05:00:00Z" } ``` **Status Polling**: ```bash curl "https://api.datamint.xyz/v1/crawl?crawl_id=crl_8f1a2b3c4d5e" \ -H "Authorization: Bearer $DATAMINT_API_KEY" ``` -------------------------------------------------------------------------------- ## 5. Model Context Protocol (MCP) Setup For Cursor, Claude Desktop, Claude Code, and Windsurf, connect via the hosted SSE endpoint or local stdio: ### Cursor Configuration (`~/.cursor/mcp.json`) ```json { "mcpServers": { "datamint": { "url": "https://mcp.datamint.xyz/sse", "env": { "DATAMINT_API_KEY": "mint_live_xxxxxxxxxxxxxxxxxxxxxxxx" } } } } ``` ### Claude Desktop Configuration (`claude_desktop_config.json`) ```json { "mcpServers": { "datamint": { "command": "npx", "args": ["-y", "@datamint/mcp"], "env": { "DATAMINT_API_KEY": "mint_live_xxxxxxxxxxxxxxxxxxxxxxxx" } } } } ``` ### Official Container Execution ```bash # Interactive stdio docker run -i --rm -e DATAMINT_API_KEY="mint_live_xxx" ghcr.io/dataminthq/datamint-mcp:latest # Background SSE service (Port 3001) docker run -d --name datamint-mcp -p 3001:3001 \ -e DATAMINT_API_KEY="mint_live_xxx" \ ghcr.io/dataminthq/datamint-mcp:latest \ node bin/datamint-mcp.js --transport sse --port 3001 ``` ### MCP Tools Reference 1. `datamint_mint`: Single-page fetch returning Markdown or JSON. 2. `datamint_extract`: Structured data extraction with canonical or custom schemas. 3. `datamint_crawl`: Launch asynchronous site crawl. 4. `datamint_crawl_status`: Check crawl progress and stream extracted pages. 5. `datamint_estimate_cost`: Calculate exact credits and USD cost before running jobs. ### MCP Resources Reference - `datamint://schemas`: Catalog of all pre-mapped canonical schemas. - `datamint://schemas/{name}`: Full JSON Schema definition for a canonical model. - `datamint://system/status`: Real-time fleet health and proxy pool availability. - `datamint://crawls/{id}`: Extracted page items for a given crawl ID. -------------------------------------------------------------------------------- ## 6. Error Handling & Stable Codes All errors return a standardized JSON error envelope: ```json { "error": { "code": "insufficient_credits", "message": "This request costs 17 credits and your balance is 4.", "required": 17, "balance": 4 } } ``` *Guidance for Agents*: Always branch on `error.code`, never on `error.message`. | Status | Code | Meaning & Recommended Agent Action | | :--- | :--- | :--- | | `400` | `invalid_request` | Check parameter types, ensure URL is absolute HTTP/HTTPS, or verify schema syntax. | | `401` | `unauthorized` | Missing, malformed, or revoked API key. Verify `Authorization: Bearer mint_live_...`. | | `402` | `insufficient_credits` | Credit balance too low. Prompt user to upgrade or purchase additional credits. | | `405` | `invalid_request` | Method not allowed. All extraction endpoints require HTTP `POST`. | | `502` | `upstream_error` | Extraction engine temporary unreachable. Retry with exponential backoff. | -------------------------------------------------------------------------------- ## 7. Python & Node.js Integration Snippets ### Python (httpx) ```python import httpx import os API_KEY = os.environ.get("DATAMINT_API_KEY", "mint_live_...") BASE_URL = "https://api.datamint.xyz" headers = { "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json" } # 1. Single Page Extraction def fetch_clean_markdown(url: str, stealth: bool = False) -> str: with httpx.Client(timeout=30.0) as client: res = client.post( f"{BASE_URL}/v1/mint", headers=headers, json={"url": url, "stealth": stealth} ) res.raise_for_status() return res.json()["markdown"] # 2. Structured JSON Schema def extract_product(url: str) -> dict: with httpx.Client(timeout=30.0) as client: res = client.post( f"{BASE_URL}/v1/extract", headers=headers, json={"url": url, "schema": "ecommerce_item"} ) res.raise_for_status() return res.json() ``` ### TypeScript / Node.js ```typescript const API_KEY = process.env.DATAMINT_API_KEY || "mint_live_..."; const BASE_URL = "https://api.datamint.xyz"; async function extractSchema(url: string, schema: string | object): Promise { const response = await fetch(`${BASE_URL}/v1/extract`, { method: "POST", headers: { "Authorization": `Bearer ${API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url, schema, stealth: true }), }); if (!response.ok) { const err = await response.json(); throw new Error(`Datamint API error: ${err.error?.code} - ${err.error?.message}`); } return response.json() as Promise; } ```