Context for AI

Read a site

Feed a whole website to your AI as clean markdown. Crawls run asynchronously: you start a job, then poll it until it's done. You only pay for pages we successfully return.

Start a crawl

Submit a seed URL and a page limit. Returns a job id immediately.

POST/v1/crawl

Parameters

urlstringrequired

The seed URL to crawl from. Same-domain pages are followed.

limitinteger

Max pages to crawl (1–500, default 100). Bounds your cost — you are never charged beyond this many pages.

formatstring

Output format for each page: markdown (default), html, or text.

Request

cURL

curl https://api.apexapi.dev/v1/crawl \
  -H "Authorization: Bearer ak-your-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 200,
    "format": "markdown"
  }'

Response

202 Accepted

{ "id": "d1f8…", "status": "running" }

Get results

Poll the job until status is completed. While running it returns { status: "running" }.

GET/v1/crawl/{id}

Completed response

200 OK

{
  "id": "d1f8…",
  "status": "completed",
  "page_count": 143,
  "cost": 0.715,
  "result_url": "https://uploads.apexapi.dev/crawl-results/d1f8….json",
  "pages": [
    { "url": "https://docs.example.com/", "title": "Docs", "markdown": "# Docs…", "status": 200 }
  ]
}

The full page set is also stored at result_url as a JSON file for larger crawls.

Pricing

$0.005 per successfully crawled page. Pages that fail or return no content are not charged. A job is capped at your limit, so the most a single crawl can cost is limit × $0.005 (e.g. 200 pages ≤ $1.00).

Notes

• Respects each site's robots.txt; private/internal addresses are rejected.
• Need a single page instead? Use /v1/scrape.
• Subject to our Acceptable Use Policy.