🎉 Limited time — 20% off all plans. View pricing →

Fire jobs. Get a clean callback.

Queue a hundred URLs or a hundred thousand. Walk away. Your endpoint receives a single signed POST when the work completes. No polling, no held connections, no retry plumbing on your side. 

# Send a batch of 100,000 URLs, walk away.
curl -X POST https://api.qcrawl.com/v1/scrape/batch/async \
  -H "Authorization: Bearer osk_..." \
  -d '{
    "urls": [...],
    "format": "markdown",
    "webhook_url": "https://yourapp.com/qcrawl-callback"
  }'

# Response: { "status": "queued", "job_id": "..." }
# We POST the full result to your URL when ready.

Fire-and-forget at any size

Queue a job of any size — a hundred URLs or a hundred thousand — and walk away. Your server gets a single clean POST when the work is done. No long-held connections, no polling loops, no retry plumbing on your side.

Every webhook is signed

We sign every callback with a cryptographic signature derived from your account's webhook secret. Your endpoint verifies the signature before trusting the payload. Forgery becomes infeasible; replay attacks become traceable.

Automatic retries with backoff

If your endpoint returns a non-2xx response, we retry with exponential backoff — up to five attempts over 24 hours. Transient outages on your side don't lose data. Permanent failures land in a dead-letter inspection view on the dashboard.

Replay from the dashboard

Every webhook delivery is logged with the request payload, your response code, and the delivery latency. Replay any delivery from the dashboard with one click — useful for testing changes to your endpoint without re-running the original job.

Production patterns

Massive corpus build for LLM training

Queue 500,000 URLs, get a webhook when the batch finishes. No infrastructure to keep open, no progress dashboards to babysit.

Overnight competitor crawls

Schedule full-site crawls of 20 competitors. The webhook delivers each completed crawl to your ingestion pipeline as it finishes — no polling required.

Pipeline integration

Wire webhooks directly into Zapier, n8n, Pipedream, or your custom event bus. Treat scrape jobs as another event source in your existing automation stack.

Mobile and serverless backends

Apps that can't hold long-lived connections — mobile clients, edge functions, serverless workers — fire async jobs and receive results when ready. Works around platform timeout limits cleanly.

Webhook questions

Which endpoints support webhooks? ▾
/v1/scrape/async, /v1/scrape/batch/async, and /v1/crawl. All return a job_id immediately and POST to your webhook_url when the work completes.
How are webhooks signed? ▾
Each request carries an X-Qcrawl-Signature header in the form t=<unix_ts>,v1=<hex>. The v1 value is an HMAC-SHA256 of the bytes <ts>.<raw_body> keyed by your account's webhook_secret. Verification code samples above show the full algorithm; compare in constant time and reject timestamps older than five minutes to defeat replay attacks.
Where do I get my webhook secret? ▾
Call GET /v1/me with your API key and read the webhook_secret field (format whsec_<hex>). Rotate at any time via POST /v1/me/webhook-secret/rotate — the old secret stops working immediately, so drain in-flight jobs first.
What's the retry policy? ▾
If your endpoint returns non-2xx or the connection errors, we retry up to three times with exponential backoff (1s, 2s, 4s) within the same delivery attempt. After three failures the job result still contains a _webhook field indicating delivery failure, and a future release adds dashboard-driven manual replay for failed deliveries.
How long is the timeout for my endpoint? ▾
15 seconds per attempt. If your endpoint hasn't returned a status code in 15 seconds, we treat it as a failure and queue the next retry. For long downstream processing, acknowledge fast with 200 and process asynchronously on your side.
Can I use webhooks without HTTPS? ▾
No. Webhook URLs must be HTTPS. We reject configuration of plain HTTP endpoints at job-creation time.
Do you also offer polling instead of webhooks? ▾
Yes. If a webhook is impractical for your environment, omit webhook_url and poll /v1/jobs/{id} for completion. Webhooks are recommended for production because they eliminate the constant polling overhead.
What payload shape do webhooks deliver? ▾
{ job_id, result }. The result block matches what you would get from the corresponding sync endpoint. The job_id is also echoed in the X-Qcrawl-Event-ID header for fast routing.

Ship production-grade async today.

Webhooks included on every plan, including the free tier.

Start free Talk to sales