How Qcrawl is built.
A plain-language tour of the platform — what each subsystem does, what we promise it will and will not do, and how the pieces stay consistent under load.
Operational promises
The subsystems
Qcrawl is one API on the outside and eight subsystems on the inside. Each one has a single responsibility, a clear contract with the others, and is scaled independently.
Gateway
Every request lands at the Gateway. It authenticates the API key, applies per-minute rate limits and per-month quotas, charges credits atomically, and routes the call to the right subsystem. Failed lookups, malformed input, and expired keys are answered here before any real work begins. Predictable behaviour at the edge means quotas never undercharge and never double-charge — the credit ledger is consistent under load.
Renderer
Pages that need JavaScript go through the Renderer — a stealth browser engine that drives clicks, scrolls, form fills, and waits from a single JSON action macro. Static pages skip the Renderer entirely via the smart-routing path and return in milliseconds. Sites with strong anti-bot defences pair the Renderer with a residential proxy you supply.
Extractor
Once a page is fetched, the Extractor turns raw HTML into the shape you asked for: clean readability text, JSON-LD and Microdata, structured product or property data, or LLM-ready markdown. The same Extractor stages power the standalone /v1/extract/* endpoints, so the output you get from a scrape matches the output you get from a direct extraction call.
Intelligence layer
DNS, WHOIS, SSL certificates, IP geolocation, technology stack detection, contact harvesting, sitemap unrolling, and email verification ship as one integrated layer. Local database lookups (IP geolocation, ASN, postal) return in milliseconds with no third-party call. Live probes (DNS, SSL handshakes, SMTP verification) hit the target directly with predictable timeouts.
Job queue
Long-running work — async scrapes, async batches, full-site crawls — is handed off to a queue and worked by a pool of background processes. The API returns a job_id immediately. Results arrive at your webhook on completion, or you can poll the job endpoint. Workers retry transient failures and never silently drop work — every job ends in completed, failed, or expired.
Storage & ledger
Three concerns are kept in three stores: account state and usage history live in a transactional database, hot rate-limit counters live in an in-memory store with sub-millisecond reads, and Stripe billing events are written through an idempotency table so a duplicate webhook can never double-charge. Nightly backups of the transactional store are written off-box.
Billing & metering
Every successful call writes one row to the usage log and decrements the credit ledger atomically — the same transaction. Stripe is the source of truth for plan changes: a webhook from Stripe is the only path that ever upgrades, downgrades, or extends a subscription, and every event is verified by signature and recorded once. Free, paid, and internal plans share the same code path; the difference is one column.
Observability
Every request is structured-logged with the user, endpoint, status, latency, and credit charge. Errors are sampled into an error-tracking service. A health probe checks the full request path every two minutes and pages on three consecutive failures. The customer-facing dashboard exposes request count, average latency, success rate, and remaining credits — the same numbers operations sees internally.
Design principles
One API key, one Bearer header
Every endpoint authenticates the same way. The key you got at signup works for scraping, extraction, intelligence, and billing endpoints with no extra setup.
Fail closed where it matters, fail open where it does not
Rate limiting on the free tier fails closed if the limiter is unreachable — abuse cannot leak through an outage. Paid-tier limiters fail open so a transient hiccup never blocks a paying customer mid-pipeline.
Charge once, never silently
Credits are deducted in the same database transaction as the call decision. Server-side failures (database unavailable, downstream timeout) refund automatically. Customer-side failures (invalid input, private IPs) do not — those are still requests we served.
Idempotent webhooks both ways
Inbound webhooks (Stripe) are deduplicated by event ID — Stripe will retry, and we acknowledge every retry without re-applying the change. Outbound webhooks (job completion) are signed with HMAC so your server can verify and dedupe.
Local data, no third-party tax
Where a lookup can be served from a maintained local dataset — IP geolocation, ASN, robots.txt parsing — we do that. Faster, cheaper, no rate limit propagating back to your pipeline.
Same code, every plan
Free, Pro, Business, and Enterprise share one code path. The differences are quota numbers and concurrency limits, not feature flags. That is why feature parity is automatic across plans.
The request lifecycle
What actually happens between the moment you send a request and the moment you get a response.
- 1. Authentication. The Gateway checks your API key against its hashed copy. Invalid keys are rejected before any work is scheduled.
- 2. Quota and rate limit. Per-minute and per-month limits are checked atomically. Headers on every response tell you how much you have left.
- 3. Credit reservation. The credit ledger is decremented in the same transaction as the call decision. Two simultaneous requests with one credit left will see exactly one succeed.
- 4. Routing. Synchronous endpoints execute inline. Async endpoints register a job and return a job_id under 50 ms.
- 5. Execution. Smart-routing decides between the fast HTTP path and the Renderer. Action macros, captcha solving, and proxy handling happen here.
- 6. Extraction. Raw HTML is shaped into your requested format. Structured-data endpoints emit JSON-LD, OpenGraph, and Microdata in a single pass.
- 7. Webhook delivery. Async results are POSTed to your URL with an HMAC signature. Failed deliveries retry with exponential backoff.
- 8. Logging. The request, latency, status, and credit charge are written to your usage log. The dashboard surfaces this in real time.
Architecture questions
Does Qcrawl scale to production volume? ▾
What happens when a downstream site is slow or fails? ▾
How do you keep the actor catalog accurate when target sites change? ▾
Where are requests processed? ▾
What is your uptime target? ▾
Do you store the pages I scrape? ▾
How is data protected in transit and at rest? ▾
See it work on your data.
1,000 free requests a month, no credit card. The same architecture serves every plan.