LLM fetch simulator
Fetch a URL as each of 11 named AI crawlers and a browser baseline. See what each one actually receives.
/v1/aeo/llm-fetch-simulator Sends 12 parallel HTTP requests to the same URL — each with a different User-Agent: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, MistralAI-User, plus a Chrome browser baseline. Compares status code, content length, and the first 500 chars of visible text each one received. Detects:
- JS-only SPAs — 200 OK with no visible text means LLMs see an empty shell
- Server-side blocking — 403/429 specific to one bot vendor
- Cloaking — bots receive materially different content than the browser baseline (a serving penalty risk)
11 named AI crawlers + 1 browser baseline = 12 parallel probes per call. Cloaking + JS-only detection from the browser-vs-bot text-length differential.
Does not simulate JavaScript execution (we are the bots — bots don't run JS). That's the entire point: if your page only works after JS, the bots get nothing, which is exactly what we want to surface.
Parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| url | string | yes | — | URL to probe with all 11 AI-crawler user agents. |
Request
curl -X POST https://api.qcrawl.com/v1/aeo/llm-fetch-simulator \
-H "Authorization: Bearer osk_..." \
-d '{"url": "https://example.com/your-cornerstone-page"}'Response
{
"status": "success",
"url": "...",
"bots_probed": 10,
"results": [
{"bot": "GPTBot", "vendor": "OpenAI", "status": 200, "content_length_bytes": 84221, "visible_text_chars": 3812, "received_full_content": true, "reason": null},
{"bot": "ClaudeBot", "vendor": "Anthropic", "status": 403, "received_full_content": false, "reason": "blocked_by_server"},
{"bot": "PerplexityBot", "vendor": "Perplexity", "status": 200, "content_length_bytes": 1842, "received_full_content": false, "reason": "js_only_spa"},
{"bot": "Browser (baseline)", "vendor": "control", "status": 200, "visible_text_chars": 3812, ...},
...
],
"summary": {
"all_bots_received_content": false,
"success_count": 7,
"blocked_count": 2,
"js_only_count": 1,
"cloaking_detected": false
}
}