Docs · AEO
AI bot allowlist audit
For each of 14 named AI crawlers, report whether your site's robots.txt allows or blocks it.
POST
/v1/aeo/ai-bot-allowlist Fetches the target site's /robots.txt and resolves it against 14 specific AI-crawler user-agents — GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-Web, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, Applebot-Extended, CCBot, Bytespider, Amazonbot, MistralAI-User. Returns per-bot allowed/blocked status with the matching rule reason, plus a summary count. This is the most-requested AEO audit deliverable in 2026 — concrete evidence of whether a site is discoverable by ChatGPT, Claude, Perplexity, and the rest.
Parameters
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| domain | string | yes | — | Domain or origin to inspect. example.com, https://example.com, or example.com/path all work — we always check /robots.txt. |
Request
curl -X POST https://api.qcrawl.com/v1/aeo/ai-bot-allowlist \
-H "Authorization: Bearer osk_..." \
-d '{"domain": "qcrawl.com"}'Response
{
"status": "success",
"domain": "qcrawl.com",
"robots_url": "https://qcrawl.com/robots.txt",
"robots_status": 200,
"results": [
{"bot": "GPTBot", "vendor": "OpenAI", "allowed": true, "explicit_rule": true, "reason": "explicit_bot_rules", "purpose": "Trains GPT models."},
{"bot": "ClaudeBot", "vendor": "Anthropic", "allowed": true, "explicit_rule": true, "reason": "explicit_bot_rules", "purpose": "Trains Claude models."},
...
],
"summary": {"total_bots_checked": 14, "explicitly_allowed": 14, "explicitly_blocked": 0, "implicit_default": 0}
}Errors
| Code | Meaning |
|---|---|
| 400 | Domain blocked by SSRF guard (private/internal target). |
| 500 | robots.txt could not be fetched (network error, non-200 status other than 404). |