purpose documented across crawlers
purpose documented across 15 crawlers from 6 vendors, from 15 claims, last verified 2026-09-14.
How the crawlers answer this one question, counted by answer. Every answer links to the vendor quote behind it.
| Crawler | Vendor | Value | Method | Verified | Claim |
|---|---|---|---|---|---|
| Applebot | Apple | mixed | vendor_doc | 2026-09-13 | claim 13 |
| Applebot-Extended | Apple | policy_only | vendor_doc | 2026-09-13 | claim 5 |
| Bingbot | Microsoft | yes | vendor_doc | 2026-09-14 | claim 125 |
| ChatGPT-User | OpenAI | user_fetch | vendor_doc | 2026-09-13 | claim 23 |
| Claude-SearchBot | Anthropic | search | vendor_doc | 2026-09-13 | claim 32 |
| Claude-User | Anthropic | user_fetch | vendor_doc | 2026-09-13 | claim 42 |
| ClaudeBot | Anthropic | training | vendor_doc | 2026-09-13 | claim 51 |
| Google-Extended | policy_only | vendor_doc | 2026-09-13 | claim 59 | |
| Googlebot | search | vendor_doc | 2026-09-13 | claim 67 | |
| GoogleOther | mixed | vendor_doc | 2026-09-13 | claim 75 | |
| GPTBot | OpenAI | training | vendor_doc | 2026-09-13 | claim 82 |
| OAI-AdsBot | OpenAI | ads | vendor_doc | 2026-09-13 | claim 90 |
| OAI-SearchBot | OpenAI | search | vendor_doc | 2026-09-13 | claim 98 |
| Perplexity-User | Perplexity | user_fetch | vendor_doc | 2026-09-13 | claim 107 |
| PerplexityBot | Perplexity | search | vendor_doc | 2026-09-13 | claim 116 |
Not documented
Every documented crawler has a claim for this field.