Check if AI crawlers can read your site.
ChatGPT search, Perplexity and Claude now send real traffic, and many sites block them by accident. One run tells you which AI bots your robots.txt shuts out, whether you have llms.txt and schema.org data, and what to fix first, page by page.
https://www.example-outdoor.co.uk
| Crawler | Used for | robots.txt |
|---|---|---|
| OAI-SearchBot | ChatGPT search | Allowed |
| PerplexityBot | Perplexity answers | Blocked |
| Claude-SearchBot | Claude search | Blocked |
| Google-Extended | Gemini | Allowed |
| GPTBot | Model training | Blocked |
View the JSON record
{
"type": "site",
"site": "https://www.example-outdoor.co.uk",
"aiReadinessScore": 64,
"averagePageScore": 74,
"pagesAudited": 10,
"aiCrawlersBlocked": ["PerplexityBot", "Claude-SearchBot", "GPTBot"],
"llmsTxt": false,
"sitemap": "https://www.example-outdoor.co.uk/sitemap.xml",
"sitemapUrlCount": 214,
"recommendations": [
"Add /llms.txt: a short Markdown guide to your key pages for AI assistants",
"Allow AI answer engines in robots.txt (PerplexityBot, Claude-SearchBot are blocked) if you want to be cited in AI search",
"Fix on 6 page(s): No schema.org structured data"
]
}Who it's for
- SEO and marketing agencies
- Run it across every client site each month and send the fix list. Site summaries are free, so you pay only for pages audited.
- Site owners and content teams
- Find out whether a robots.txt rule or a CDN bot setting is hiding you from AI search, before your competitors are the ones getting cited.
- Developers
- Add it to CI or a weekly schedule to catch an accidental noindex or a newly blocked bot after a deploy.
How it works
- Your own site, read politely
- It reads your robots.txt, llms.txt and sitemap, then crawls same-site pages one at a time and obeys robots.txt for its own user agent.
- No personal data
- It records page structure and SEO signals: titles, headings, schema types, status codes. Not form contents, not visitor data.
- Pay per page
- You're charged per page audited. The site summary, robots.txt, llms.txt and sitemap checks are included free.
- Re-check after changes
- Save the task and schedule it weekly. Compare scores between runs to confirm a fix worked or catch a regression.
Pricing
Billed by Apify per page audited. Higher Apify plans pay less per page automatically.
| Apify plan | Per 1,000 pages |
|---|---|
| Starter | $10.00 |
| Scale | $7.50 |
| Business | $5.50 |
A 10-page first audit costs 10 cents. On the free Apify plan you can audit up to 50 pages per run at the Starter price.
API and integrations
Call it from any language over HTTP, or wire it up without code.
- Weekly schedules and saved tasks in the Apify console
- Slack, email, Google Sheets, Zapier, Make, n8n and webhooks
- The Apify MCP server, for AI agents
- A CI step that fails the build when the score drops
curl -X POST \
"https://api.apify.com/v2/acts/locaihost~site-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"urls": ["https://example.com"],
"maxPagesPerSite": 10
}'Questions
Which AI crawlers does it check?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot and Bytespider. Each is reported as allowed or blocked by your robots.txt.
Will blocking AI training crawlers lower my score?
No. Blocking training crawlers such as GPTBot or CCBot is a legitimate choice and does not affect the score. Only the AI answer engines that cite and link to you (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) count towards it.
How is the AI-readiness score calculated?
60% comes from the average page score, plus 10 points each for an llms.txt file, an XML sitemap, schema.org data on the homepage, and access for the AI answer engines.
Does it render JavaScript?
Not yet. Pages are analysed as your server delivers them, which is also what most AI crawlers see.
Is the crawl polite?
Yes. It respects robots.txt for its own user agent (locaihost-site-audit), fetches one page at a time per site with a short pause, and only follows same-site links. It is built for sites you own or manage.
Find out what AI search sees.
Enter your domain, keep the 10-page default, and get a score and a fix list back.