Algorithm, Search & Distribution
AI crawler
Also called: GPTBot, Google-Extended, LLM crawler, AI bot
Two jobs hide behind one label. Training crawlers collect material to build models. Retrieval crawlers fetch a page at the moment somebody asks a question, so the answer can quote it and link to you. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are named user agents you can allow or refuse in robots.txt, and Google-Extended governs whether Google may use your content in its AI products rather than in search.
Blocking training does not block retrieval, and refusing both means an answer engine cannot cite you at all. One detail catches people out. Most of these bots never run JavaScript. A page that builds its content in the browser therefore reads as empty to them.