Why each bot visits

Every bot in the directory is grouped by why it visits. The difference matters: blocking a training crawler is a policy choice, blocking an assistant answering a person loses you that answer.

Answering a person

Fetches a page right now because someone asked an assistant, such as ChatGPT-User.

AI search index

Builds the index AI answers are drawn from, such as OAI-SearchBot.

AI training

Collects content that may be used to train models, such as GPTBot.

AI agent

Browses and acts on a person's behalf.

Search engine

Classic crawlers like Googlebot and Bingbot, kept for comparison.

SEO tool, link preview, monitoring, other

Everything else that's known, so it isn't mistaken for AI.

See all 55 in the bot directory.