robots.txt

Sits at the root of a domain and lists rules grouped by user agent, each one allowing or disallowing a path. Obedience is voluntary. Well-behaved crawlers read it and comply, badly-behaved ones ignore it entirely, so it is a signpost rather than a lock. It also controls crawling instead of indexing: a disallowed URL can still show up in search results if other sites link to it, and stopping that needs a noindex tag or a login. Site owners now use it to name AI training and answer-engine crawlers specifically, opening or closing their content to each one.