Algorithm, Search & Distribution
Crawling
Also called: crawler, crawl, spider
Search engines run software, called crawlers or spiders, that follows links from page to page and downloads what it finds. Crawling is only the first stage. A page can be crawled and still never appear in results. Indexing is a separate decision about whether the content is worth storing and showing at all. The two stages are controlled differently, which trips people up. Block a URL in robots.txt and the crawl stops, so the engine never reads the noindex tag on that page and the URL can linger in results anyway. For creators it matters mostly for a website or blog, since Google crawls YouTube pages like any others.