Fill in the blank: The primary function of a robots.txt file is_______________.

The best fit is robots.txt as a crawl-control file, because its core role is to guide crawler access before indexing decisions are made. In HubSpot’s SEO documentation, robots.txt is the file used to tell search engines how they should handle crawling for site content, while noindex is the stronger mechanism for removing already indexed pages from search results. That means the file’s primary purpose is managing crawler behavior rather than controlling final search-result visibility with precision. In practice, this supports crawl efficiency and avoids unnecessary crawler requests on content that should not be fetched. HubSpot