When should you use a robots.txt file?
Choose an answer
Tap an option to check your answer.
Correct answer: When you have pages that you don't want search engines to crawl and index.
Why this is the answer
A robots.txt file instructs search engines on which pages or sections of a website should not be crawled or indexed. It helps control crawler access to sensitive, duplicate, or non-public content. Proper use of robots.txt ensures that search engines focus on important pages and preserves crawl budget.
Pass your exam — without the endless answer hunt
Get every verified question and explanation for this exam in one place, and save hours of prep. 1,000+ certifications · 20+ languages · free to start.
Pass your exam faster → No card needed