Back to Glossary
FrontendBeginner Friendly
What is Robots.txt?
A file on a website that tells crawlers and bots which pages they are allowed or not allowed to access.
Why It Matters
Respecting robots.txt is important for ethical web scraping and ensuring your site controls what search engines index.
Real-World Example
A robots.txt file that blocks search engines from indexing your staging site or admin pages.
“Understanding terms like Robots.txt matters because it helps you have better conversations with developers and make smarter decisions about your software. You do not need to be technical. You just need to know enough to ask the right questions.”
From definition to deployment
Knowing the term is step one. Using it in something real is the rest.