What is Robots.txt?
A file on a website that tells crawlers and bots which pages they are allowed or not allowed to access.
Why It Matters
Respecting robots.txt is important for ethical web scraping and ensuring your site controls what search engines index.
Real-World Example
A robots.txt file that blocks search engines from indexing your staging site or admin pages.
“Understanding terms like Robots.txt matters because it helps you have better conversations with developers and make smarter decisions about your software. You do not need to be technical. You just need to know enough to ask the right questions.”
Learn More at buildDay Melbourne
Want to understand these concepts hands-on? Join our one-day workshop and build a real web application from scratch.
Related Terms
SEO
Techniques to help your website appear higher in search engine results
Sitemap
A file that lists all the pages on your website to help search engines find and index them
Web Scraping
Automatically extracting data from websites using code that reads and processes web page content.
Frontend
The part of a website or app that users see and interact with directly
React
A popular tool for building user interfaces, created by Facebook, that makes complex UIs easier to manage
Next.js
A framework built on React that adds features like page routing, server-side rendering, and API handling