robots.txt is the oldest agreement between websites and crawlers. It is simply structured and still a frequent cause of SEO problems, because it is often mistaken for an indexing tool.
Structure
The file sits at /robots.txt and consists of groups. Each group starts with one or more User-agent lines specifying which crawlers it applies to, followed by Disallow and Allow rules for paths. It can also reference the XML sitemap. Since 2022 the protocol has been standardised as RFC 9309; the major search engines follow it.
Crawling is not indexing
A Disallow rule prevents a crawler from fetching a page. It does not prevent the URL from appearing in the index if other pages link to it — then often without a description. To keep a page out of the index, leave it fetchable and mark it noindex. Google has not supported a noindex directive in robots.txt itself since 2019. The difference is described under indexing.
Controlling AI crawlers
Separate user-agent groups let you allow or exclude AI crawlers specifically, separately for training, search and user fetches where providers use distinct identifiers. For visibility in AI answers, that is a deliberate decision, not a default.
Common mistakes
A site-wide block carried over from the development server. Blocked CSS and JavaScript files that prevent rendering. Pages marked noindex that are also blocked via Disallow, so the crawler never sees the noindex. And blocks on areas that are actually meant to rank. After every change, checking the robots.txt report in Search Console pays off.
In Next.js
In Next.js, robots.txt can be generated as code, for example with a robots.ts in the app directory. That allows different rules per environment, so previews are blocked and production is open, and a maintained list of user agents. Reviewing the overall configuration is part of our technical SEO audit.