Glossaries
Robots.txt

Robots.txt

Robots.txt is a file that tells search engine crawlers which pages to access or avoid.

Table of contents

Heading 2
Heading 3
Heading 4
Heading 5
Heading 6

Heading 2

Dolor enim eu tortor urna sed duis nulla. Aliquam vestibulum, nulla odio nisl vitae. In aliquet pellentesque aenean hac vestibulum turpis mi bibendum diam. Tempor integer aliquam in vitae malesuada fringilla.

Mi tincidunt elit, id quisque ligula ac diam, amet. Vel etiam suspendisse morbi eleifend faucibus eget vestibulum felis. Dictum quis montes, sit sit. Tellus aliquam enim urna, etiam. Mauris posuere vulputate arcu amet, vitae nisi, tellus tincidunt. At feugiat sapien varius id.

Heading 3

Eget quis mi enim, leo lacinia pharetra, semper. Eget in volutpat mollis at volutpat lectus velit, sed auctor. Porttitor fames arcu quis fusce augue enim. Quis at habitant diam at. Suscipit tristique risus, at donec. In turpis vel et quam imperdiet. Ipsum molestie aliquet sodales id est ac volutpat.

Tristique odio senectus nam posuere ornare leo metus, ultricies. Blandit duis ultricies vulputate morbi feugiat cras placerat elit. Aliquam tellus lorem sed ac. Montes, sed mattis pellentesque suscipit accumsan. Cursus viverra aenean magna risus elementum faucibus molestie pellentesque. Arcu ultricies sed mauris vestibulum.

Heading 4

Morbi sed imperdiet in ipsum, adipiscing elit dui lectus. Tellus id scelerisque est ultricies ultricies. Duis est sit sed leo nisl, blandit elit sagittis. Quisque tristique consequat quam sed. Nisl at scelerisque amet nulla purus habitasse.

Image caption goes here
Heading 5

Morbi sed imperdiet in ipsum, adipiscing elit dui lectus. Tellus id scelerisque est ultricies ultricies. Duis est sit sed leo nisl, blandit elit sagittis. Quisque tristique consequat quam sed. Nisl at scelerisque amet nulla purus habitasse.

"Ipsum sit mattis nulla quam nulla. Gravida id gravida ac enim mauris id. Non pellentesque congue eget consectetur turpis. Sapien, dictum molestie sem tempor. Diam elit, orci, tincidunt aenean tempus."
Heading 6

Nunc sed faucibus bibendum feugiat sed interdum. Ipsum egestas condimentum mi massa. In tincidunt pharetra consectetur sed duis facilisis metus. Etiam egestas in nec sed et. Quis lobortis at sit dictum eget nibh tortor commodo cursus.

Definition

Robots.txt provides directives for web crawlers about which parts of a site to crawl or ignore.

Why it matters

Improper settings can block important pages or expose sensitive ones.

Use Cases

Example: Disallowing /admin/ in robots.txt prevents bots from crawling admin pages.

Use Cases

Crawl Control

Guide bots on what to crawl.

Prevent Indexing

Block private or duplicate sections.

Optimize Crawl Budget

Focus bots on valuable content.

Validate file settings with tools.

Testing

Frequently Asked Questions

What is robots.txt?

A text file that sets crawler rules.

Can it block Google?

Yes, if misconfigured.

Where is it located?

At site.com/robots.txt.

Does it guarantee privacy?

With GSC or robots testing tools.

How to test robots.txt?

With GSC or robots testing tools.

Related Terms

Canonical Tag

Learn More

Semantic SEO

Learn More

Indexing API

Learn More

Content Pruning

Learn More