Robots.txt Blocking Stats Dataset
Most common directories blocked in robots.txt across sites.
Key Takaways

Based on 5k robots.txt files

Shows blocked sections

Highlights crawl restrictions

Useful for audits

Technical SEO relevance

CSV downloadable
Frequently Asked Questions
What’s included?
Blocked directories and % of sites blocking them.
Why is this useful?
It benchmarks common robots.txt practices.
Which directories are most blocked?
/search/, /filters/, /wp-admin/, /private/, /tmp/.
How should SEOs use this?
Ensure critical pages aren’t blocked.