LinkGraph

Robots.txt validator

Test pasted robots.txt rules and understand which paths crawlers can access.

Check my robots.txt

The companies you trust, trust LinkGraph

Zynga LogoInternational Olympic Committee LogoShutterfly LogoP & G logo on a white background for legal services.Verkada Logo

Check your robots.txt rules

See which rule applies before you publish a change. Paste the file, choose a crawler and test a path.

Browser-only check

Copy the plain-text file from your website’s /robots.txt address. This tool does not fetch your website.

Use the file for the same host, protocol and port. Only the path and query are tested.

For example, Googlebot or Bingbot. Enter one token, not a full browser user-agent string.

The checker processes these inputs locally. It does not upload them or change your robots.txt file.

From file to a clear decision

A focused check of crawl rules, with the matching line and practical next steps.

01

Use the right file

Copy robots.txt from the root of the exact website you are checking. HTTP, HTTPS and different subdomains can have different files.

02

Test important paths

Check public pages, private directories and exceptions. Try the path both with and without a trailing slash, and include any relevant query string.

03

Review, then publish

Read the deciding rule and line warnings. Update the file through your own hosting workflow, then verify the live response and your search engine tools.

Robots.txt questions

Need a broader technical SEO review?

Discuss crawlability, site structure and your next steps with the LinkGraph team.

Schedule a Consultation

Technical references: Robots Exclusion Protocol (RFC 9309) and Google’s robots.txt documentation