How to Create an Effective robots.txt File (With Examples)

A robots.txt file tells search engine crawlers which URLs on your site they’re allowed to access. It sits at your domain’s root (yoursite.com/robots.txt) and is the first file every crawler checks before crawling your site.

robots.txt Syntax

The format is simple: you specify a user agent (the crawler) and rules for what it can or can’t access.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://www.example.com/sitemap.xml

Directives

Directive	What It Does
User-agent	Which bot the rules apply to (* means all bots)
Allow	Explicitly allows access to a path
Disallow	Blocks access to a path
Sitemap	Tells crawlers where your sitemap is
Crawl-delay	Requests bots wait N seconds between requests (not respected by Google)

Pattern Matching

You can use * (wildcard) and $ (end of URL) for pattern matching:

# Block all URLs containing "?print"
Disallow: /*?print

# Block all PDF files
Disallow: /*.pdf$

# Block all URLs in any /temp/ directory
Disallow: /*/temp/

Common robots.txt Examples

Basic Blog or Content Site

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /wp-login.php
Disallow: /search/

Sitemap: https://www.myblog.com/sitemap.xml

Ecommerce Site

User-agent: *
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /search/
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://www.mystore.com/sitemap.xml

SaaS Application

User-agent: *
Allow: /
Allow: /blog/
Allow: /docs/
Disallow: /app/
Disallow: /dashboard/
Disallow: /api/
Disallow: /internal/

Sitemap: https://www.myapp.com/sitemap.xml

Development or Staging Site

# Block everything — this site shouldn't be indexed
User-agent: *
Disallow: /

robots.txt Rules to Know

1. More Specific Rules Win

When Allow and Disallow rules conflict, the more specific (longer) rule wins:

User-agent: *
Disallow: /blog/drafts/
Allow: /blog/

This allows /blog/ but blocks /blog/drafts/. The longer path takes priority.

2. Bot-Specific Rules Override Wildcard Rules

User-agent: *
Disallow: /private/

User-agent: Googlebot
Allow: /private/special-page

Googlebot can access /private/special-page, but all other bots are blocked from /private/.

3. robots.txt Blocks Crawling, Not Indexing

If a page is blocked by robots.txt but has inbound links, Google may still index the URL — it just won’t crawl the content. The search result will show the URL with no description.

To prevent indexing, use the noindex meta tag instead:

<meta name="robots" content="noindex" />

4. Empty Disallow Means Allow

User-agent: *
Disallow:

An empty Disallow directive allows everything. This is equivalent to not having a robots.txt at all.

Where to Put robots.txt

Place it at the root of your domain:

https://www.yoursite.com/robots.txt

Not in a subdirectory, not at a different path. Crawlers only look at the root.

For static sites (Astro, Next.js, Hugo): Put it in your public/ directory.

For WordPress: Edit it through Yoast SEO settings or upload directly via FTP.

For Shopify: Shopify auto-generates robots.txt. You can customize it through your theme’s robots.txt.liquid file.

Common robots.txt Mistakes

1. Blocking Your Entire Site

User-agent: *
Disallow: /

This blocks all crawlers from your entire site. This is correct for staging sites but catastrophic for production — your site will disappear from search results entirely.

2. Blocking CSS and JavaScript

Disallow: /css/
Disallow: /js/

Google needs to render your pages to understand them. Blocking CSS and JS prevents rendering, which can hurt your rankings significantly.

3. Blocking Images

Disallow: /images/

This prevents your images from appearing in Google Image Search, which can be a significant traffic source.

4. Conflicting with Canonical Tags

If you block a URL in robots.txt but it’s the canonical URL for other pages, search engines can’t access the canonical — creating a conflict with no good resolution.

5. No Sitemap Reference

Always include your sitemap URL:

Sitemap: https://www.yoursite.com/sitemap.xml

This is the most reliable way to ensure search engines find your sitemap.

6. Forgetting robots.txt Exists

Changes made months or years ago can quietly block important content. Review your robots.txt at least quarterly.

How Do I Check My robots.txt?

Visit yoursite.com/robots.txt in your browser to see if one exists. Google Search Console also has a robots.txt tester that shows you how Google interprets your rules.

What seoutils.dev checks: Our free audit validates your robots.txt for accessibility, syntax correctness, sitemap references, and overly broad Disallow rules that might be blocking important content.

Run a free audit to check your robots.txt configuration.

---

This is part of our Complete SEO Audit Checklist. Related: XML Sitemap Best Practices | How to Manage AI Crawlers in robots.txt.