robots-txt

When the user wants to configure, audit, or optimize robots.txt. Also use when the user mentions "robots.txt," "crawler rules," "block crawlers," "AI crawlers," "GPTBot," "allow/disallow," "disallow path," "crawl directives," "user-agent," "block Googlebot," "fix robots.txt," "robots.txt blocking," or "search engine crawling." For indexing, use indexing.

1,269
installs: 0 up / 0 down, score 0.00
votes: 1,058
repo stars
Updated: less than a month ago
In the catalog since: 5 Oct 2026
From: kostja94/marketing-skills

Get it through your agent npx skills add kostja94/marketing-skills@robots-txt

Skill instructions

Rendered from the published SKILL.md, 135 lines.

SEO Technical: robots.txt

Guides configuration and auditing of robots.txt for search engine and AI crawler control.

When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)

Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
Crawler access: Path-level crawl control; AI crawler allow/block strategy
Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions.

Initial Assessment

Project context: Read root contextus.md when present and load only the modules relevant to this task. Without Contextus, use available project material or user-provided facts and ask for missing information; do not create a parallel context system.

Identify:

Site URL: Base domain (e.g., https://example.com)
Indexing scope: Full site, partial, or specific paths to exclude
AI crawler strategy: Allow search/indexing vs. block training data crawlers

Best Practices

Purpose and Limitations

Point	Note
Purpose	Controls crawler access; does NOT prevent indexing (disallowed URLs may still appear in search without snippet)
Advisory	Rules are advisory; malicious crawlers may ignore
Public	robots.txt is publicly readable; use noindex or auth for sensitive content. See indexing

Crawl vs Index vs Link Equity (Quick Reference)

Tool	Controls	Prevents indexing?
robots.txt	Crawl (path-level)	No—blocked URLs may still appear in SERP
noindex (meta / X-Robots-Tag)	Index (page-level)	Yes. See indexing
nofollow	Link equity only	No—does not control indexing

When to Use robots.txt vs noindex

Use	Tool	Example
Path-level (whole directory)	robots.txt	`Disallow: /admin/`, `Disallow: /api/`, `Disallow: /staging/`
Page-level (specific pages)	noindex meta / X-Robots-Tag	Login, signup, thank-you, 404, legal. See indexing for full list
Critical	Do NOT block in robots.txt	Pages that use noindex—crawlers must access the page to read the directive

Paths to block in robots.txt: /admin/, /api/, /staging/, temp files. Paths to use noindex (allow crawl): /login/, /signup/, /thank-you/, etc.—see indexing.

Location and Format

Item	Requirement
Path	Site root: `https://example.com/robots.txt`
Encoding	UTF-8 plain text
Standard	RFC 9309 (Robots Exclusion Protocol)

Core Directives

Directive	Purpose	Example
`User-agent:`	Target crawler	`User-agent: Googlebot`, `User-agent: *`
`Disallow:`	Block path prefix	`Disallow: /admin/`
`Allow:`	Allow path (can override Disallow)	`Allow: /public/`
`Sitemap:`	Declare sitemap absolute URL	`Sitemap: https://example.com/sitemap.xml`
`Clean-param:`	Strip query params (Yandex)	See below

Path Format Rules (Critical)

Different directives use different path formats — a common source of errors:

Directive	Format	Correct	Wrong
Disallow / Allow	Root-relative path only (starts with `/`)	`Disallow: /admin/`	`Disallow: https://example.com/admin/`
Sitemap	Absolute URL only	`Sitemap: https://example.com/sitemap.xml`	`Sitemap: /sitemap.xml`

Wildcards: * matches any character sequence (Disallow: /tmp/*). $ marks exact URL ending (Allow: /news/.html$).

Priority: More specific paths take precedence. Allow: /shop/shoes/ overrides Disallow: /shop/. Path matching is case-sensitive: Disallow: /PDF/ does not match /pdf/.

Critical: Do Not Block

Do not block	Reason
CSS, JS, images	Google needs them to render pages; blocking breaks indexing
`/_next/` (Next.js)	Breaks CSS/JS loading; static assets in GSC "Crawled - not indexed" is expected. See indexing
Pages that use noindex	Crawlers must access the page to read the noindex directive; blocking in robots.txt prevents that

Only block: paths that don't need crawling: /admin/, /api/, /staging/, temp files.

AI Crawler Strategy

robots.txt is effective for all measured AI crawlers. Set rules per user-agent; check each vendor's docs for current tokens.

User-agent	Purpose	Typical	Notes
OAI-SearchBot	ChatGPT search	Allow	Respects robots.txt
GPTBot	OpenAI training	Disallow	Respects robots.txt; shares crawl data with OAI-SearchBot if both allowed
ChatGPT-User	User-initiated browsing	N/A	No longer respects robots.txt (Dec 2025); use server-side controls instead
Claude-SearchBot	Claude search	Allow	Respects robots.txt
Claude-User	Anthropic user-initiated browsing	Allow	Respects robots.txt (unlike ChatGPT-User)
ClaudeBot	Anthropic training	Disallow	Respects robots.txt

Deprecated: anthropic-ai — retired by Anthropic, replaced by ClaudeBot / Claude-User / Claude-SearchBot. References to anthropic-ai in robots.txt have no effect. | PerplexityBot | Perplexity search | Allow | Respects robots.txt | | Google-Extended | Gemini training | Disallow | Respects robots.txt | | CCBot | Common Crawl (LLM training) | Disallow | Respects robots.txt | | Bytespider | ByteDance | Disallow | Respects robots.txt | | Meta-ExternalAgent | Meta | Disallow | Respects robots.txt | | AppleBot | Apple (Siri, Spotlight); renders JS | Allow for indexing | Respects robots.txt |

Allow vs Disallow: Allow search/indexing bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot); Disallow training-only bots (GPTBot, ClaudeBot, CCBot) if you don't want content used for model training.

Important — ChatGPT-User exemption: As of December 2025, ChatGPT-User no longer respects robots.txt directives. OpenAI considers it a proxy for human-initiated browsing. If you need to block it, use server-side controls (WAF rules, IP rate-limiting), not robots.txt. See site-crawlability for AI crawler optimization (SSR, URL management).

Clean-param (Yandex)

Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content&ref&fbclid&gclid

Show the whole skill (15 more lines)

Output Format

Current state (if auditing)
Recommended robots.txt (full file)
Compliance checklist
References: Google robots.txt

Related Skills

indexing: Full noindex page-type list; when to use noindex vs robots.txt; GSC indexing diagnosis
page-metadata: Meta robots (noindex, nofollow) implementation
xml-sitemap: Sitemap URL to reference in robots.txt
site-crawlability: Broader crawl and structure guidance; AI crawler optimization
rendering-strategies: SSR, SSG, CSR; content in initial HTML for crawlers

Get it through your agent

Call the Graph-of-Skills MCP server's retrieve_skill_bundle tool and it returns this skill's text inside your budget.

{
  "tool": "retrieve_skill_bundle",
  "arguments": {
    "query": "robots-txt"
  }
}

New to this? Set up Graph-of-Skills or get a free key. The text on this page comes from its authors: how skills are sourced and what we check. If none fits your task, your agent can write a skill for your own library. To keep a skill from your own library on a machine, see how to install one of your skills.

Share on X Share on LinkedIn Share on Bluesky Share on Hacker News

Share on Reddit

Source

Source repository kostja94/marketing-skills
Listing on skills.sh

From kostja94/marketing-skills via skills.sh

Related skills

technical-seo-checker
Use when the user asks to "check technical SEO"; audits crawlability, indexing, Core Web Vitals, robots.txt, sitemaps, canonicals, redirects, and migrations. Not for on-page tags or content — use on-page-seo-checker. 技术SEO/网站速度
1,513 installs
seo-technical
Audit technical SEO across crawlability, indexability, security, URLs, mobile, Core Web Vitals, rendering, structured data, and IndexNow. Use when the user asks for a technical SEO audit or about crawling, indexing, rendering, or site speed. Exclude content strategy and backlinks.
7,097 installs
ai-seo
When the user wants to optimize content for AI search engines, get cited by LLMs, or appear in AI-generated answers. Also use when the user mentions 'AI SEO,' 'AEO,' 'GEO,' 'LLMO,' 'answer engine optimization,' 'generative engine optimization,' 'LLM optimization,' 'AI Overviews,' 'optimize for ChatGPT,' 'optimize for Perplexity,' 'AI citations,' 'AI visibility,' 'zero-click search,' 'how do I show up in AI answers,' 'LLM mentions,' 'optimize for Claude/Gemini,' 'llms.txt,' 'llms-full.txt,' 'OKF,' 'Open Knowledge Format,' 'knowledge bundle,' 'agent-readable site,' 'agent readiness,' 'is my site agent-ready,' 'WebMCP,' 'do listicles still work for AI,' 'ChatGPT stopped citing comparison pages,' 'how do LLMs see our brand,' 'LinkedIn for AEO,' or 'AI citation format shift.' Use this whenever someone wants their content to be cited or surfaced by AI assistants and AI search engines. For traditional technical and on-page SEO audits, see seo-audit. For structured data implementation, see schema.
149,438 installs
seo-geo
Audit and improve content for AI Overviews and answer engines, including citability, entity clarity, crawler access, brand signals, and passage structure. Use when the user asks about AI Overviews, AI Mode, ChatGPT, Perplexity or Gemini citations, GEO, or AI search visibility.
7,098 installs
blog-geo
Audit one blog post file for AI-citation readiness (self-contained, sourced passages, entity clarity, structure, crawler access) and score it with an internal 0-100 heuristic plus per-surface notes for AI Overviews, AI Mode, ChatGPT, Perplexity, Claude, Gemini, and Copilot. Use when the user says "geo", "aeo", "ai citation", "citation audit", "chatgpt citation". Not for a site-wide audit, live AI-visibility tracking, or schema generation (use claude-seo seo-geo, blog-audit, blog-schema), and not a promise of citations.
2,557 installs

More from kostja94/marketing-skills

copywriting
When the user wants to write or optimize short-form marketing copy—headlines, CTAs, ad copy, landing page copy, email copy. Also use when the user mentions "copywriting," "headline," "CTA copy," "ad copy," "landing page copy," "sales copy," "conversion copy," "PAS," "AIDA," "BAB," "copy formula," or "differentiation." For long-form article bodies (blog posts, guides), use article-content.
3,896 installs
linkedin-posts
When the user wants to create LinkedIn post copy or optimize for LinkedIn. Also use when the user mentions "LinkedIn post," "LinkedIn article," "professional post," "post to LinkedIn," "LinkedIn content," "LinkedIn copy," "B2B LinkedIn," "LinkedIn engagement," "LinkedIn feed," "share box," "document post," "poll," "Newsletter," "reshare," or "LinkedIn marketing." For LinkedIn ads, use linkedin-ads.
3,561 installs
youtube-seo
When the user wants to optimize YouTube videos for search, create video descriptions, or improve channel visibility. Also use when the user mentions "YouTube SEO," "YouTube description," "YouTube tags," "YouTube thumbnail," "YouTube title," "YouTube channel," or "video SEO." For YouTube ads, use youtube-ads.
3,374 installs
video-marketing
When the user wants to plan video marketing, create video scripts, or optimize for short-form or long-form video. Also use when the user mentions "video marketing," "video script," "short-form video," "long-form video," "TikTok script," "Reels script," "YouTube script," "video hook," or "video content strategy." For on-site video SEO, use video-optimization.
3,080 installs
tiktok-captions
When the user wants to create TikTok video captions, scripts, or optimize for TikTok. Also use when the user mentions "TikTok post," "TikTok caption," "TikTok video," "post to TikTok," "TikTok script," "TikTok content," "TikTok copy," "TikTok hashtags," or "TikTok marketing." For TikTok ads, use tiktok-ads.
2,776 installs

Content is provided as published by its creators; you are responsible for how you use it.

To request removal, use the contact form.