# =============================================================================
# Robots.txt for Killerspots Agency
# Developed by Killerspots Agency | killerspots.com | KillerRank.ai
# Last Updated: September 2026
# =============================================================================
#
# Killerspots house standard. AI crawlers explicitly welcomed.
# Pure ASCII. Keep it that way.
#
# HOW TO EDIT THIS FILE WITHOUT BREAKING IT
# -----------------------------------------
# Consecutive "User-agent:" lines form ONE group. A named group completely
# REPLACES the "*" group for that bot. It does NOT inherit from it.
#
# So every Disallow has to be repeated inside every named group. If you add a
# crawler, add it to an existing stack. Do NOT give it its own
# "User-agent: X / Allow: /" block, because that block silently grants it
# access to everything the "*" group blocks.
#
# Two of the tokens below are NOT crawlers. Google-Extended and
# Applebot-Extended never make a request. They are robots.txt control tokens
# that govern whether content already crawled may be used for AI training.
# Listing them with "Allow: /" is how you opt IN. You will never see them in
# server logs, and that is not a bug.
# =============================================================================

# -----------------------------------------------------------------------------
# DEFAULT : everything open except machine endpoints and conversion pages
# Add client-specific Disallow lines here AND in both AI groups below.
# -----------------------------------------------------------------------------
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/

# -----------------------------------------------------------------------------
# AI TRAINING AND MODEL DEVELOPMENT : explicitly welcomed
# We want the client's entity, services and service area inside the models.
# Disallow lines repeated on purpose. See the note at the top.
# -----------------------------------------------------------------------------
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
User-agent: cohere-ai
User-agent: Bytespider
User-agent: DeepSeekBot
User-agent: Diffbot
User-agent: Timpibot
User-agent: ImagesiftBot
User-agent: omgili
User-agent: Omgilibot
Allow: /
Disallow: /admin/
Disallow: /api/

# -----------------------------------------------------------------------------
# AI SEARCH, ASSISTANT AND VOICE : the citation layer
# These decide whether the client gets named in an answer. This group matters
# more than the training group for lead flow.
#   Applebot        -> Siri and Spotlight
#   Bingbot         -> Copilot
#   OAI-AdsBot      -> validates ad landing pages (matters on any PPC account)
#   Claude-User     -> fires when someone asks Claude about this business
# Disallow lines repeated on purpose. See the note at the top.
# -----------------------------------------------------------------------------
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: Claude-Web
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Applebot
User-agent: Bingbot
User-agent: DuckAssistBot
User-agent: Google-CloudVertexBot
User-agent: meta-externalfetcher
User-agent: YouBot
User-agent: PhindBot
Allow: /
Disallow: /admin/
Disallow: /api/

# -----------------------------------------------------------------------------
# SITEMAPS
# -----------------------------------------------------------------------------
Sitemap: https://killerspots.com/sitemap-index.xml

# -----------------------------------------------------------------------------
# LLM CONTEXT
# There is no standard robots.txt directive for llms.txt, so these are comments.
# They document intent for humans. Crawlers find the files by convention at the
# site root, not from these lines. Do not mistake them for directives.
# -----------------------------------------------------------------------------
# LLM context: https://killerspots.com/llms.txt
# LLM context (full): https://killerspots.com/llms-full.txt