User-agent: *
Allow: /
Disallow: /api/
Disallow: /lot-finder/api/
# RUN67 Item 5a — every lot page mints its own /get-matched/ URL, carrying
# lf_lot=<subject uuid> for lead attribution. RUN 64 measured 1,967 distinct
# parameterised URLs already linked (one per Tier A lot page plus one per city
# hub) against 2,435 canonical URLs on the whole property, with a ceiling of
# 13,523 because every Tier B lot page mints one too. Google had already found
# 268 of them: the ENTIRE "Alternate page with proper canonical tag" bucket,
# 27% of everything it has declined to index on this domain.
#
# They are not an index problem — each returns 200 with a correct
# rel=canonical to /get-matched/ and Google has classified all 268 correctly.
# They are a crawl problem: each one must be fetched before the canonical can
# be read.
#
# THIS COSTS NOTHING, AND THAT WAS VERIFIED RATHER THAN ASSUMED. lf_lot and
# lf_city are read in exactly two places, both inside <script> blocks in
# src/pages/get-matched/index.astro (lines 292 and 466), both from
# `new URLSearchParams(location.search)` — the browser's own URL. Zero
# occurrences in the Astro frontmatter, which is the only part that runs on
# the server, and src/pages/api/lead.ts never reads either one. robots.txt is
# invisible to a browser, so attribution is untouched; only crawlers stop
# fetching the variants.
Disallow: /get-matched/?

# RUN 80 Item 5b — AhrefsBot and SemrushBot are allowed again.
# CRAWL-1 blocked them on 2026-08-05 because they fetched heavily and returned
# nothing; that measured 5,969 hits, and the block demonstrably worked (5,969
# before, 5 after, while GPTBot and ClaudeBot rose across the same boundary).
# The crawl-budget case is much weaker now: those 5,969 hits are a rounding
# error against 191,414 from a single Meta slug in 90 days. Reversed because a
# builder's agency runs this domain through one of these two before signing,
# and cannot tell "blocked" from "thin" — and because link-building measurement
# runs through their site-audit and internal-link-graph features, which require
# crawling us.
#
# The full rule set is RESTATED in each group deliberately. A named group
# REPLACES the wildcard (RFC 9309 2.2.1); it does not extend it. `Allow: /`
# alone here would have exempted both crawlers from every Disallow above.
User-agent: AhrefsBot
Allow: /
Disallow: /api/
Disallow: /lot-finder/api/
Disallow: /get-matched/?

User-agent: SemrushBot
Allow: /
Disallow: /api/
Disallow: /lot-finder/api/
Disallow: /get-matched/?

# ── RUN 80 Item 5c — the AI-crawler posture, reaffirmed 2026-08-26 ──────────
# Every AI crawler is allowed, training and citation alike, by the wildcard
# above. That is a decision, not an oversight, and it is the same one CRAWL-1
# made on 2026-08-05: AI-surface citation is a live acquisition channel for
# structured local content.
#
# The reasoning, restated so the next reader does not have to reconstruct it:
# this corpus is aggregated public record — DLI licence status, municipal
# ordinance text, county plat data. There is no moat in the content to protect
# by withholding it. The moat is the aggregation and the funnel, and no model
# reproduces either by training on the pages. Against that, being in the
# training corpus is the long-horizon brand play.
#
# THIS IS DELIBERATELY A COMMENT AND NOT A SET OF `User-agent` GROUPS.
# Under RFC 9309 2.2.1 a crawler obeys exactly one group — the most specific
# one naming it — and ignores every other group INCLUDING the wildcard. So
# adding `User-agent: GPTBot` + `Allow: /` as a documentation gesture would
# remove GPTBot from the wildcard and thereby DELETE its /api/,
# /lot-finder/api/ and /get-matched/? disallows. Naming an agent here is never
# free. If a future edit must name one, it has to restate every rule above,
# the way the Amazonbot, PetalBot, AhrefsBot and SemrushBot groups now do.
#
# Two tokens post-date the 2026-08-05 decision and are intentionally NOT named,
# for exactly that reason — both are covered by the wildcard:
#   Meta-WebIndexer  Meta's CITATION crawler. Distinct from meta-externalagent,
#                    which is the training one. Blocking the wrong one of that
#                    pair would cost the citation surface and keep the training.
#   Amzn-SearchBot   Amazon's search crawler, distinct from Amazonbot.

# RUN 99 Item 2 — meta-externalagent, Meta's AI-TRAINING crawler.
#
# THIS LINE IS THE STATED POLICY. THE ENFORCEMENT IS THE 403 IN
# infra/lot-finder-proxy (lot-finder repo, RUN 98). Meta's crawler was measured
# ignoring a Disallow on this property 20,136 times in RUN 80. DO NOT REMOVE THE
# WORKER BLOCK ON THE STRENGTH OF THIS LINE.
#
# Why it exists at all: without it the policy and the enforcement disagree on
# paper, and anyone reading robots.txt would conclude the crawler is welcome.
#
# WHY A BLANKET Disallow: / AND NOTHING ELSE. RFC 9309 2.2.1 — a named group
# REPLACES the wildcard for that agent; it does not extend it. A partial rule
# like `Disallow: /lot-finder/` would therefore silently GRANT this crawler
# everything the wildcard denies, including /api/ and /get-matched/?. A total
# disallow needs nothing restated and is the only safe shape here.
#
# NOT facebookexternalhit. That is the LINK-PREVIEW fetcher: it fires because a
# person shared a URL and is waiting for the card to render. It is never blocked
# and never rate-limited, by decision (RUN 82, reaffirmed RUN 98).
User-agent: meta-externalagent
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: Amazonbot
Crawl-delay: 10
Allow: /
Disallow: /api/
Disallow: /lot-finder/api/
Disallow: /get-matched/?

User-agent: PetalBot
Crawl-delay: 10
Allow: /
Disallow: /api/
Disallow: /lot-finder/api/
Disallow: /get-matched/?

Sitemap: https://www.homebuildersmn.com/sitemap-index.xml
Sitemap: https://www.homebuildersmn.com/lot-finder/sitemap.xml