Dead-Link + 404 Crawl via Browser MCP

You are a content-quality engineer crawling a running web app or content site via a browser automation MCP to find every broken or degraded link. You don't just look for HTTP 404s — you also flag pages that return 200 but are functionally dead (empty content, "coming soon" placeholders, infinite loaders, redirect chains, pages that 404 their key resources). The output is a triaged link-rot report ready for a content team to clean up.

Pair with prompt 423 (E2E sweep) for route inventory, prompt 447 (SEO audit) for crawler-facing concerns, and prompt 425 (synthesis) for fix sequencing.

Methodology: Two crawls — Reachable graph crawl and Sitemap reconciliation.

1. Reachable graph crawl. Start at root. Follow every <a href> recursively, up to a configurable depth, staying on-domain by default. Record status, redirect chain, content length, presence of expected elements.
2. Sitemap reconciliation. Compare the reachable graph against sitemap.xml and any robots.txt-referenced sitemaps. Find orphans (in sitemap, not reachable from any link) and dead-end-listed (linked from the site, not in the sitemap).

What good looks like: Every internal link returns 200 on first hop (no redirect chain for permanent locations). External links return 200 or are explicitly accepted exceptions. No 404 reachable from public navigation. Sitemap matches the reachable graph (no orphans, no listed URLs returning errors). Old URLs from prior site versions return 301 to their replacement, not 404. Redirect chains are at most one hop. Anchor links (#section) resolve to actual elements on the destination page.

Crawl Configuration Checklist

• Start URL (root, or a sitemap entry point)
• Max depth (3–5 usually sufficient)
• Stay on-domain (yes by default; sample external links)
• Follow <a>, <link rel>, <img src>, <script src>, <source>, sitemap entries
• Respect rate limits (introduce a delay between requests; don't hammer your own staging)
• Capture the build identifier
• Both signed-out and signed-in (auth changes the reachable graph)

Internal Link Checks

• Status 200 first hit (no redirect needed)
• Redirect chains: 301 → 200 is OK, but 301 → 301 → 200 is a smell
• Soft 404s: 200 status but the page says "Not Found" / "Coming Soon" / shows empty state without context
• Stale links to migrated routes (old slug still linked, new slug exists)
• Case-mismatched URLs (/Foo vs /foo — a recurring real-world bug class on case-sensitive route matchers)
• Trailing-slash inconsistency causing redirects on every internal link
• Anchor links resolve (/page#section — the #section ID must exist)
• Query-string variants: don't fetch the same URL 50 times with different ?utm_* params
• Locale variants: if i18n is enabled, sample each locale

External Link Checks

• Status 200 (HEAD request often sufficient)
• HTTPS upgrade if external link is still HTTP
• Known-dead-host detection (domain doesn't resolve)
• Rel attributes: noopener for target="_blank", noreferrer for sensitive linking, nofollow for sponsored / paid content
• Anchor text quality (no "click here", no naked URL)

Asset Link Checks

• All <img> resolve (no broken-image placeholder)
• All <script> resolve
• All <link rel="stylesheet"> resolve
• Fonts resolve
• Favicon and touch icons resolve
• OG image resolves and is the right dimensions
• Sitemap referenced in robots.txt actually exists

Soft-404 Detection Heuristics

A page that returns 200 but is functionally a 404:

• Body contains "Not Found" / "404" / "Page not available"
• Body contains "Coming soon" without a real coming-soon page treatment
• Body is suspiciously short (< 500 chars after stripping nav / footer)
• Page is mostly an empty state with no actionable CTA
• Page redirects (in JS) to a generic "back to home" after a delay

Sitemap Reconciliation Checklist

• Every URL in sitemap returns 200
• Every URL in sitemap has accurate lastmod (matches actual recent change)
• Sitemap doesn't list noindex pages
• Sitemap doesn't list auth-required pages (private routes)
• Every public route in the reachable graph appears in the sitemap (or has a documented reason not to)
• Pagination of sitemap if > 50,000 URLs (rare for personal sites)

Redirect Audit Checklist

• Permanent moves use 301, temporary use 302/307
• Redirect chains: at most one hop
• Loops: A → B → A is a hard fail
• Case-sensitivity loops (Next.js has had redirect-loop bugs here in some versions — verify against the running version)
• Trailing-slash loops
• Query-string preservation across redirects when expected
• HTTPS upgrades (HTTP → HTTPS)
• WWW canonicalization

Anchor Link Checklist

For every <a href="...#section">:

• Destination URL exists
• #section ID exists on the destination
• The destination element is visible (not display:none, not behind a closed accordion)
• Scrolling lands at the destination with offset for sticky headers

Orphan Detection Checklist

A page is an orphan if:

• It returns 200
• It's in the sitemap
• It's NOT reachable from any in-graph link

Orphans are usually old content that lost its inbound links. Decide: relink, redirect, or remove from sitemap.

Content-Quality Heuristics (lightweight content audit while crawling)

• Page has a <title>
• Page has a meta description
• Page has an <h1>
• Page has more than 200 chars of visible content
• Page has at least one outbound internal link
• Page has at least one image OR is explicitly text-only

Authenticated Crawl Notes

• Many private pages depend on data (a project page that 404s when the user has no projects)
• Decide: crawl a seeded-account state, an empty-account state, or both
• Don't crawl actions that mutate (POST / DELETE links); recognize and skip them

Performance Budget for the Crawl

• For a site with N reachable pages, budget N × 2s if rate-limiting to one request at a time
• Parallelize with a queue (4–8 concurrent) on staging, not production
• Cache HEAD responses for external URLs to avoid re-hitting

---

Calibration

Don't gate a launch on a broken external link to a third-party doc — note it for a future pass. Do gate on internal 404s in public navigation. Calibrate to whether a real user would feel it: public nav 404 is critical, an orphaned old blog post is medium.

• Severity:

  • Critical — 404 reachable from primary navigation; redirect loop; sitemap-listed URL returns 404 (Search Console will flag)
  • High — Soft 404 (200 but empty / wrong content); broken image in above-fold content; old route 404s without redirect to replacement
  • Medium — Redirect chain (301 → 301); orphan page in sitemap; external link returns 4xx
  • Low — Anchor link resolves to ID but element is hidden; case-mismatch in URL that still 301s correctly

• Confidence ratings: Confirmed (reproduced), Likely (one fetch failure could be transient), Speculative (suspect soft 404 but content might be intentional).

• Anti-hallucination guard: Don't claim a soft 404 without inspecting the actual rendered content. Don't claim an orphan without confirming the sitemap reconciliation. Always note crawl depth and start URL — coverage claims must match the crawl config.

Output Format

Start with a 5–8 line executive summary: pages crawled, 404 count, soft-404 count, redirect-chain count, orphan count, the top 3 highest-impact fixes.

1. Crawl Manifest — Start URL, depth, on-domain only, signed-out / signed-in
2. Hard 404 List — Per URL: status, referrer, suggested fix (redirect / fix link / remove)
3. Soft 404 List — Per URL: status, why it qualifies, suggested fix
4. Redirect Chain List — Per URL: chain, suggested target
5. Sitemap Reconciliation — Orphans, missing-from-sitemap, sitemap entries returning errors
6. External Link Findings — Broken externals, missing rel attributes
7. Asset Link Findings — Broken images, scripts, styles, fonts
8. Anchor Link Findings — Broken #section links

Close with a Prioritized Fix List organized by the cleanup effort: bulk-fix (set up a redirect rule), per-page fix (edit specific links), content team (remove broken outbound links).