Client-Ready Weekly Crawler Digests

AI crawler tooling often stops at “generate a file” or “glance at today’s bot hits.” The operational gap sits between those two: keep discovery files live on the right origin, express policy preferences honestly, retain path-level crawler history for weeks, and deliver a digest someone can open without digging through five vendor UIs.

That gap is the CrawlerDesk product thesis—the plumbing layer for brand.com, docs, SaaS, and agency-managed origins.

Layer 1: Managed llms.txt / llms-full.txt

Discovery starts with a correct, served file. CrawlerDesk provisions a Worker + D1-backed route. Customers point a llms. subdomain, CNAME, or workers.dev path. The dashboard editor publishes llms.txt and optional llms-full.txt with the agreed content type.

One body for every client—no alternate “AI-only” cloaking. Unauthenticated strangers cannot edit another tenant’s files. On unpaid accounts, digests and edits stop immediately; last-published files keep serving for 15 days, then hard-cut. That grace is ops-honest: crawlers still find something during a billing hiccup, without pretending the product is free forever.

This layer is for origins you control. It is not a WordPress plugin, not a Cloudflare-dashboard-only workflow, and not a pitch to “add llms.txt to Shopify Liquid”—Shopify Liquid already serves native agent discovery files. Use CrawlerDesk where brand.com / docs / SaaS still need managed hosting.

Layer 2: Robots and Content Signals helpers

Policy preferences matter. Blanket-blocking “AI” can kill useful search and agent crawlers while still leaving you without proof of who crawled what.

CrawlerDesk generates robots / Content Signals snippets for copy-paste or fetch-merge instructions. Clear limit: helpers do not claim edge enforcement on traffic that never passes through the Worker. If you need enforcement, put a Worker or proxy in path—or use your CDN’s native controls. Status and badge copy stay in the same honesty lane (AI discovery: live / AI discovery: setup) without overstating reach.

Layer 3: ≥30-day path × bot analytics

When known AI user-agents hit the Worker route (or a small beacon/log-drain Worker on a customer Cloudflare zone), path counts increment in D1. Retention is 30 days—an explicit contrast to short free Cloudflare AI Crawl Control windows. Dashboard views cover multi-day path×bot counts; CSV export covers the retention window.

You get measurable crawler attention on the paths that matter: docs, changelogs, pricing, guides. You do not get citation share-of-voice, prompt tracking, or “rank in ChatGPT” promises. Price and positioning stay on plumbing + retention + digest.

Layer 4: Weekly email + shareable /d + agency PDF

Measurement without delivery still fails the JTBD. CrawlerDesk sends a weekly email digest for billing-active sites (email only in v0—no Slack).

Distribution hooks built for agencies:

/d/{token} — signed, read-only HTML of the latest weekly digest; optional expiry; revoke → 404. Primary forward loop for clients who should not need a login.
White-label PDF on the Agency $49/mo plan (up to 5 sites)—for retainer packs and QBRs.
/i/{code} — claim a site under the inviting agency seat within the 5-site limit.

Single site $19/mo runs the same discovery + 30-day analytics + weekly digest stack for one origin. Agency is the hero plan because multi-client share and PDF are the weekly ritual most GEO/SEO shops need.

File generators and visibility assistants help draft content; they typically do not host multi-week crawler logs or white-label digests as the product. Partner framing stays simple: GetIntel drafts the fix; CrawlerDesk keeps it live and measurable. Enterprise agent-analytics stacks aimed at reverse-proxy Shopify paths sit at a different price—CrawlerDesk stays the $19–49 foundation.

See the plumbing, then start a seat

The narrative is intentionally unglamorous: discovery files → honest preference snippets → 30-day path hits → client-ready weekly digests. That is what agencies and origin owners can operate every week.

Review live demo surfaces at https://demo.crawlerdesk.com/ready (discovery files, readiness, sample digest when shown). Start an agency seat—or a single-site plan—at https://app.crawlerdesk.com/signup.