CrawlerDesk

AI crawler plumbing for any origin.

Managed llms.txt + 30-day crawler digests — not WordPress, not CF-dashboard-only.

Discovery files, 30-day AI crawler hits, and a weekly digest clients open with no login.

Start agency seat — $49/mo See demo digest

Email digests only in v0 · no Shopify Liquid · signup on app.

Agency $49 · 5 sites Single site $19/mo Compare plans

demo.crawlerdesk.com/d/…

Weekly AI crawler digest

CrawlerDesk public demo

Sample week (illustrative)
1,284
AI bot hits (week)
18
Paths hit
GPTBot
Top bot
  • /llms.txt512
  • /docs/api301
  • /pricing188
  • /blog/agents121
Sample week (illustrative)

Live on demo.crawlerdesk.com

What you get

Managed discovery files

Edit llms.txt / llms-full.txt once; one body served to every client.

30-day crawler analytics

Path × known AI bot hits for 30 days, with CSV — longer than typical free CF windows.

Client-ready digests

Weekly email + shareable /d link + white-label PDF on the $49 plan.

Who it’s for

Lead

GEO/SEO agencies · multi-client · forwardable proof

Also

Single-site docs/SaaS/brand.com · $19

Not for

SOV buyers · Shopify Liquid install · CF-dashboard-only

How it works

  1. 1

    Create agency seat

    on app.crawlerdesk.com.

  2. 2

    Attach client sites

    up to 5; invite with /i/{code}.

  3. 3

    Publish & measure

    files live; hits roll 30 days.

  4. 4

    Forward proof

    /d link or white-label PDF; revoke anytime.

Pricing

Agency — recommended

$49/mo
  • Up to 5 sites
  • Client invites
  • White-label PDF digests
  • Shareable weekly digests
Start agency seat

Single site

$19/mo
  • One origin
  • Same plumbing: files, 30-day hits, weekly digest
Start single site
  • Unpaid: edits/digests stop immediately; files keep serving 15 days, then hard cut.
  • Custom hostname attach uses Cloudflare for SaaS (may show staged until zone ops complete).
  • Referral links are attribution only — no payouts.

Works beside GetIntel — not instead of it.

GetIntel drafts the fix; CrawlerDesk keeps it live and measurable.

FAQ

Does this block AI bots at the edge?

No. We serve files and measure hits that reach CrawlerDesk. We do not claim off-Worker enforcement.

Shopify storefronts?

We don’t sell “add llms.txt to Shopify Liquid.” Use CrawlerDesk for brand.com / docs / SaaS origins you control.

Slack digests?

Not in v0 — email only.

Why not just Cloudflare AI Crawl Control?

Great if you’re deep in CF already; free metrics windows are short and multi-tenant white-label digests aren’t the product. CrawlerDesk is origin-agnostic plumbing + agency share.

Can clients pay separately?

On the agency plan, invited client sites bill to the agency seat within the 5-site limit.

Agency $49/mo · 5 sites · client invites — or single site $19/mo

Start agency seat — $49/mo Sign in