llms.txt and the AI Crawlers Report
Inkwell generates /llms.txt and /llms-full.txt from a site's settings and posts, starting with an Identity block. Each AI crawler token can be allowed or blocked by name in Settings, and the AI Crawlers report shows which operators visited, on which days, and which pages they read.
On this page
AI answer engines send their own crawlers, with their own names, and read a file of their own. Inkwell generates /llms.txt and /llms-full.txt for every site from its settings and posts, lets you allow or block each crawler by name, and shows you who actually came.
The identity section
llms.txt opens with an Identity block: the site's name, what it is, who publishes it and the statement from Settings. Below it, every published post and page with a one-line description. llms-full.txt carries the full text. Both are generated on request, so a new post appears immediately.
Crawler policy
Under Settings, Crawlers, each known token (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended) has an allow or disallow switch that writes the right lines into robots.txt. The names are the published ones, not guesses.
The AI Crawlers report
Desk, AI Crawlers lists identified crawler visits by operator and day, which paths they fetched, and whether the address was verified as the operator's. Finished days are summarised by a job so the report stays fast however much traffic there is. Pair it with answer blocks to see whether the pages you shaped for quoting are the ones being read, and with the firewall, which never scores a verified crawler on a 404.
