There is a saying about cobblers and their children’s shoes. This month I audited my own website the way I would a client’s, and I would like to tell you what I found, partly as penance and partly because I suspect your site has the same problems.

The context: more and more people do not start with Google. They ask an assistant. “Who can help a small business get started with AI in Sydney?” The assistant went away, read what it could, and came back with names. Getting onto that list has picked up a couple of acronyms, GEO and LLMO, but the first step has nothing clever about it. The machines have to be able to read you.

What I found on my own site

The sitemap Google held was from 2021. I rebuilt this site from scratch not long ago. The new build produces a perfectly good sitemap. Google did not have it. The only one it had ever been given was five years old and returning errors. Every page I had added since was something Google might stumble across, or might not.

Three essays had never been indexed. Weeks after publishing, Search Console listed them as “discovered, currently not indexed”. If Google has not indexed a page, the assistants that lean on search results will not find it either.

My welcome mat named a guest who no longer exists. My robots.txt politely invited a list of AI crawlers by name. One of the Anthropic names on it had been retired. It did no harm, since everything was allowed anyway, but it told me nobody had looked at the file since it was written. That nobody was me.

Nothing was measuring enquiries. Analytics counted visits and not the one thing a site like this exists to produce. A different problem, but it came out of the same hour.

None of this was exotic. All of it was invisible until I looked.

The two files worth knowing

robots.txt is the gate. It says which automated visitors may read what. The AI companies publish the names their crawlers use, and they come in three kinds: those that collect material for training, those that index for search, and those that fetch a page because a person just asked about it. You can treat each differently. I allow all three, because I would rather be in the answer than out of it.

The trap is being blocked without knowing. Some hosting and security services now block AI crawlers unless you say otherwise. If yours does, you can write the best page in your industry and no assistant will ever quote it.

llms.txt is newer and optional. It is a short plain-text file at the root of your site that says, in effect, “here is who I am, and here are the eight pages that matter”. Mine took ten minutes.

What I honestly think llms.txt is worth

Not much yet, and I would be wary of anyone who tells you otherwise. None of the big assistants has said it uses the file when composing answers, and Google has said it does not. Some tools do read it, and I have pasted mine into a conversation more than once as a quick way of briefing an AI about what I do.

I still think you should have one, for a reason that has little to do with robots. Writing it forces you to say what you do in two sentences and choose the handful of pages you would show a stranger. I found that harder than it should have been, and the homepage got clearer as a result.

The ten-minute check

  1. Type your address followed by /robots.txt. Look for the word Disallow. If it appears next to anything you do not understand, ask someone.
  2. Ask your host one question: do you block AI crawlers by default?
  3. Right-click your homepage and view the source. Search for a sentence from the page. If it is not in there, your site is built in a way many AI crawlers cannot read.
  4. Open Search Console and look at two things: which sitemap Google holds, and how many of your pages are actually indexed.
  5. Ask two assistants what your business does. Read the answers as a customer would. What they get wrong tells you what your site fails to say plainly.

Then, if you like, write the llms.txt.

The part that matters more than any file

Every fix above is plumbing. What decides whether an assistant recommends you is whether your site says, in ordinary words, what you do, who it is for, and why someone should believe you, and whether the rest of the internet agrees. No text file substitutes for that.

If you run the check and do not like what you find, send me the address. I will tell you what the machines can see. I am in no position to judge.