AI agents were recommending my competitor.
You’re not optimizing your website just for humans anymore. You’re optimizing it for AI agents too: Grok, Meta’s Muse, ChatGPT, Claude, and every new one coming. I noticed these agents were recommending my competitors over MagicSlides, so on 28 Oct 2026 I audited 10 of our pages.
24 words a crawler got from /chatgpt and /tools/ai-pitchdeck-generator
0 / 10 pages showed a date
1 link in the footer 404’d, on every page
0 Disallow lines in the robots.txt groups for AI crawlers
ChatGPT Presentation Maker – Make PPT in ChatGPT | MagicSlides GPT AI agents: site index at /llms.txt · any page as markdown at <path>.md = 24 words. No headline. No features. No price. No reviews.
The cause: the content loaded client-side. A browser runs the JavaScript and fills the page; most AI crawlers don’t, so they saw an empty shell. I’m moving those pages to server-side rendering so they arrive pre-built. After the fix: 1,867 and 2,230 words.
Make every page say what it is.
Label every form field. Give images real alt text. An agent filling your form reads the label, not the placeholder. An image with no alt is a blank box to it. Write what the image shows: “slide 3 of a sales deck with a bar chart”, not “image1”.
Pricing, proof and specs in the page HTML. Not loaded later by a script. If your price arrives through JavaScript, the agent may never see it. Render it on the server (see page 2).
First H2 = the buyer’s question, answered in ~50 words. “What is the fastest way to make a deck from a YouTube video?” then a straight answer. Make your questions real H2 tags so the AI knows what the main thing is.
Add an “it depends” answer. Who should pick you, and who should pick a competitor. Agents are asked to compare. Give them an honest comparison to quote.
Swap slogans for one clear line. What it is, who it’s for, what it does. “Presentations, reimagined” tells an agent nothing. “AI presentation maker for students and teams: topic, PDF or YouTube link in, editable PPT out” does.
Testimonials as plain text with name + title. Not screenshots or images. A review inside a picture can’t be read or quoted. And use only reviews you can stand behind, with a real person’s name.
Quick test: open your page and try to highlight your price and your reviews. If you can’t select the text, AI can’t read it either.
Prove it’s fresh. Let the bots in.
Write “as of Oct 2026” next to prices and facts that change. None of our 10 pages had a date, so an agent can’t tell how old the data is. That makes it hard for it to recommend you.
robots.txt: allow the AI crawlers. meta-externalagent, meta-webindexer, GPTBot, ClaudeBot, PerplexityBot. And repeat your Disallow lines in each group (below).
Fix broken links. Screaming Frog is free up to 500 URLs. Our footer had a link that 404’d on every page of the site.
Use real tags: header, nav, main, article, button. Not a pile of divs. A <button> tells an agent “click me”; a styled div doesn’t.
Show published + updated dates. Refresh key pages every 30–90 days. Fresh content, with dates, on the pages you want recommended.
The robots.txt trap
Our User-agent: * group had 36 Disallow lines (/api/, /dashboard/, /signin/…). The AI groups below it, in this screenshot, (ClaudeBot, PerplexityBot…) only say Allow: /.
A crawler obeys only the group that names it. Its own group replaces the * group, so all those Disallow lines didn’t apply to the AI bots.
# fix: repeat the rules per bot User-agent: GPTBot Disallow: /api/ Disallow: /dashboard/ Allow: /
Three checks. No tools to buy.
curl the page and count the words
This fetches your page the way a crawler does (no JavaScript) and counts the words. Replace the URL with yours.
curl -sL https://yoursite.com/page | python3 -c "import sys,re,html;s=sys.stdin.read();s=re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>',' ',s);print(len(html.unescape(re.sub(r'<[^>]+>',' ',s)).split()),'words')"
# my real output, 28 Oct 2026 magicslides.app/chatgpt 24 words magicslides.app/tools/youtube-to-ppt 1974 words
A real page with a few hundred words should come back with a few hundred. If you get 20–50, the content is loading client-side.
Open view-source and Ctrl-F your price
Type view-source: before your URL in Chrome (or press Ctrl/Cmd + U). Search for your price, your product name and one review. If it’s not in the source, it’s not in what the crawler reads.
Chrome DevTools → Accessibility tree
Right-click your form or button → Inspect → open the Accessibility tab and look at the tree. That’s roughly what an agent “sees”. A field with no name, a button that shows up as plain text, or an image with no description means points 1 and 10 need work.
- Every field has a label, every image has alt text
- Price, proof and specs are in view-source
- First H2 is the buyer’s question
- An honest “who should pick us vs them”
- One clear line: what, who, does what
- Reviews are text, with name + title
- “As of” date next to prices and facts
- AI bots allowed, Disallow lines repeated
- No broken links
- header, nav, main, article, button
- Published + updated dates, refreshed
The fix list from the audit.
2 pages gave crawlers 24 words Server-side rendering, so the pages arrive pre-built: 1,867 and 2,230 words.
0 of 10 pages had a date “Updated October 2026” near each H1, matching dateModified in the schema, and “Prices as of October 2026” on pricing.
Footer link 404’d on every page Removed it. It pointed at a page that no longer exists.
AI crawler groups had no Disallow lines Every bot group now repeats the same Disallow lines as User-agent: *. Added Meta’s crawlers too.
Unlabelled inputs, images with no meaning Labels on the prompt box and file upload, real alt text on example slides.
Div soup A real <header> and exactly one <main> per page.
The page the footer link led to, on every page of the site. Humans rarely click it. Crawlers follow every link, so each one is a dead end on your site.
Next for me: fresh content with dates on the pages that matter, and checking every week whether ChatGPT, Perplexity and Google AI Mode name MagicSlides. That check has its own guide.
