📖 In This Issue
Featured Snippets: (News & Resources)
Cover Story: How to Use AI to Analyze Server Logs for SEO & AI Search
Operator of Interest: Mark Traphagen
Learn This: Semantic Similarity
Small Favor: Consider Donating?
📰 Featured Snippets (News & Resources)
Anthropic accidentally allowed Search Engines to crawl and index shared chats. To most SEOs this seems like an obvious mistake that should have been accounted for. However even OpenAI was caught doing the same a year ago.
“Generate an SVG of a frog with a Habsburg jaw.“ That is the prompt that Jay Mollica sends 14 LLMs 3 times a month to test cross-model variation. It’s weird, fun, and I like it.
Mozilla released it’s annual State of Open Source AI. I am a big fan of open models especially the ones you can run locally on your own. It seems like the industry is not sure how to feel about them though, with some in full support, and others pushing for a ban.
Glenn Gabe writes about a situation where a site that no longer recieves visibility in Google, and has been deindexed in Bing, is now surging in ChatGPT responses. It is odd for sure, but to me just demonstrates that OpenAI is becoming more reliant on it’s own training data and less on outside search results.
How to Use AI to Analyze Server Logs for SEO & AI Search
If you're only checking rankings dashboards, you're missing half the picture. Rankings tell you how you're doing in the results people see. They tell you nothing about whether the crawlers behind AI answer engines, GPTBot, ClaudeBot, PerplexityBot, and others are reaching your site at all.
Server logs are the only ground-truth record of that. Every request a bot makes, every status code it gets back, every page it decides is worth crawling (or ignoring) lives in that file. Most sites generate this data constantly and never look at it, because reading raw logs by hand is tedious and the patterns worth noticing are buried in noise.
This article walks through a prompt you can use with your AI assistant to turn a raw log file into a structured report, one that separates traditional search crawlers from AI crawlers and flags real issues instead of routine traffic.
What you need before you start
A log export. Where this comes from depends on your setup:
Traditional hosting: usually available through your server admin panel (cPanel, Plesk) or by asking your host directly
CDN-fronted sites: Cloudflare, Akamai, Fastly, and similar all have log export options in their dashboards
If you don't manage the server yourself, this is a two-minute ask for whoever does.
The right format. Raw Apache or Nginx combined logs work best because they include the user-agent string on every line, that's what lets the analysis tell GPTBot apart from a regular browser. Some CDN exports strip the user-agent field by default to save space, which quietly breaks bot identification. Check this before you pull a full month of data only to find out it's unusable.
Enough history. Seven days is the minimum for a useful read. Thirty days gives a much steadier picture, especially for lower-traffic sites where a single unusual day can skew the numbers.
Why a generic prompt falls short
Ask an AI assistant to "analyze this log file" and you'll get an answer, but it's usually a generic tally of status codes and a list of the most-requested URLs. Two problems show up immediately:
AI crawlers get lumped in with everything else. Without an explicit bot taxonomy, GPTBot and ClaudeBot end up in the same bucket as Googlebot, or worse, alongside SEO tool crawlers like AhrefsBot. You lose the exact comparison you actually want: is this site as visible to AI answer engines as it is to Google?
The output isn't repeatable. If you run the same request next month with different phrasing, you get a differently structured report. That makes it impossible to compare trend over time or across clients.
The prompt below solves both by defining the bot categories up front and locking the output into a consistent format.
<role>
You are a technical SEO analyst specializing in log file analysis, crawl budget
optimization, and AI search crawler behavior (LLM/AI answer-engine crawlers as
distinct from traditional search engine bots).
</role>
<context>
I'm attaching a server log file (format: [Apache combined / Nginx / CDN — specify]).
This is for [client name / site — e.g., "an enterprise e-commerce site"].
Primary goals: identify crawl budget waste, confirm important pages are being
crawled, and assess visibility to AI answer engines vs. traditional search.
</context>
<bot_taxonomy>
Classify all bot/crawler traffic into these buckets before analysis:
1. Traditional search: Googlebot, Bingbot, DuckDuckBot, Yandex, Baidu
2. AI search / answer engines: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot,
Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended,
Amazonbot, Bytespider, CCBot
3. SEO tools (not search engines): AhrefsBot, SemrushBot, MJ12bot, DotBot
4. Unknown/unverified bots (flag user-agent strings claiming to be a known bot
but with an IP that doesn't reverse-DNS-verify — note as a caveat, don't
attempt to verify IPs yourself)
5. Human/browser traffic (only if relevant to the request)
</bot_taxonomy>
<task>
Step 1 — Parse the log and produce internal counts (do this before writing the summary):
- Requests per bot bucket above, and per individual bot within each bucket
- Status code distribution per bot (2xx / 3xx / 4xx / 5xx)
- Most-crawled URLs (top 20) and least-crawled but important URL patterns if
identifiable (e.g., product/category paths vs. filter/parameter URLs)
- Crawl frequency trend over the log's date range (daily hits per major bot)
- Any URL patterns generating disproportionate crawl volume relative to
apparent page value (parameter strings, faceted nav, pagination, redirect
chains)
Step 2 — Flag signals, not noise. Worth flagging:
- 4xx/5xx spikes concentrated on specific bots or paths
- Redirect chains (3xx → 3xx → 200) that waste crawl budget
- AI crawlers being blocked (robots.txt/403) where the client likely wants
visibility, or hitting the site heavily where they don't
- Large gaps between AI crawler and Googlebot coverage of the same important
URLs (a signal the site may rank well in Google but be invisible to AI answers)
- Asset/static file requests are NOT a signal unless volume is anomalous —
don't report favicon.ico or routine CSS/JS hits as findings
Step 3 — Write the summary using the output format below.
</task>
<output_format>
## Executive Summary
3-5 bullets, business-impact framing (not raw stats)
## Crawl Volume by Bot Category
Table: bot category | bot name | request count | % of total | status code mix
## AI Crawler Visibility
- Which AI crawlers are hitting the site, at what volume
- Whether they're accessing the same priority URLs as Googlebot
- Any blocks (robots.txt, WAF, 403s) affecting AI crawlers specifically
## Crawl Budget Issues
- Ranked list of waste sources with estimated % of crawl volume each represents
## Recommendations
- Prioritized, specific (e.g., "disallow /search?* for AI bots" not "improve
crawl efficiency")
## Data Caveats
- Log format assumptions made, date range covered, anything unparseable
</output_format>
<constraints>
- Base every number on what's actually in the attached file — don't estimate
or fill gaps with typical/industry benchmarks unless explicitly asked for
a comparison, and label those clearly as external benchmarks if used.
- If the log lacks a field needed for a section (e.g., no user-agent column),
say so in Data Caveats rather than guessing.
</constraints>Running the prompt: step by step
Export your log file for the date range you want to review.
Open a new chat and attach the log file.
Paste the prompt, filling in the bracketed context line at the top, your log format, the site or client name, and what you're trying to find out.
Read the Data Caveats section first, before anything else. This is where the analysis tells you if a field was missing or something didn't parse cleanly. Skipping this is how bad numbers end up in a client deck.
Spot-check two or three flagged findings against the raw file. AI analysis of large files is generally reliable, but a quick sanity check before you act on a number costs a minute and saves a bad recommendation.
Save the output somewhere you'll compare it against next month's run — a folder, a tracker, whatever keeps this a repeatable habit rather than a one-off exercise.
Reading the output: what actually matters
The report will have several sections, but two deserve most of your attention:
AI Crawler Visibility. This is the genuinely new signal here. It shows which AI crawlers are hitting the site and, critically, whether they're reaching the same priority pages Googlebot reaches. A site can rank well in Google while being largely invisible to AI answer engines, this section is where that gap shows up.
Crawl Budget Issues. Not everything flagged here is worth fixing. The report ranks issues by estimated share of crawl volume, so prioritize from the top down rather than chasing every item on the list.
Everything else: the executive summary, the crawl volume table, is useful context but secondary to those two.
Common pitfalls
Blocking AI crawlers by accident. A WAF rule or an old robots.txt directive written before GPTBot existed can silently block AI crawlers without anyone noticing. This shows up clearly in the AI Crawler Visibility section if you know to look for it.
Trusting unverified user-agents. Any bot can claim to be Googlebot in its user-agent string. The prompt flags unverified claims separately, but treat that bucket with extra skepticism, don't report it as confirmed traffic.
Drawing conclusions from too short a window. A three-day log pull can make a routine crawl pattern look like a trend. Stick to at least a week.
What to do next
Run this monthly and track AI crawler volume over time, the same way you'd track organic sessions. The bot taxonomy in the prompt will need occasional updates, new AI crawlers show up often enough that a list from even six months ago may be missing one that matters.
Once you have two or three months of this data side by side, you'll have something most sites don't: an actual answer to whether your content is reachable by the crawlers powering AI search, not just a guess.
👤 Operator of Interest: Mark Traphagen
Known for: Enterprise AEO, SEO, SEO tools, bluegrass music
Works at: seoClarity
Follow: LinkedIn
Learn This:
Semantic Similarity: A measure of how closely related two pieces of text are in meaning.
Small Favor?
As you might have already noticed, this newsletter is completely ad free with no sponsorship or promotions. With that in mind, please excuse me when I tell you about my dear friend who is in need of support during a very hard moment in her life. Tabitha and her daughter are being forced out of her home by her landlord and is in desperate need of assistance. If you are able to donate, or share her story, I would be very grateful. Thank you!
One more thing: AI is only as good as its operator, and if you are reading this newsletter, you’re better than most!
Till next time,
Joe Hall
PS: Let me know what you think of this issue, or anything else here: [email protected]