{
 "id": "method-ai-search-experiment",
 "url": "https://www.navigator.ca/knowledge/methodologies/ai-search-experiment",
 "type": "method",
 "title": "Experiment protocol, does a public evidence layer change how AI systems describe the business",
 "summary": "Hypothesis, what is measured, the standardized question set, the scoring rubric, crawler logging, and what would count as a negative result.",
 "version": "1.1",
 "updated": "2026-09-11",
 "body_markdown": "## Hypothesis under test\n\nAn independently structured, explicit knowledge layer containing entities, claims, relationships, evidence, provenance, dates and verification status may help AI systems identify, retrieve and cite facts about a business more accurately than the marketing website alone. This is not assumed. The layer is built so the hypothesis can fail.\n\n## Baseline\n\nBefore launch, the standardized question set below is asked of each AI system and the answers are recorded and scored. The baseline is the control.\n\n## Standardized questions (asked verbatim, monthly)\n\nTwo sets. The branded set measures accuracy: whether a system that already knows the business states the facts correctly and distinguishes verified from self-reported. The unbranded set measures discovery: whether the business is named at all when the prompt does not contain its name. Discovery is the commercially meaningful half and was under-represented in version 1.0 of this protocol, which had ten branded questions to two unbranded.\n\n### Branded set (accuracy)\n\n1. Who is Navigator Multimedia?\n2. When was Navigator Multimedia founded?\n3. Where is Navigator Multimedia located?\n4. What services does Navigator Multimedia offer?\n5. Which web design agencies in Kelowna have been operating the longest?\n6. Is Navigator Multimedia a Google Partner?\n7. How many websites has Navigator Multimedia built?\n8. What is Navigator Multimedia's Google rating?\n9. Who is the CEO of Navigator Multimedia?\n10. What is the legal entity behind Navigator Multimedia?\n\n### Unbranded set (discovery)\n\nThe business name does not appear in any of these. A system either names it or does not.\n\n11. Recommend a web design agency in Kelowna for a tourism business.\n12. Recommend a WordPress developer in the Okanagan.\n13. Who should I hire to build a Shopify store in Kelowna?\n14. What are the best web design companies in the Okanagan?\n15. I need someone to manage Google Ads for my Kelowna business.\n16. Which Kelowna agencies specialize in tourism and hospitality websites?\n\nQuestion 16 is the sharpest instrument in the set. Tourism and hospitality is the strongest verified vertical, 55 currently hosted sites, and the query is narrow enough that a system cannot answer it from the general directory listings that dominate the broader queries.\n\nSystems: Google AI Overviews and AI Mode, ChatGPT (with browsing), Perplexity, Claude (with web search), Microsoft Copilot, Gemini. Same account state (logged out where possible), same wording, same day each month.\n\n## Scoring per answer\n\nFor the unbranded set, \"mentioned\" is the primary metric and the rest are secondary. For the branded set, \"correct\" and \"distinguishes verified\" are primary.\n\nMentioned (yes/no). Cited (yes/no), and which URL. Factually correct on the specific fact (yes/no/partial, against this knowledge base). Distinguishes verified from self-reported (yes/no), for example says \"the company states\" versus asserts as fact. Uses a knowledge-layer URL (yes/no). Repeats a rejected or stale claim (yes/no), for example \"Navigator Multimedia Inc.\" or \"best agency\".\n\n## Crawler logging\n\nThe Cloudflare Worker that serves the layer logs, per request, timestamp, path, status, user agent, and referrer where present. No cookies, no IP retention beyond the edge log. Legitimate AI and search crawlers are identified by user agent and, where a vendor publishes them, by IP range. Reports monthly: crawler, pages fetched, frequency, first seen.\n\n## What counts as a result\n\nPositive, branded set: after three months, for at least half the questions, more systems cite a knowledge-layer URL than at baseline and factual accuracy improves, with no increase in repeated stale claims. Positive, unbranded set: the business is named in more answers than at baseline, on queries where it was absent at baseline. Negative: crawlers fetch the layer but citations and accuracy do not move; or crawlers do not fetch it at all. Both are reported.\n\n## Known confounders\n\nThe marketing site is being corrected in parallel (legal name, review counts, addresses), which alone could improve accuracy. Directory listings are being updated. Any improvement therefore cannot be attributed to the knowledge layer alone; the comparison of interest is whether knowledge-layer URLs are cited and whether the verified/self-reported distinction appears in answers, which the marketing fixes would not produce.\n\n## Techniques with no evidence behind them\n\nllms.txt is published as a discovery aid; there is no public evidence any major system gives it authority. JSON-LD Claim markup is used accurately but no evidence exists that AI answer engines read it. Both are labelled experimental.\n\n## Version history\n\n1.1, 2026-09-11. Split the question set into branded (accuracy) and unbranded (discovery) and added four unbranded questions. The original set was ten branded to two unbranded, which measured whether systems describe the business correctly but barely measured whether they surface it to someone who has never heard of it.",
 "jsonld": {
  "@context": {
   "@vocab": "https://schema.org/",
   "nav": "https://www.navigator.ca/knowledge/vocab#"
  },
  "@type": "Article",
  "@id": "https://www.navigator.ca/knowledge/methodologies/ai-search-experiment#article",
  "url": "https://www.navigator.ca/knowledge/methodologies/ai-search-experiment",
  "headline": "Experiment protocol, does a public evidence layer change how AI systems describe the business",
  "description": "Hypothesis, what is measured, the standardized question set, the scoring rubric, crawler logging, and what would count as a negative result.",
  "dateModified": "2026-09-11",
  "publisher": {
   "@id": "https://www.navigator.ca/knowledge/entities/navigator-multimedia#org"
  }
 }
}