Optimizing Websites for AI (GEO) – Information Gain, Markdown Negotiation and Agent Readiness

Why do well-written developer articles get ignored by Perplexity, ChatGPT Search, or Google AI Overviews while a single forum thread containing a raw error log gets cited at the very top? Publishing the tenth overview tutorial on a standard web API only feeds large language models information they already hold in their pretraining weights. At the same time, deeply nested DOM trees, inline SVGs, and React hydration scripts consume up to 85 percent of an agent's context window over HTTP before the first paragraph of actual engineering content is even parsed.

Generative Engine Optimization (GEO) pairs verifiable primary telemetry (Information Gain) with a machine-readable delivery architecture. Server-side content negotiation via Accept: text/markdown, explicit JSON-LD entity graphs, and structured benchmark tables cut retrieval token overhead by over 80 percent and increase citation probability across RAG answer engines. However, /llms.txt or .md routes never guarantee rankings on their own: without indexable HTML, empirical measurements, and documented failure boundaries, neural rerankers discard any page as redundant.

Why Do RAG Pipelines Discard Classic Summaries—and How Does Information Gain Work?

When a developer asks a technical question in Perplexity, ChatGPT Search, or Google AI Mode, the model runs a multi-stage RAG Pipeline (Retrieval-Augmented Generation: An architecture that retrieves relevant document chunks from a search index and injects them into the model prompt before generating a response.): the query is expanded into sub-queries (query fan-out), candidate web pages are fetched, split into chunks, and scored by a Cross-Encoder Reranker (A neural scoring model that evaluates retrieved passages for factual density, evidence quality, and novelty before they enter the final LLM context.).

Classic introductory paragraphs fail at this exact filter. A 400-word section explaining "what the Model Context Protocol is" adds zero new entropy beyond the model's base weights. In information retrieval research and Google search patents, this differential value is called Information Gain (A measure of the novel, additive information a document provides compared to previously seen sources or pretraining knowledge.).

The empirical study GEO: Generative Engine Optimization (Aggarwal et al., Princeton University & Georgia Tech, KDD 2024) demonstrated across thousands of queries that three structural signals boost visibility in generative answers by 30 to 40 percent:

1. Quantifiable Metrics Instead of Adjectives: Concrete numbers with units (e.g., "INP dropped from 280 ms to 42 ms" or "Tool selection accuracy dropped by 64% beyond 23 registered MCP tools") rather than vague claims.
2. Documented Failure Modes & System Boundaries: First-hand operational observations ("On iOS 18 Safari, the WebGPU worker terminates with an Out-of-Memory error above 1.8 GB VRAM"). Models cannot hallucinate authentic hardware post-mortems—they must cite the primary source.
3. Self-Contained Citation Blocks & Comparison Tables: Standalone 40–80 word answer paragraphs stating both mechanism and constraint, paired with Markdown tables sharing uniform evaluation criteria.

The Four Technical Layers of an AI-Optimized Website

To ensure primary engineering evidence isn't lost in DOM noise, lukiluki.me combines four architectural layers:

What Infrastructure Do I Need to Implement Markdown Content Negotiation in Next.js 16?

When a live retrieval bot such as ChatGPT-User, Perplexity-User, or ClaudeBot fetches a standard React page, it receives full server HTML including navigation chrome, inline SVG icons, Tailwind utility strings, and serialized React Server Components payloads (self.__next_f.push(...)).

Measured Payload Comparison on lukiluki.me: SSR HTML vs. Markdown Stream

We measured actual byte payloads and estimated token counts (~4 characters per token) across five routes of this Digital Garden. The standard SSR HTML response carries roughly 38.4 KB of layout and hydration shell overhead before the article prose begins:

Prerequisites for Implementation

• Next.js App Router (v16.0+): Request header inspection via src/proxy.ts (Next.js 16 replaced middleware.ts with proxy.ts) paired with a Route Handler at src/app/api/markdown/[slug]/route.ts.
• Pre-compiled Clean Markdown Corpus: A prebuild step (using gray-matter) that strips interactive client imports from MDX files and writes a clean JSON map.
• HTTP Discovery Headers: Vary: Accept (preventing CDN caches from serving raw Markdown to regular browsers) and Link headers with rel="alternate"; type="text/markdown".

Step-by-Step Implementation: src/proxy.ts

The production code below runs live on lukiluki.me. It intercepts both explicit /garden/[slug].md requests and standard article URLs carrying Accept: text/markdown:

1// src/proxy.ts (Next.js 16 File Convention)
2import { NextResponse } from "next/server";
3import type { NextRequest } from "next/server";
4import { prefersMarkdown } from "@/lib/ai-telemetry";
5
6const siteUrl = process.env.NEXT_PUBLIC_SITE_URL || "https://lukiluki.me";
7
8export function proxy(request: NextRequest) {
9  const { pathname } = request.nextUrl;
10  const acceptHeader = request.headers.get("accept") || "";
11
12  // 1. Direct .md suffix (e.g. /garden/geo.md or /en/garden/geo.md)
13  const mdMatch = pathname.match(/^(?:\/(en))?\/garden\/([^/]+)\.md$/i);
14  if (mdMatch) {
15    const locale = mdMatch[1] === "en" ? "en" : "de";
16    const slug = mdMatch[2];
17    const rewriteUrl = new URL(`/api/markdown/${slug}`, request.url);
18    rewriteUrl.searchParams.set("locale", locale);
19    return NextResponse.rewrite(rewriteUrl);
20  }
21
22  // 2. HTTP Content Negotiation (Accept: text/markdown prioritized over text/html)
23  const gardenMatch = pathname.match(/^(?:\/(en))?\/garden\/([^/.]+)$/);
24  if (gardenMatch && prefersMarkdown(acceptHeader)) {
25    const locale = gardenMatch[1] === "en" ? "en" : "de";
26    const slug = gardenMatch[2];
27    const rewriteUrl = new URL(`/api/markdown/${slug}`, request.url);
28    rewriteUrl.searchParams.set("locale", locale);
29    return NextResponse.rewrite(rewriteUrl);
30  }
31
32  // 3. Standard HTML delivery with HTTP discovery headers for AI agents
33  const response = NextResponse.next();
34  if (gardenMatch) {
35    const isEn = gardenMatch[1] === "en";
36    const slug = gardenMatch[2];
37    const mdHref = `${siteUrl}${isEn ? "/en" : ""}/garden/${slug}.md`;
38    response.headers.set("Vary", "Accept");
39    response.headers.set(
40      "Link",
41      `<${mdHref}>; rel="alternate"; type="text/markdown", <${siteUrl}/llms.txt>; rel="llms-txt"`
42    );
43  }
44
45  return response;
46}

How Do I Audit Information Gain and Test Live Markdown Streaming?

Use the interactive auditor below to test two mechanisms:

1. Top Panel (Information Gain Auditor): Select one of the three presets or paste your own technical prose to evaluate token economy (SSR HTML vs. .md), quantitative metrics, boundary signals, and banned marketing fluff.
2. Bottom Terminal (Live Endpoint Tester): Trigger a real fetch request with Accept: text/markdown against the live routes of this website (/garden/geo.md, /garden/web-mcp.md, etc.) and inspect the actual HTTP headers returned by src/proxy.ts.

What Does This Auditor Prove—and Where Are Its Limits?

Comparing Preset 1 (Empirical Post-Mortem) against Preset 2 (Generic Summary) illustrates why retrieval pipelines filter out generic prose: Preset 2 contains zero extractable numbers and triggers marketing buzzword penalties, whereas Preset 1 provides concrete failure thresholds (23 tools, 64%, 1.8 GB) and a machine-readable Markdown table.

Limitation: The browser-side score is a structural heuristic. It verifies whether formal building blocks for Information Gain exist, but cannot verify the factual accuracy of your benchmarks or external domain authority.

Which AI Crawler Reads What—and Why Does Google Ignore /llms.txt?

A common misconception is that dropping an /llms.txt file into the root directory automatically ranks a site in Google AI Overviews. Server telemetry and official search documentation show a clear split across three crawler classes:

Verification Checklist: Is Your Website Ready for AI Search and Agents?

• Information Gain Verified: Does the article include at least one original measurement table, reproducible benchmark, or real-world failure post-mortem not found in boilerplate docs?
• Direct Answer in First Quarter: Is there a standalone 40–80 word paragraph right after the hook stating mechanism, benefit, and primary limitation without requiring any click?
• Content Negotiation Active: Does curl -I -H "Accept: text/markdown" https://your-domain.com/path return 200 OK with Content-Type: text/markdown; charset=utf-8 and Vary: Accept?
• Retrieval Bots Allowed in robots.txt: Are OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, and Googlebot explicitly allowed on article routes, /llms.txt, and /api/markdown/?
• Entity Graph & Accessibility Tree: Does the embedded TechArticle JSON-LD validate cleanly, and do all interactive controls expose accessible labels and 44×44 px touch targets?

Sources and Next Steps

Primary Sources & Research

• Aggarwal, P. et al. (2024): GEO: Generative Engine Optimization – Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '24), Princeton University, Georgia Tech & Allen Institute for AI.
• Google Search Central: AI features and your website (AI Overviews & AI Mode) – Official technical guidelines for indexing, structured data, and snippet controls.
• Next.js 16.3 Documentation: How to set up your Next.js project for AI coding agents – Specification for AGENTS.md, .md routes, and Accept: text/markdown content negotiation.
• Howard, J. / Answer.AI (2024): The /llms.txt Standard Proposal – Specification for curated Markdown indices for LLM context injection.

Related Digital Garden Tiles

• Web MCP – Tools for AI Agents in the Browser: Explains how to go beyond text retrieval and expose deterministic JavaScript tools directly to browser AI agents via navigator.modelContext.
• Context Engineering with Obsidian & Graphify: Shows how to apply structured knowledge graphs and llms.txt principles to complex software repositories for coding agents.