# Optimizing Websites for AI (GEO) – Information Gain, Markdown Negotiation and Agent Readiness

> **Author:** Lukas Kopischke (Lead Software Engineer, Düsseldorf, Germany)
> **Canonical URL:** https://lukiluki.me/en/garden/geo
> **Markdown Endpoint:** https://lukiluki.me/en/garden/geo.md
> **Category:** KI & Agents | **Tags:** GEO, llms.txt, Next.js 16, JSON-LD, RAG, Content Negotiation
> **Published:** 2026-10-23 | **Updated:** 2026-10-23
> **Summary:** Practical engineering guide to Generative Engine Optimization (GEO): Information Gain, Accept: text/markdown in Next.js 16, JSON-LD and token telemetry.

---
Why do well-written developer articles get ignored by Perplexity, ChatGPT Search, or Google AI Overviews while a single forum thread containing a raw error log gets cited at the very top? Publishing the tenth overview tutorial on a standard web API only feeds large language models information they already hold in their pretraining weights. At the same time, deeply nested DOM trees, inline SVGs, and React hydration scripts consume up to 85 percent of an agent's context window over HTTP before the first paragraph of actual engineering content is even parsed.

Generative Engine Optimization (GEO) pairs verifiable primary telemetry (Information Gain) with a machine-readable delivery architecture. Server-side content negotiation via `Accept: text/markdown`, explicit JSON-LD entity graphs, and structured benchmark tables cut retrieval token overhead by over 80 percent and increase citation probability across RAG answer engines. However, `/llms.txt` or `.md` routes never guarantee rankings on their own: without indexable HTML, empirical measurements, and documented failure boundaries, neural rerankers discard any page as redundant.

> **Merksatz (Information Gain Over Synthesis):**
> Language models synthesize textbook definitions on their own—they cite external websites only when those pages provide empirical metrics, documented failure modes, or configuration boundaries absent from pretraining weights. Technical protocols like Accept: text/markdown and JSON-LD ensure that this primary evidence reaches the model's context window without HTML noise.

## Why Do RAG Pipelines Discard Classic Summaries—and How Does Information Gain Work?

When a developer asks a technical question in Perplexity, ChatGPT Search, or Google AI Mode, the model runs a multi-stage RAG Pipeline (Retrieval-Augmented Generation: An architecture that retrieves relevant document chunks from a search index and injects them into the model prompt before generating a response.): the query is expanded into sub-queries (*query fan-out*), candidate web pages are fetched, split into chunks, and scored by a Cross-Encoder Reranker (A neural scoring model that evaluates retrieved passages for factual density, evidence quality, and novelty before they enter the final LLM context.).

Classic introductory paragraphs fail at this exact filter. A 400-word section explaining *"what the Model Context Protocol is"* adds zero new entropy beyond the model's base weights. In information retrieval research and Google search patents, this differential value is called Information Gain (A measure of the novel, additive information a document provides compared to previously seen sources or pretraining knowledge.).

The empirical study *GEO: Generative Engine Optimization* (Aggarwal et al., Princeton University & Georgia Tech, KDD 2024) demonstrated across thousands of queries that three structural signals boost visibility in generative answers by **30 to 40 percent**:
1. **Quantifiable Metrics Instead of Adjectives:** Concrete numbers with units (e.g., *"INP dropped from 280 ms to 42 ms"* or *"Tool selection accuracy dropped by 64% beyond 23 registered MCP tools"*) rather than vague claims.
2. **Documented Failure Modes & System Boundaries:** First-hand operational observations (*"On iOS 18 Safari, the WebGPU worker terminates with an Out-of-Memory error above 1.8 GB VRAM"*). Models cannot hallucinate authentic hardware post-mortems—they must cite the primary source.
3. **Self-Contained Citation Blocks & Comparison Tables:** Standalone 40–80 word answer paragraphs stating both mechanism and constraint, paired with Markdown tables sharing uniform evaluation criteria.

### The Four Technical Layers of an AI-Optimized Website

To ensure primary engineering evidence isn't lost in DOM noise, `lukiluki.me` combines four architectural layers:

  ### 1. Primary Telemetry & Post-Mortems (Content)
Every guide answers the core question in the first quarter (40–80 words) and backs claims with measured benchmarks, exact package versions, and reproducible failure boundaries.
  ### 2. HTTP Content Negotiation (.md) (Transport)
When an AI agent or CLI requests a route with `Accept: text/markdown` or a `.md` extension, `src/proxy.ts` streams clean semantic Markdown stripped of navigation HTML and hydration scripts.
  ### 3. Dense JSON-LD Entity Graphs (Semantics)
A connected Schema.org `@graph` (`TechArticle`, `Person`, `knowsAbout`, `sameAs`, `encoding`) links author identity, technology entities, and the Markdown mirror in the document head.
  ### 4. Semantic Accessibility Tree (Actuation)
Strict ARIA roles, labeled form controls, and browser tools ([Web MCP](/en/garden/web-mcp)) allow autonomous browser agents to navigate interactive components without brittle DOM scraping.

---

## What Infrastructure Do I Need to Implement Markdown Content Negotiation in Next.js 16?

When a live retrieval bot such as `ChatGPT-User`, `Perplexity-User`, or `ClaudeBot` fetches a standard React page, it receives full server HTML including navigation chrome, inline SVG icons, Tailwind utility strings, and serialized React Server Components payloads (`self.__next_f.push(...)`).

### Measured Payload Comparison on lukiluki.me: SSR HTML vs. Markdown Stream

We measured actual byte payloads and estimated token counts (~4 characters per token) across five routes of this Digital Garden. The standard SSR HTML response carries roughly **38.4 KB of layout and hydration shell overhead** before the article prose begins:

| Garden Route on `lukiluki.me` | SSR HTML Payload | Estimated HTML Tokens | Clean `.md` Stream | Estimated `.md` Tokens | Token Overhead Saved |
| :--- | :--- | :--- | :--- | :--- | :--- |
| `/garden/geo` | 54.8 KB | ~13,700 Tokens | 9.6 KB | ~2,400 Tokens | **-82.5%** |
| `/garden/web-mcp` | 51.2 KB | ~12,800 Tokens | 7.4 KB | ~1,850 Tokens | **-85.5%** |
| `/garden/brownfield-ai` | 56.4 KB | ~14,100 Tokens | 10.2 KB | ~2,550 Tokens | **-81.9%** |
| `/garden/3d-print-mcp` | 52.0 KB | ~13,000 Tokens | 7.8 KB | ~1,950 Tokens | **-85.0%** |
| `/garden/accessibility` | 49.6 KB | ~12,400 Tokens | 6.8 KB | ~1,700 Tokens | **-86.3%** |

### Prerequisites for Implementation
* **Next.js App Router (v16.0+):** Request header inspection via `src/proxy.ts` (Next.js 16 replaced `middleware.ts` with `proxy.ts`) paired with a Route Handler at `src/app/api/markdown/[slug]/route.ts`.
* **Pre-compiled Clean Markdown Corpus:** A prebuild step (using `gray-matter`) that strips interactive client imports from MDX files and writes a clean JSON map.
* **HTTP Discovery Headers:** `Vary: Accept` (preventing CDN caches from serving raw Markdown to regular browsers) and `Link` headers with `rel="alternate"; type="text/markdown"`.

> **Real-World Failure Mode: Outdated Training Weights on Next.js 16 Proxy**
> If you prompt an LLM without local documentation context to write content negotiation for Next.js, it almost always generates a `middleware.ts` file. Starting with Next.js 16, `middleware.ts` is deprecated and renamed to `proxy.ts` exporting `proxy(request: NextRequest)`. Documenting such version-specific breaking changes provides high Information Gain.

### Step-by-Step Implementation: `src/proxy.ts`

The production code below runs live on `lukiluki.me`. It intercepts both explicit `/garden/[slug].md` requests and standard article URLs carrying `Accept: text/markdown`:

```typescript
// src/proxy.ts (Next.js 16 File Convention)
import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
import { prefersMarkdown } from "@/lib/ai-telemetry";

const siteUrl = process.env.NEXT_PUBLIC_SITE_URL || "https://lukiluki.me";

export function proxy(request: NextRequest) {
  const { pathname } = request.nextUrl;
  const acceptHeader = request.headers.get("accept") || "";

  // 1. Direct .md suffix (e.g. /garden/geo.md or /en/garden/geo.md)
  const mdMatch = pathname.match(/^(?:\/(en))?\/garden\/([^/]+)\.md$/i);
  if (mdMatch) {
    const locale = mdMatch[1] === "en" ? "en" : "de";
    const slug = mdMatch[2];
    const rewriteUrl = new URL(`/api/markdown/${slug}`, request.url);
    rewriteUrl.searchParams.set("locale", locale);
    return NextResponse.rewrite(rewriteUrl);
  }

  // 2. HTTP Content Negotiation (Accept: text/markdown prioritized over text/html)
  const gardenMatch = pathname.match(/^(?:\/(en))?\/garden\/([^/.]+)$/);
  if (gardenMatch && prefersMarkdown(acceptHeader)) {
    const locale = gardenMatch[1] === "en" ? "en" : "de";
    const slug = gardenMatch[2];
    const rewriteUrl = new URL(`/api/markdown/${slug}`, request.url);
    rewriteUrl.searchParams.set("locale", locale);
    return NextResponse.rewrite(rewriteUrl);
  }

  // 3. Standard HTML delivery with HTTP discovery headers for AI agents
  const response = NextResponse.next();
  if (gardenMatch) {
    const isEn = gardenMatch[1] === "en";
    const slug = gardenMatch[2];
    const mdHref = `${siteUrl}${isEn ? "/en" : ""}/garden/${slug}.md`;
    response.headers.set("Vary", "Accept");
    response.headers.set(
      "Link",
      `<${mdHref}>; rel="alternate"; type="text/markdown", <${siteUrl}/llms.txt>; rel="llms-txt"`
    );
  }

  return response;
}
```

  #### Empfohlen (Do): Idempotent Markdown with Vary: Accept and Canonical Metadata
Always send `Vary: Accept` and include the canonical URL, publication date, and author entity inside the Markdown header so context remains intact when an agent ingests only the `.md` stream.
  #### Vermeiden (Don't): User-Agent Cloaking with Divergent Claims for Search Bots
Never serve factually different claims to `Googlebot` than to human readers. Serving divergent content without transparent content negotiation violates search engine cloaking policies.

---

## How Do I Audit Information Gain and Test Live Markdown Streaming?

Use the interactive auditor below to test two mechanisms:
1. **Top Panel (Information Gain Auditor):** Select one of the three presets or paste your own technical prose to evaluate token economy (SSR HTML vs. `.md`), quantitative metrics, boundary signals, and banned marketing fluff.
2. **Bottom Terminal (Live Endpoint Tester):** Trigger a real `fetch` request with `Accept: text/markdown` against the live routes of this website (`/garden/geo.md`, `/garden/web-mcp.md`, etc.) and inspect the actual HTTP headers returned by `src/proxy.ts`.

### What Does This Auditor Prove—and Where Are Its Limits?
Comparing Preset 1 (*Empirical Post-Mortem*) against Preset 2 (*Generic Summary*) illustrates why retrieval pipelines filter out generic prose: Preset 2 contains zero extractable numbers and triggers marketing buzzword penalties, whereas Preset 1 provides concrete failure thresholds (`23 tools`, `64%`, `1.8 GB`) and a machine-readable Markdown table.

**Limitation:** The browser-side score is a **structural heuristic**. It verifies whether formal building blocks for Information Gain exist, but cannot verify the factual accuracy of your benchmarks or external domain authority.

---

## Which AI Crawler Reads What—and Why Does Google Ignore `/llms.txt`?

A common misconception is that dropping an `/llms.txt` file into the root directory automatically ranks a site in Google AI Overviews. Server telemetry and official search documentation show a clear split across three crawler classes:

| Crawler / User-Agent | Operator & Purpose | Respects `robots.txt`? | Reads `/llms.txt` Automatically? | Preferred Format & Lever |
| :--- | :--- | :--- | :--- | :--- |
| **`Googlebot` / `Google-Extended`** | Google Search, AI Overviews & Gemini training | **Yes** (`Google-Extended` only governs training, not search citations) | **No** (Google relies strictly on its regular HTML/DOM index) | Server-rendered HTML (SSR), internal links, visible prose, and valid `TechArticle` JSON-LD. |
| **`OAI-SearchBot`** | OpenAI Search Index for ChatGPT Search | **Yes** | **Rarely** (Crawls sitemap & HTML pages) | Clear question headings, comparison tables, and high factual density in the top quarter. |
| **`ChatGPT-User` / `Perplexity-User`** | User-initiated live fetch during a chat session | **Partially** (Treated as user-requested fetch) | **Yes, when prompted** or via agent skills | Benefits heavily from `Accept: text/markdown` and fast TTFB. |
| **`PerplexityBot`** | Perplexity AI Citation Index (~10 sources per answer) | **Yes** | **Rarely** | Strongly favors first-hand post-mortems, benchmark tables, and specific error fixes. |
| **`ClaudeBot` / Claude Code** | Anthropic Web Index & Coding Agents | **Yes** | **Yes** (Especially in developer documentation workflows) | Direct `.md` URLs (`/garden/geo.md`), `/llms-full.txt`, and versioned code blocks. |

---

## Verification Checklist: Is Your Website Ready for AI Search and Agents?

- [ ] **Information Gain Verified:** Does the article include at least one original measurement table, reproducible benchmark, or real-world failure post-mortem not found in boilerplate docs?
- [ ] **Direct Answer in First Quarter:** Is there a standalone 40–80 word paragraph right after the hook stating mechanism, benefit, and primary limitation without requiring any click?
- [ ] **Content Negotiation Active:** Does `curl -I -H "Accept: text/markdown" https://your-domain.com/path` return `200 OK` with `Content-Type: text/markdown; charset=utf-8` and `Vary: Accept`?
- [ ] **Retrieval Bots Allowed in `robots.txt`:** Are `OAI-SearchBot`, `ChatGPT-User`, `PerplexityBot`, `ClaudeBot`, and `Googlebot` explicitly allowed on article routes, `/llms.txt`, and `/api/markdown/`?
- [ ] **Entity Graph & Accessibility Tree:** Does the embedded `TechArticle` JSON-LD validate cleanly, and do all interactive controls expose accessible labels and 44×44 px touch targets?

---

## Sources and Next Steps

### Primary Sources & Research
* **Aggarwal, P. et al. (2024):** [*GEO: Generative Engine Optimization*](https://arxiv.org/abs/2311.09735) – Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '24), Princeton University, Georgia Tech & Allen Institute for AI.
* **Google Search Central:** [*AI features and your website (AI Overviews & AI Mode)*](https://developers.google.com/search/docs/appearance/ai-features) – Official technical guidelines for indexing, structured data, and snippet controls.
* **Next.js 16.3 Documentation:** [*How to set up your Next.js project for AI coding agents*](https://nextjs.org/docs/app/guides/ai-agents) – Specification for `AGENTS.md`, `.md` routes, and `Accept: text/markdown` content negotiation.
* **Howard, J. / Answer.AI (2024):** [*The /llms.txt Standard Proposal*](https://llmstxt.org/) – Specification for curated Markdown indices for LLM context injection.

### Related Digital Garden Tiles
* **[Web MCP – Tools for AI Agents in the Browser](/en/garden/web-mcp):** Explains how to go beyond text retrieval and expose deterministic JavaScript tools directly to browser AI agents via `navigator.modelContext`.
* **[Context Engineering with Obsidian & Graphify](/en/garden/brownfield-ai):** Shows how to apply structured knowledge graphs and `llms.txt` principles to complex software repositories for coding agents.

---
*Quelle: [Lukas Kopischke – Digital Garden](https://lukiluki.me/en/garden/geo) | Vollständiger Index: [llms.txt](https://lukiluki.me/llms.txt)*