Five hidden failure modes in AEO/GEO measurement pipelines

1

Background

GEO/AEO research increasingly relies on automated multi-LLM probes (Profound, BrightEdge AEO, ScrunchAI, ChatRank, etc.). Most tools do not expose their temperature settings, provider-specific config edge cases, rate-limit behaviour, or timeout heuristics. We set out five such failure modes that we found in our own pipeline (Cloudflare Workers + Anthropic + OpenAI + Gemini + Workers AI) — and argue that these are systemic, not implementation-specific.

2

Mode 1 — Workers AI default temperature = 0 (the determinism trap)

Symptom: citation rate identical to the second decimal place across 3 consecutive daily runs.

Root cause: Cloudflare Workers AI env.AI.run() defaults to temperature: 0 when not set explicitly. With deterministic decoding: same question → same answer → same score. The pipeline correctly probed 5 Workers AI models daily, but each produced byte-identical output.

Why this matters for AEO: determinism is invisible in scorecards. A stable 9 % is not a "finding" but an artefact.

Fix: explicit temperature: 0.5 in askWorkersAi(env, model, q). Post-fix variance: ±2.3 pp day-to-day.

3

Mode 2 — OpenAI Search Preview rejects temperature

Symptom: the openai_search stage returns HTTP 400 — temperature is incompatible with search-preview models.

Root cause: gpt-4o-mini-search-preview-2025-03-11 has a different parameter contract than the base model. It rejects temperature, top_p, and several other sampling parameters.

Fix: conditional if (!isSearchPreview) body.temperature = 0.3; in askOpenAI().

4

Mode 3 — Anthropic Tier-1 rate limit (5 RPM) causes 16 parallel timeouts

Symptom: all 16 Claude probes time out simultaneously when called in parallel.

Root cause: Anthropic Tier-1 enforces 5 requests per minute. Sixteen parallel fetch() calls to /v1/messages cause requests 6–16 to queue server-side past the 45-second Cloudflare sub-request timeout.

Fix: the Anthropic Message Batches API (/v1/messages/batches) — async submit, max 24h delivery, −50 % cost discount. The 16 × 3 Claude tiers = 48 prompts are submitted in a single batch, polled hourly via a separate cron, and the results persisted retroactively into the snapshot.

The identical pattern applies to the Gemini Direct Batch API (from v2.8) and replaces the sync path for both providers.

5

Mode 4 — sub-request timeout too tight for long-tail probes

Symptom: ~5 % of probes return AbortError: The operation was aborted despite a valid provider response.

Root cause: the Cloudflare Worker sub-request timeout was set globally to 45 s. Workers AI Llama-3.1-8b on long-tail questions occasionally responds at 46–48 s. The Worker aborted the connection before the provider had finished streaming.

Fix: CHUNK_TIMEOUT_MS 24 s → 50 s. Trade-off: slightly higher Worker CPU cost, in return for eliminating the 5 % silent-failure rate.

6

Mode 5 — implicit score dilution through misclassified errors most impactful

Symptom: aggregate citation hit rate of 2–13 % over 4 days, even though per-LLM inspection shows individual LLMs at 14–28 %.

Root cause: when a provider API call fails (quota exhaustion, HTTP 5xx, network timeout), the standard pipeline pattern is:

async function askProvider(env, q) {
  try {
    const r = await fetch(...);
    return r.ok ? await r.json() : null;  // ← silent failure
  } catch (e) {
    return null;  // ← silent failure
  }
}

The downstream code receives null, passes it to scoreAnswer(''), which returns { score: 0, status: 'not_found' }. The row is persisted as a legitimate "not_found" with score=0. The aggregate calculation treats it identically to a genuine "LLM honestly says don't know" answer — both contribute 0/3 to the percentage calculation.

For one LLM out of eleven this is invisible (one row × 16 questions = −48 max points from both numerator and denominator, which cancels in the percentage calculation). For five LLMs down simultaneously (e.g. Workers AI quota fully exhausted), the dilution is severe:

Detection: inspect the raw answer_excerpt field. Genuine "not_found" answers contain LLM-generated text. Misclassified errors have an empty answer_excerpt plus status='not_found'. The combination is the fingerprint.

Fix:

1. Remove silent error catching: let API errors propagate into the outer chunk runner, so that status='error' (not 'not_found') is captured.
2. Aggregate filter: in reaggregateSnapshot(), exclude status='error' rows from BOTH numerator and denominator.
3. Per-LLM transparency: report n_attempted + n_legit + n_error per LLM. Pure-error LLMs (n_legit=0) are given status: 'unavailable'.
4. Retroactive correction with versioning: historical snapshots remain in legacy columns (score_percent); new, methodologically correct values are stored in parallel columns (score_percent_v271). Both remain visible in the time series for a full audit trail.

Empirical effect in our pipeline (44 runs, 2026-05-10 to 2026-05-17):

The pre-fix narrative "LLM citation for a pre-launch author stabilised at 9–13 %" was systematically under-reported by 5–15 pp. The corrected narrative is "LLM citation across measurable LLMs averaged between 14 and 22 % in the first week, with per-LLM noise driven primarily by provider availability rather than entity-recognition variance."

Detection heuristic for AEO tooling auditors: inspect the answer_excerpt distribution per LLM per run. If >50 % of rows for a given LLM in a given run have an empty answer_excerpt with status='not_found', assume API-call-failure misclassification until proven otherwise. Industry tools that do not expose raw answer_excerpt values may have silently distributed this bug across their entire customer base.

7

Six best-practice recommendations for AEO tooling builders

1. Default temperature ≥ 0.3 on every provider, every call. Document the value in the scorecard methodology.
2. Test each provider's parameter contract individually before scaling. A compatibility test is the unit test for AEO.
3. Use provider batch APIs for high-volume probes. API Batches, the Gemini Batch API, and OpenAI Batch give −50 % cost + zero rate-limit friction. Async ≠ slow when you do daily collects.
4. Calibrate timeouts on p99 response time, not p50. Long-tail probes will burn you.
5. Treat zero-variance result sets as suspect. Determinism in AEO is almost always a bug, not a property.
6. Treat an empty answer_excerpt + status='not_found' as suspect, not legitimate. Genuine LLM "don't know" answers contain LLM-generated text. Empty answers with a not_found classification are misclassified API errors. Audit the raw answer_excerpt distribution per LLM per run — anything > 50 % empty strings is a smoking gun for Mode 5 dilution.

8

Limitations

• Single-pipeline observation (Cloudflare Workers); may not generalise to other serverless platforms.
• Provider behaviour changes over time — these specific bugs may be patched provider-side by the time of publication.
• We did not test other batch APIs (Vertex Direct, OpenAI Batch) — extrapolating to them is conjecture.

9

Replication

• Code: github.com/marintkael/marin-research-tools (MIT)
• Methodology Note 01 v2.8: DOI 10.5281/zenodo.20308495
• Pre-registration: DOI 10.5281/zenodo.20125967
• Companion paper: Working Paper 02 · Pre-Launch LLM Citation
• Living engineering journal: Project challenges and solutions — all pipeline bugs in full detail, including symptom + root cause + solution per entry

10

How to cite

Kael, M. T. (2026). Five hidden failure modes in AEO/GEO measurement pipelines: lessons from a 7-day pre-registered experiment. Working Paper 04 v0.2, Marin T. Kael — KI-Zitations-Feldlabor. Status: outline. URL: marin-t-kael.de/en/research/working-papers/wp-04-five-failure-modes.