Evaluate AI citations and answer accuracy and relevance

Divide AI observation questions into three types

Even if an AI answer links to the library, advising a visit on the first Sunday can lead the user to a wrong decision. GEO and AEO count citations to see whether material was used; they assess accuracy and conditions to see whether that use answered the question properly. Read both results to decide whether to fix discovery paths or the text explaining exceptions.

Natural-discovery questions ask a real user’s need without supplying the site name or URL. Specified-source questions explicitly request a particular site to check access and interpretation. Specific-document questions narrow the title or topic to check whether the article can be reached. Their conditions differ, so calculate citation rates separately.

Specified-source questions already supply the address, bypassing natural discovery. If these answers are correct but natural-discovery answers omit the site, investigate interpretation and discovery separately. If both miss the same exception, first read the cited passage and placement of the condition. This comparison is why question types are separated.

Run under identical conditions and retain originals

For each observed service, record web-search availability and the selected mode. Ask one question in a fresh conversation without mixing context from an earlier conversation where the site was supplied. If region, language, login, or personalization cannot be completely fixed, record their values at observation time. Do not guess a model name when the screen does not disclose it.

Determine repetitions per question before running. If set to three, save all three answers instead of selecting those with good citations. If answers vary widely, increase observations and duration to measure variation. Record service errors, restrictions, and response failures as execution failures, distinct from normal answers without citations.

Store the full answer, actual citation links, and execution time. Open each link to verify the intended document. Separate URL mentions, actual source citations, and name-only mentions. Keep browser service results separate from API model results. Their search tools, models, and personalization may differ and produce different answers.

Judge citations and answer relevance separately

Split each answer’s core claims into sentences and compare them with cited pages. Accuracy concerns factual correctness; timeliness, validity at the question’s reference time; relevance, the question’s subject, conditions, and intent; sufficiency, whether key subquestions were omitted. Adding a link does not make these four values agree.

For example, correctly quoting weekday hours in answer to “this Sunday’s opening hours” transcribes facts accurately but poorly fits the question. Listing only documents for “required documents and submission method” is insufficient. Treating pre-change guidance as current is a timeliness error. Beside each judgment, record which sentence differed from which evidence.

Aggregate counts together with denominators

Calculate the observed citation rate using normal answers whose relevant-document citations were checked to completion as the denominator. Even normal answers with truncated or unchecked source areas remain unknown. Present execution failures, completed responses, and assessable responses together. Calculate name-mention rate, relevant-document citation rate, and accurate-answer rate separately.

Observed citation rate = normal answers citing a relevant document / normal answers assessed for citation Accurate-answer rate = answers with all core facts correct / answers assessed for accuracy Relevant-answer rate = answers matching the question’s subject, conditions, and intent / answers assessed for relevance Also report = total attempts, execution failures, unknowns

For example, if two of twelve attempts fail and four of ten normal answers cite the document, report 4/10 together with two failures. Do not classify unknowns as right or wrong; display them beside the assessable denominator. With small samples, show actual counts rather than percentages alone.

Before and after changes, collect responses for the same URLs, questions, services, and conditions. Mark dates of simultaneous advertising or external coverage. Comparing citation increases with changes in text, service, and traffic conditions helps choose the next variable to compare.

Assess one answer by claim, source, and condition

Compare two answers for the fictional library. The question asks when it is open this Sunday, whose reference date is the first Sunday. Current opening guidance explicitly says it closes on the first Sunday.

If answer A says “Open Sunday from 10 to 17” and links current guidance, it has a citation but omits the closure exception. If answer B says with evidence, “Closed this Sunday; check the separate schedule for the next opening day,” it reflects the condition. In actual assessment, check both the calendar reference date and the cited text’s effective period.

Do not count unrelated external sites as target-site citations. Separate site citations from target-detail-document citations. Record links lacking a passage supporting the claim as insufficient evidence. Separate unassessable answers from denominators with reasons.

Count source support and factual correctness separately

metrics.mjs aggregates cited, supported, accurate, timely, relevant, and complete: relevant-document citation, claim support, factual accuracy, time-appropriate material, alignment with intent and conditions, and sufficient coverage. The last complete is separate from response status: complete. Record a reason for every true, false, and null judgment; null means unassessable.

Fix the question as “Can I visit on November 1, 2026?” If the source says “Sundays 11–16 in 2025” while the answer says “Still open 11–16,” the source does not support the current claim. Both supported and accurate are false. If the answer limits itself to “It was 11–16 in 2025,” supported and accurate can be true for that historical claim. Yet timely, relevant, and complete are false because the current question is unanswered. This is why a factually correct sentence differs from a sufficient answer.

Save metrics.mjs, the assessment-fixture.mjs below, and metrics.test.mjs together and run node --test metrics.test.mjs. Six tests cover zero denominators, failures and unknowns, historical facts, partial claim support, missing fields, and separate aggregation conditions. The fixture has four attempts, three completed, and one failed. Support is one of two assessable responses, with one unknown.

// Fixture data for answer assessment. const common = { questionId: 'sunday-hours', questionVersion: '2', service: 'fixture-assistant', mode: 'offline', model: null, question: 'Can I use the fictional library on Sunday, November 1, 2026?', reference: 'Opening hours effective October 2026: closed on the first Sunday and public holidays.', rawPath: 'raw/answer-1.json', status: 'complete' }; export const examples = [ { ...common, answer: 'Sunday hours are 10–17. You can visit on November 1 too.', cited: true, supported: false, accurate: false, timely: true, relevant: true, complete: false, reasons: { cited: 'Links current opening guidance as the source', supported: 'The original does not support the second claim', accurate: 'Conflicts with first-Sunday closure', timely: 'Uses material effective October 2026', relevant: 'Addresses the date and facility in the question', complete: 'Omits the closure exception' }, claims: [ { text: 'Regular Sunday hours are 10–17', sourceUrl: 'https://example.com/policies/opening-hours', quote: 'Sunday 10–17. Closed on the first Sunday and public holidays.', supported: true, accurate: true, reason: 'The claim limited to regular hours agrees' }, { text: 'A visit on November 1 is possible', sourceUrl: 'https://example.com/policies/opening-hours', quote: 'Closed on the first Sunday', supported: false, accurate: false, reason: '2026-11-01 is the first Sunday' }, ] }, { ...common, rawPath: 'raw/answer-2.json', answer: 'Sunday hours were 11–16 in 2025.', cited: true, supported: true, accurate: true, timely: false, relevant: false, complete: false, reasons: { cited: 'Links the 2025 record', supported: 'The historical fact with its year matches the source', accurate: 'Matches the historical fact in the fictional reference', timely: 'Not material applicable to 2026 in the question', relevant: 'Does not answer whether a visit on 2026-11-01 is possible', complete: 'Omits the current closure status' }, claims: [{ text: 'Sunday hours 11–16 in 2025', sourceUrl: 'https://example.com/archive/opening-2025', quote: 'Sunday hours 11–16 in 2025', supported: true, accurate: true, reason: 'Both sentence and source are limited to 2025' }] }, { ...common, rawPath: 'raw/answer-3.json', answer: 'The response was truncated during storage.', cited: null, supported: null, accurate: null, timely: null, relevant: null, complete: null, reasons: Object.fromEntries(['cited','supported','accurate','timely','relevant','complete'] .map(key => [key, 'Stored original is truncated and cannot be assessed'])), claims: [{ text: 'Unknown', sourceUrl: null, quote: null, supported: null, accurate: null, reason: 'Incomplete original' }] }, { ...common, rawPath: 'raw/attempt-4.json', status: 'failed', failure: 'HTTP 429', answer: null }, ];

import test from 'node:test'; import assert from 'node:assert/strict'; import { ratio, evaluate, grouped } from './metrics.mjs'; import { examples } from './assessment-fixture.mjs'; test('Zero denominator returns null and invalid counts are rejected', () => { assert.equal(ratio(0, 0).rate, null); for (const pair of [[2, 1], [-1, 1], [0.5, 1]]) assert.throws(() => ratio(...pair)); }); test('Separate execution failures and unknowns from their denominators', () => { const r = evaluate(examples); assert.equal(r.attempts, 4); assert.equal(r.completed, 3); assert.equal(r.failed, 1); assert.deepEqual(r.metrics.supported, { numerator: 1, denominator: 2, rate: 0.5, unknown: 1 }); assert.equal(r.metrics.complete.numerator, 0); }); test('Correct historical facts may fail timeliness and relevance for a current question', () => { const r = evaluate([examples[1]]).metrics; assert.equal(r.accurate.rate, 1); assert.equal(r.timely.rate, 0); assert.equal(r.relevant.rate, 0); }); test('Partial claim support cannot count as support for the whole answer', () => { assert.throws(() => evaluate([{ ...examples[0], supported: true }]), /summary mismatch/); }); test('Reject missing judgments, reasons, originals, or claim evidence', () => { for (const key of ['timely', 'reasons', 'claims', 'rawPath']) { const broken = structuredClone(examples[0]); delete broken[key]; assert.throws(() => evaluate([broken])); } const broken = structuredClone(examples[1]); broken.claims[0].quote = null; assert.throws(() => evaluate([broken])); }); test('Aggregate changed question versions, services, and modes separately', () => { const rows = [examples[0], { ...examples[0], questionVersion: '3' }, { ...examples[0], service: 'another' }, { ...examples[0], mode: 'browser' }]; rows.push({ ...examples[0], model: 'model-b' }); assert.equal(grouped(rows).length, 5); }); console.log(JSON.stringify(grouped(examples), null, 2));

This function takes human-recorded true, false, and null judgments and response status to calculate counts and rates. First store the original question, reference date, expected facts, original response, and supporting passages, then apply consistent criteria. ALCE’s citation evaluation and practical answer-level summary metrics use different units; do not combine them under one score name. ALCE study

evaluate requires each completed row to include its original path, question version, service, mode, and judgment reasons. Break all core claims into claims with source passages and judgments. If any claim is unsupported, overall supported is false. It is true only when all are true; remaining unknowns make it null. grouped separates responses by question, question version, service, mode, and confirmed model. Leave an unidentifiable model null. Include confirmed models in the aggregation key so model changes are not mixed.

Separating supported from accurate distinguishes source-selection errors from factual-transmission errors. A correct closure fact cited to a page lacking that fact receives different factual and support judgments. Conversely, accurately copying historical hours may not answer a current question. Separate metrics.mjs values expose these failures instead of hiding them in one citation rate and guide the next investigation.

Distinguish research visibility metrics from operational metrics

The 2024 GEO paper’s GEO-bench divides 10,000 questions into 8,000 training, 1,000 validation, and 1,000 test questions. The baseline retrieves Google’s top five sources and generates answers with gpt-3.5-turbo, sampling five responses at temperature 0.7. It selects one source and compares modifications such as adding quotations, statistics, or citations. GEO paper §3

In Table 1, Position-Adjusted Word Count Overall is 19.3 before modification and 27.2 for Quotation Addition. (27.2−19.3)÷19.3 gives an increase of about 40.9%. This measure reflects the amount and position of answer content linked to a source. Subjective Impression is a separate metric: the same table gives 19.3 before modification and 24.7 with quotations added. GEO paper Table 1

Answer-level citation rate is calculated differently. Four answers citing a document out of ten gives 40%. The rate is identical whether each cites one line at the end or several sentences from the beginning. A research metric weighting amount and position may score the cases differently.

To assess adding quotations to a site, run the same questions before and after. Compare the number of citing answers, amount of cited text, and inclusion of necessary exceptions. If model or search mode changes, group those responses separately.

Test current roles and video exceptions with questions

For organizational roles, ask both “Who currently operates the library?” and “Which organization prepared its opening in 2025?” Expected evidence is the current introduction for the first and contemporaneous minutes for the second. Answering the current question with the preparatory committee is temporally wrong even with a real citation URL. Answering the historical question with the current operations team makes the same error.

When errors occur, read the retrieved document’s ID, effective period, and cited passage. If the current introduction itself states an old role, fix the original. If the original has the new role but search holds an older publication, fix recrawling. If both are current yet an old document is selected for a current question, investigate temporal conditions and ranking. Adding a date to the answer does not finish the causal investigation.

The video exercise answer “Open every Sunday 10–17” read the hours at 0–4 seconds but omitted the closure exception at 4–8 seconds. Separate hours, recurrence conditions, and closure exceptions into claims and record the segment supporting each. Check in order whether the linked URL opens that segment, whether its original supports the answer, and whether the question’s date is a closure day.