Project 06 / Case study
Search Grounding & Hallucination Evaluation
Evaluated three search-enabled answer pipelines using 120 bilingual adversarial questions. Separate judgments of evidence grounding and answer behavior revealed different provider trade-offs, supported by failure tracking and 31 offline regression tests.
Problem
Search-enabled answers can still rely on stale, conflicting, or unsupported evidence. This study examined hallucination risk, useful answers, and operational failures across three provider chains.
Context
Compared You.com + Qwen, GLM Quark + Qwen, and Qwen Web Answer. The experiment contains no no-search control, so it assesses differences among search-enabled pipelines rather than the causal effect of adding search.
My Role
Extended the You.com open-source evaluation framework with HalluQA contracts, three opt-in provider chains, evidence metadata, dual judging, and 31 offline regression tests. Added retries, timeouts, rate-limit cooldown, batch persistence, and stage-level failure attribution.
Approach
You.com and GLM Quark shared Top-5 retrieval, Qwen-plus synthesis, and a GLM-4.5 judge. V2 measured consistency with returned evidence. V1 evaluated final-answer behavior without that evidence. Qwen was excluded from V2 comparisons because it exposed no provider evidence.
Architecture / Workflow
Dataset → provider sampler → final answer and available evidence → V1/V2 judging → raw CSV and run manifest → provider, language, category, and answerability analysis. Search, synthesis, and judge failures remain separately attributable.
Experiments
Ran 120 cases per chain: 60 Chinese and 60 English, with 105 answerable and 15 unanswerable questions. Eight categories covered timely facts, long-tail entities, dates and versions, multi-hop reasoning, ambiguity, stale evidence, conflicting information, and unanswerable prompts, producing 360 result rows.
Results
V2 risk was 17/120 (14.2%) for You.com and 21/120 (17.5%) for GLM Quark. Useful answers among 105 answerable cases were 66/105 (62.9%) and 48/105 (45.7%). V1 favored GLM directionally, but valid-judgment coverage differed: 93/120 for You.com, 80/120 for GLM, and 105/120 for Qwen.
Lessons Learned
Evidence consistency and answer behavior are distinct from independently verified factual accuracy. This single online snapshot has unequal judge coverage, no human calibration, and no cost telemetry. The judge misread some 2026 events, making temporal cases a priority for human review.