Personal experiment / Search quality & grounding
Search Grounding & Hallucination Evaluation
Outcome & decision
Extended the search framework to distinguish evidence-grounding risk from useful answers. Evaluated 120 bilingual adversarial cases across three pipelines, with 360 saved rows and 31 tests across the two evaluation suites.
- My responsibility
- Framework extension, failure handling, and dual-judge evaluation
- Project scope
- 120 bilingual cases per pipeline; 360 saved result rows
Evidence basis. Evaluation report, source code, regression suite, manifests, and saved results. Evidence-grounding and answer-behavior scores use different definitions and coverage.
Results & decision
A pipeline can avoid an unsupported claim yet still fail to give a useful answer. I reported both measures and retained judgment coverage instead of compressing them into one hallucination score.
Valid V1 judgments gave risk rates of 39/93 (41.9%) for You.com, 28/80 (35.0%) for GLM Quark, and 52/105 (49.5%) for Qwen. Those denominators exclude invalid judgments and differ from V2’s 120 attempted cases.
| Pipeline | V2 risk · all cases | Useful · answerable cases | V1 valid judgments |
|---|---|---|---|
| You.com + Qwen | 17/120 · 14.2% | 66/105 · 62.9% | 93/120 |
| GLM Quark + Qwen | 21/120 · 17.5% | 48/105 · 45.7% | 80/120 |
| Qwen Web Answer | Not comparable¹ | — | 105/120 |
Problem & constraints
Search-enabled answers can still rely on stale, conflicting, or unsupported evidence. This study examined hallucination risk, useful answers, and operational failures across three provider chains.
There was no no-search control. The study compares search-enabled pipelines; it cannot show the causal effect of adding search.
My implementation & decisions
Extended the You.com open-source evaluation framework with HalluQA contracts, three opt-in provider chains, evidence metadata, and dual judging. The comparison and grounding suites contain 31 tests in total. Added retries, timeouts, rate-limit cooldown, batch persistence, and stage-level failure attribution.
I kept search, synthesis, and judge errors separately attributable. This made missing output and failed judgment visible to anyone deciding whether a pipeline was reliable enough for a follow-up pilot.
Method & evaluation coverage
Ran 120 cases per chain: 60 Chinese and 60 English, with 105 answerable and 15 unanswerable questions. Eight categories covered timely facts, long-tail entities, dates and versions, multi-hop reasoning, ambiguity, stale evidence, conflicting information, and unanswerable prompts, producing 360 result rows.
You.com and GLM Quark shared Top-5 retrieval, Qwen-plus synthesis, and a GLM-4.5 judge. V2 measured consistency with returned evidence. V1 evaluated final-answer behavior without that evidence. Qwen was excluded from V2 comparisons because it exposed no provider evidence.
Limits & interpretation
Evidence consistency and answer behavior are distinct from independently verified factual accuracy. This single online snapshot has unequal judge coverage, no human calibration, and no cost telemetry. The judge misread some 2026 events, making temporal cases a priority for human review.
V1 favored GLM directionally, while V2 favored You.com on the reported risk measure. Unequal valid-judgment coverage and distinct judge definitions prevent treating those as interchangeable rankings.