Personal experiment / Search quality & grounding
Search Provider Benchmark
Outcome & decision
Built a repeatable three-pipeline comparison. You.com + Qwen led judged correctness on the 50-question English sample; GLM Quark had the faster external-search median. The result exposed a quality–latency trade-off for further testing.
- My responsibility
- Evaluation harness design, API integration, and result analysis
- Project scope
- Historical comparison: 50 English + 50 Chinese questions per pipeline
Evidence basis. Experiment report, evaluation code, run manifests, and saved result files. Scores describe the recorded study, not current vendor performance.
Results & decision
The strongest answer-quality result and the fastest observable retrieval came from different pipelines. I kept both measures visible so provider choice could be evaluated against the workload.
| Pipeline | Judged correct · English | Raw search P50 |
|---|---|---|
| You.com + Qwen | 44/50 · 88% | 954 ms |
| GLM Quark + Qwen | 30/50 · 60% | 786 ms |
| Qwen forced-search QA | 21/50 · 42% | Not separately observable |
Problem & comparison boundaries
Enterprise teams need evidence for choosing a search-and-answer pipeline. The study compared answer quality and retrieval speed while keeping technical failures visible in the results.
The integrated Qwen path returns an answer without separately observable retrieval. It is a structurally different baseline, not a like-for-like external search adapter.
My implementation & decisions
Developed common dataset, adapter, and result contracts with batch persistence, stage-level errors, and latency reporting. Preserved raw result CSVs and run manifests recording the seed, model IDs, provider settings, and code version.
I used shared qwen-plus synthesis for You.com and GLM Quark and a glm-4.5 judge for all three paths. Common contracts made errors comparable; including technical failures in the denominator prevented successful-only runs from overstating quality.
Method & deliverables
Evaluated 50 English questions sampled from SimpleQA with seed 42 and a complete local set of 50 Chinese questions for each provider path. Measured correctness, output availability, external-search P50/P95, and pairwise outcomes.
The external retrievers returned Top-10 results. The harness preserved raw results and manifests for aggregate, pairwise, and bad-case analysis, keeping retrieval timing separate from synthesis and judging.
Limits & next decision
Quality and retrieval speed favored different paths in this sample. The small seeded run and saturated Chinese set limit generalization. Reported confidence intervals cover sampling uncertainty, not judge or run-to-run variance. Repeated runs and a harder Chinese set are proposed next steps.
Automated judgments are not human-verified factual accuracy. The study supports a workload-specific follow-up benchmark, not a universal vendor ranking or a customer adoption claim.