Project 05 / Case study
Search Provider Benchmark
Compared You.com, GLM Quark, and Qwen answer pipelines on shared English and Chinese question sets. Built reproducible comparisons that separate answer correctness, external-search latency, and failures at each pipeline stage.
Problem
Enterprise teams need evidence for choosing a search-and-answer pipeline. The study compared answer quality and retrieval speed while keeping technical failures visible in the results.
Context
An independent comparison of You.com + Qwen, GLM Quark + Qwen, and Qwen forced-search QA. The integrated Qwen path returns an answer without separately observable retrieval, so it serves as a structurally different baseline.
My Role
Developed common dataset, adapter, and result contracts with batch persistence, stage-level errors, and latency reporting. Preserved raw result CSVs and run manifests recording the seed, model IDs, provider settings, and code version.
Approach
Used the same question sets for every path. You.com and GLM Quark retrieved Top-10 results and shared qwen-plus synthesis; glm-4.5 judged all generated answers. Correctness used all attempted questions, including technical failures.
Architecture / Workflow
Dataset → run controller → provider adapter → normalized result → Qwen synthesis where applicable → GLM-4.5 judge → aggregate, pairwise, and bad-case analysis. Immutable run directories preserve the configuration and outputs.
Experiments
Evaluated 50 English questions sampled from SimpleQA with seed 42 and a complete local set of 50 Chinese questions for each provider path. Measured correctness, output availability, external-search P50/P95, and pairwise outcomes.
Results
English correctness was 44/50 (88%) for You.com + Qwen, 30/50 (60%) for GLM Quark + Qwen, and 21/50 (42%) for Qwen forced-search QA. Raw search P50 was 954 ms for You.com and 786 ms for GLM Quark, excluding synthesis and judging. Qwen retrieval latency was unavailable.
Lessons Learned
Quality and retrieval speed favored different paths in this sample. The small seeded run and saturated Chinese set limit generalization. Reported confidence intervals cover sampling uncertainty, not judge or run-to-run variance. Repeated runs and a harder Chinese set are proposed next steps.