Experiment / Case 06
QwenTax — Local RAG vs. Qwen3.7 Plus
Outcome & decision
Expanded a pilot to 330 tax-review questions. Base Qwen3-4B + RAG averaged 75.33/100 against 81.52 for Qwen3.7 Plus without RAG. The local system led on a 16-question evidence-gap subset, supporting targeted evaluation rather than a full replacement.
- My responsibility
- Resumable comparison workflow, evidence-input controls, and result analysis
- Project scope
- Independent study · 330 questions per comparison · Completed September 2026
Evidence basis. Three complete answer sets and two new remote-comparison rounds, each with 330 paired GLM judgments. Reported totals are normalized rubric scores. Local models received frozen Method 9 evidence; the remote model received only the question. Compare scores within each pair, not across judging rounds.
330-question results & decision
Qwen3.7 Plus received higher average scores than both local RAG configurations across the full 330-question set. The result does not support replacing the remote system across this workload. It does identify a narrower role for local RAG when recognizing missing evidence matters.
The judge preferred Base + RAG on 107/330 questions and the remote model on 223/330. In the separate V4 comparison, it preferred V4 + RAG on 104/330 and the remote model on 226/330. These are recorded judge preferences; they are not the same as counting which numerical total was higher.
An earlier three-question pilot favored local RAG, 87.7 versus 71.3/100. The full comparison used a larger question set and different saved local outputs; it did not reproduce that overall lead. The pilot alone was insufficient for an architecture decision, and its scores are not pooled with the full run.
| Local configuration | Local mean /100 | Qwen3.7 Plus mean /100 |
|---|---|---|
| Base Qwen3-4B + RAG | 75.33 | 81.52 |
| SFT V4 Qwen3-4B + RAG | 74.73 | 81.73 |
Architecture question & constraints
For a domain assistant, when can a small local model with relevant documents serve as an alternative to a remote API? I compared complete answer configurations to inform that choice, including a fine-tuned local candidate rather than assuming adaptation would close the gap.
Base Qwen3-4B-Instruct-2507 and its SFT V4 adapter used the same frozen Method 9 evidence and local answer policy. The remote endpoint, recorded as qwen3.7-plus-2026-05-26, received the same questions and a Chinese-answer policy without supplied retrieval. Model, prompts, and evidence access therefore changed together.
I reused the existing 330 Base and 330 V4 answers, retained three validated remote pilot answers, and completed the remaining 327 remote answers. The original Base–V4 comparison remains a separate evaluation: its 84.73 and 85.02 scores cannot be combined with these new rounds into a three-model ranking.
My implementation & decisions
Built a resumable generation-and-judging workflow that reused validated answers, retried missing work, and rejected conflicting records. Keeping generation separate from judging allowed the comparison to finish without discarding existing outputs or mixing incompatible inputs.
Added checks for all QA001–QA330 IDs, nonempty answers, model and prompt settings, frozen evidence hashes, answer-to-judge bindings, and normalized score totals. The final report reconciled 990 answer records and 990 paired judgments: 330 existing Base–V4 judgments plus 660 new judgments against the remote model.
Preserved prompts, answers, hashes, available usage, and completion metadata for review. I reported each pair independently and distinguished the judge’s chosen winner from numerical score order, so a convenient combined leaderboard would not misrepresent the experiment.
Where local RAG performed better
Across all 330 questions, the remote model scored higher on expected-point coverage, evidence grounding, jurisdiction handling, and clarity. Both local systems scored higher on uncertainty handling and compliance with prohibited claims. This suggests a useful distinction between supplying a fuller answer and recognizing what the available evidence cannot establish.
In the existing 16-question category covering state tax and missing or current-policy evidence, Base + RAG averaged 88.00 versus 72.75 for the remote model. V4 + RAG averaged 87.88 versus 73.31 in its separate round. This is an exploratory slice of the same benchmark, not an independent validation set or a result for all tax questions.
The finding supports testing a local evidence-constrained workflow for that narrower use case, with an explicit request for missing information or further verification. It does not establish that such routing is ready for production.
Limits & next decision
The evaluation used one GLM-4.6V-FlashX judge and a six-part, 100-point rubric. Candidate identities were hidden, but evidence access was disclosed. Grounding contributed 25 points against the frozen reference supplied to the judge; the remote model had not seen that reference. There is no matched-evidence remote control or independent tax-expert calibration.
All configurations declared a 1,200-token output limit, but recorded API usage exceeded it. The remote run recorded 240/330 length terminations. Base and V4 reached the configured cap on 289/330 and 269/330 answers respectively, but their finish reasons were not saved. Equal effective output budgets are unverified; neither speed nor cost savings are established.
The benchmark was reused during development, and answers and judging rounds were collected at different times. The evaluator clamps rubric components to their valid ranges and recomputes totals; some raw totals and winner labels disagree with those scores. Citation-audit fields also remain inconsistent. These are automated evaluation results, not verified factual accuracy.
Before choosing a deployment, test a held-out workload, add the remote model with the same evidence, calibrate with expert review, verify output budgets, and measure latency including retrieval and operating cost. The current decision is to investigate local RAG for a bounded evidence workflow while retaining the remote model as the stronger overall baseline in this evaluation.