Personal experiment / QwenTax
QwenTax — RAG & Retrieval Evaluation
Outcome & decision
Selected a heuristic GraphRAG profile after comparing ten retrieval configurations on 330 cases. It recovered 821/1,553 expected sources (52.9%), yet covered every required source for only 22/313 source-bearing questions (7.0%).
- My responsibility
- Retrieval design, expected-source evaluation, and failure analysis
- Project scope
- Independent study · 330-case study of source coverage; separate from answer evaluation
Evidence basis. Ten original per-case retrieval result files and a 3,267-chunk knowledge base. Source recall uses 1,553 expected sources; question-level hit rates use 313 source-bearing questions.
Results & decision
Method 9 led the reported comparison. An LLM parser recovered slightly fewer sources, so I retained the deterministic heuristic as the evidence profile for the paired model study.
Adaptive Top-K improved recall to 42.6% but increased average context from 2,636 to 4,527 tokens. Domain rules alone added less than half a percentage point. More context helped coverage but did not reliably complete multi-source evidence chains.
| Retrieval profile | Expected-source recall |
|---|---|
| Single retrievers | 31.2–34.8% |
| Three-way reciprocal rank fusion | 38.0% |
| Adaptive Top-K | 42.6% |
| Method 9 · heuristic GraphRAG | 821/1,553 · 52.9% |
| Method 10 · LLM parser | 812/1,553 · 52.3% |
Problem & constraints
Retrieval-augmented generation (RAG) combines retrieved documents with model-generated answers. Weak answers can originate in retrieval, context selection, or generation. I isolated source coverage: did the selected context include the evidence roles and jurisdictions needed for a multi-source tax answer?
QwenTax uses a 3,267-chunk local knowledge base for China–U.S. cross-border tax questions. The retrieval benchmark is the evidence layer for the companion SFT study: this page measures source coverage, while the SFT page holds retrieval constant and measures how Base and adapted models use the same evidence.
My implementation & decisions
Built and benchmarked TF-IDF, BM25, and BGE-M3 indexes; implemented domain reranking, three-way RRF, adaptive Top-K, and heuristic and LLM GraphRAG routing; designed expected-source evaluation; analyzed missing-source and context-budget failures; and froze the strongest profile as the shared input for Base-versus-SFT experiments.
Started with single retrievers to expose lexical and semantic trade-offs, added low-weight domain rules, combined candidate ranks with reciprocal rank fusion (RRF), and expanded the result set for complex questions. Because larger contexts still missed distinct evidence roles, Method 9 added deterministic graph planning and coverage-aware selection. Method 10 tested an LLM parser against that heuristic.
Method & source coverage
Used the same 330 questions across ten profiles. Of these, 313 had 1,553 expected sources and 17 deliberately required outside verification or acknowledged insufficient local evidence. Metrics were any-source hit, all-source hit, source recall, selected chunks, and context tokens; RAG evaluation remained separate from answer judging.
Method 9 found at least one expected source for 309/313 source-bearing questions (98.7%) and every expected source for 22/313 (7.0%). The other 17 questions deliberately called for outside verification or insufficient-evidence handling.
The graph planned retrieval; only source documents entered the legal evidence context. This boundary prevents routing hints from being treated as authoritative tax evidence.
Limits & interpretation
Combining lexical and dense results improved source recall. Adaptive selection and coverage-aware planning improved it further, with larger evidence contexts. Method 9 was selected for this benchmark; the 7.0% complete-source coverage shows that missing evidence remains a major limitation.
These are retrieval metrics, not tax-answer accuracy. Expected-source labels and one domain-specific benchmark constrain the result; production quality would require separate answer evaluation and expert review.