Personal experiment / QwenTax
QwenTax — SFT & Model Evaluation
Outcome & decision
Used paired evaluation to diagnose a severe fine-tuning regression and guide four QLoRA iterations. On the 330-case benchmark, V4 scored 85.02 versus 84.73 for its Base control: a small +0.29-point gain that still needs replication and expert review.
- My responsibility
- Dataset engineering, QLoRA training, and paired evaluation
- Project scope
- Independent study · Four adaptation iterations; a paired Base control for each evaluation
Evidence basis. Original per-case judgment files confirm 330 scored pairs per run, using a GLM-4.6V-FlashX judge. Training and dataset artifacts support the experiment; results have no human tax-expert calibration.
Results & decision
Paired evaluations exposed a severe V2 regression despite a low development loss. V4 was the only positive paired score; its +0.29-point difference is too small to establish a dependable improvement.
| Run | Adapted score | Paired Base | Difference |
|---|---|---|---|
| V1 | 84.21 | 84.42 | −0.21 |
| V2 | 26.15 | 85.56 | −59.41 |
| V3 | 82.79 | 83.99 | −1.20 |
| V4 | 85.02 | 84.73 | +0.29 |
Problem & constraints
Can supervised fine-tuning (SFT) improve how Qwen3-4B-Instruct-2507 uses bilingual tax evidence without increasing unsupported confidence? I used QLoRA, a memory-efficient adapter-training method, and evaluated answer behavior separately from training loss, output length, and retrieval variation.
This is an experimental tax QA system, not a production advisory service. Training ran on a 12 GB RTX 3060, requiring a 4-bit frozen base with LoRA adapters.
My implementation & decisions
Built versioned, evidence-bound datasets with train/development splits and duplicate, token, end-of-sequence, and assistant-mask checks. Configured four QLoRA iterations, compared Base and adapted models, analyzed failures, and revised data and training controls while preserving configs, hashes, adapters, and evaluation artifacts.
Established Base and conservative V1, then iterated through observation, hypothesis, controlled change, and paired evaluation. V2 increased data, adapter capacity, and training intensity but collapsed on QA evaluation. V3 restored conservative capacity and stronger evidence and length audits. V4 curated prior data, lowered training intensity, fixed bilingual controls, and retained checkpoints for review.
Method & evaluation controls
Held questions, Method 9 evidence, prompts, and generation settings constant within each Base-versus-adapter comparison. Anonymous paired judging assessed evidence use, coverage, jurisdiction handling, uncertainty, compliance, and clarity.
V4 used 1,067 curated bilingual records: 947 training and 120 development examples. Low-rank adapters (r=8), a 1e-5 learning rate, and one training epoch reduced training intensity; a fixed 40-case panel and saved checkpoints supported error review.
Limits & interpretation
V2 reached a dev loss of 0.2620 yet failed on the RAG-conditioned evaluation. Matching training data to inference conditions and checking answer behavior proved essential. Frozen evidence and a paired Base control made the regressions interpretable; the small V4 gain remains preliminary.
V4 reduced sub-60 answers from six to two in the recorded run. The same 330-case benchmark was reused across iterations, creating a risk of selection bias. The 1,200-token generation cap was reached in 269/330 V4 answers and 289/330 Base answers, so output length limits also affect interpretation.
Formal checkpoint selection still lacked completed semantic review. The paired results have no statistical significance test or tax-expert calibration; they do not establish a business or advisory improvement.