Small Language Model Spec Experiments
Where does a small, local coding model stop producing numerically correct business logic?
An empirical study of a small, locally-hosted coding model (Qwen2.5-Coder-3B on Apple MLX, 8 GB M1) and the point where it stops generating numerically correct business logic.
A five-rung spec ladder holds the rules and expected outputs constant and varies only how the spec is phrased, which pins the exact failure boundary: the model handles plain prose fine but breaks the moment literal lookup tables are removed, not on the multi-tier and boolean constructs presumed to be the culprits.
Re-phrasing the same ten rules in the model-friendly dialect moved it from 1 of 12 correct to 12 of 12 on the first attempt, and the result holds at three times the rule count (23 of 23).
- Python
- Apple MLX
- Qwen2.5-Coder
- LoRA fine-tuning
// Small Language Model Spec Experiments: where does a small local model stop being numerically correct?
// One-line: it breaks when literal lookup tables are removed, not on the multi-tier/boolean constructs presumed.
import { Project } from '@/portfolio'
export const localCodingModelExp: Project = {
name: 'Small Language Model Spec Experiments',
hook: 'Where does a small, local coding model stop producing numerically correct business logic?',
// Qwen2.5-Coder-3B on Apple MLX (8 GB M1). A five-rung spec ladder holds rules and targets constant,
// varying only phrasing. Failure boundary: removing literal lookup tables, not multi-tier/boolean logic.
// Re-phrasing the same 10 rules in a model-friendly dialect moved it from 1/12 to 12/12, holding at 23/23.
stack: ['Python', 'Apple MLX', 'Qwen2.5-Coder', 'LoRA fine-tuning'],
github: 'https://github.com/IjunaidI/LocalCodingModelExp',
}