What a week of teaching a local coding model actually taught me
Posted on:
One week of making the experiment measurable
This week I worked on oktopai, an experiment in local coding specialists. The idea is simple: use a stronger model as a teacher, use compiler-verified examples to train a smaller student, and eventually load the right specialist only when it is needed.
The important part is the word verified. A fluent answer is not necessarily a good programming answer. For TypeScript, the first useful question is whether the code compiles. For repository work, that is still not enough: the edit must touch the right file, fix the reported problem, and survive an independent replay.
I group related problems into task families. A family is simply a category of TypeScript work, such as generic indexed access, null narrowing, overloads, or record dictionaries. Measuring families matters because an average score can hide a specialist that is excellent at one kind of problem and unusable at another.
The panels use different denominators: 2,450 teacher attempts, 200 held-out student tasks, and 20 patch-emitter tasks. They should be read as separate measurements, not as one combined accuracy score.
The teacher–student setup
The teacher was Qwen3-Coder 30B. The student was Qwen2.5-Coder 3B with a LoRA adapter. LoRA leaves the base model frozen and stores the specialization in a small, replaceable set of weights.
The pipeline looked like this:
task → teacher answer → strict compiler/test gate → student training
↓
rejected examples retained
The teacher produced a corpus of 2,450 compiler-verified examples. That sounds like a success, but the first student trained on it verified only 57 of 200 unseen tasks. The unchanged 3B base verified 94 of 200.
Those 200 tasks are a fixed evaluation suite kept apart from training. The student never studies their answers. Both the base and the student receive the same 200 tasks, and their generated code is checked with the same executable validators. This makes the comparison fair and prevents a training score from being mistaken for real improvement.
The student was faster in some measurements, but speed did not compensate for writing less correct code. The adapter was therefore kept as a research artifact and not exported or promoted.
The numbers, in one view
These are two different measurements, so I am deliberately not putting them on one fake shared scale.
Teacher data acceptance:
Qwen2.5-Coder 7B, first prompt 71 / 2450 2.9% |
Qwen2.5-Coder 7B, corrected prompt 503 / 2450 20.5% |████
Qwen3-Coder 30B teacher 2450 / 2450 100.0% |████████████████████
Fixed 200-task student verification:
Qwen2.5-Coder 3B base 94 / 200 47.0% |█████████
Original teacher-trained student 57 / 200 28.5% |██████
Balanced external-data probe 31 / 200 15.5% |███
The second chart is the more important one for deciding whether a student is useful. The balanced external-data probe was carefully prepared: it had 585 candidate training records, no duplicate IDs, no exact overlap with the teacher corpus, and a better source balance. It was split into 467 training records, 61 validation records, and 57 probe-test records. It was not the 200-task evaluation suite. The actual 200-task result above comes from the fixed held-out suite, not from those 585 records. The probe still made the student worse. More data was not the missing ingredient.
The validation loss shown in the third panel is the model's prediction error on examples reserved during training. Lower validation loss generally means the model is getting better at predicting those validation answers. It is useful for detecting learning and overfitting, but it is not the same as producing a correct repair on a new task. That is why the patch-emitter chart can show loss going up and down while compiler-verified quality stays exactly the same.
How far are we from a useful oktopai specialist?
There are two different finish lines. The engineering pipeline needs to work, and the model needs to beat the unchanged base model on new TypeScript tasks. We have mostly reached the first finish line. We have not reached the second.
| Gate | Current state | Meaning |
|---|---|---|
| Local training and adapter loading | Complete | The 3B student can be trained and run on the RTX 5080. |
| Trusted data and executable verification | Complete | Compiler checks, replay, manifests, and a machine heartbeat exist. |
| Useful student quality | Not met | The best teacher-trained student scored 57/200 versus 94/200 for the base. |
| Release confidence | Not met | No candidate has cleared the required cross-family and regression gates. |
In other words, oktopai has a working research instrument, not yet a useful TypeScript product specialist. The trajectory model is a promising separate line: one synthetic repository gate reached 80/80 valid replays and a multi-file gate reached 60/60. That proves the inspect/diagnose/edit/verify format can be learned, but those synthetic results do not yet prove broad generalization.
The practical distance is therefore one quality breakthrough, not one missing piece of infrastructure. A candidate becomes useful when it beats the base on the fixed 200-task suite, improves rather than sacrifices several task families, and avoids a regression on general coding. Until then, every adapter is deliberately kept as an experiment that can be removed.
What helped
The compiler gate changed the quality of the discussion. An early external teacher pilot accepted only 69 of 500 answers. That stopped a tempting but bad idea: spending more money to generate thousands of similarly weak examples.
The trajectory work helped in another way. Instead of asking only for a final answer, I built a contract for inspect, diagnose, edit, observe, retry, and final events. The verifier replays edits in a temporary repository and checks the compiler result. A synthetic trajectory candidate reached 80/80 validity and replay on one held-out gate, then 60/60 on a multi-file gate. That proves the format and verifier can work, although it is not yet proof of broad coding ability.
I also added a bounded patch-emitter stage for large files. It asks the model for compact exact replacements instead of reproducing an entire file. The corpus replay is clean, but four generations—v5, v7, v9, and v10—landed at the same 15/20 compiler-success rate on the unchanged local gate. The remaining five failures all chose an unchanged downstream canvas condition instead of editing the actual numeric source expression.
What did not help
Low training loss did not predict useful generalization. The student learned the shape of the teacher answers, but not reliably the task distribution in the held-out suite.
Adding external examples did not help either. Even after capping families and checking for overlap, the new student fell from 57/200 to 31/200. The failures clustered in mapped types, null narrowing, overloads, and record dictionaries. That looks more like answer-style or objective interference than a simple lack of examples.
This is a useful failure. It tells me to stop treating corpus size as progress. The next experiment should isolate task families or change the training objective, and it should use the same executable 200-task gate. A new adapter must beat the base and the current student before it earns any packaging step.
Where this leaves oktopai
The project has a working local training loop, a persistent machine heartbeat, compiler-backed corpus audits, replayable trajectory verification, and a clear promotion rule. It does not yet have a TypeScript specialist that has earned promotion.
That distinction is the real progress of this week. The experiment is now good at saying no to attractive but unsupported results. Tomorrow’s work can focus on a smaller, more controlled question: can a revised objective teach one failure family without damaging the others?
Discussion
Replies are loaded from the public Mastodon thread for this article.
Loading replies from Mastodon...