auto-learn
An ESL writing tool that withholds the better word until you open the card that teaches it.
- result
- An evaluation harness caught a prompt change that improved one score while degrading another. The app also keeps writing on the device when syncing the word bank.
- my contribution
- Built the server-enforced learning gate, validated the LLM judge against labeled examples, and implemented word-bank sync.
- 2026–Present
- web
how it works
auto-learn fixes a sentence and teaches the word that fixed it. Typos, spacing, and punctuation are corrected inline and never discussed. Anything that is a real choice (grammar, word choice, register, wordiness) is marked but held back: the API drops the replacement from its response and keeps it server-side, so the only way to read the stronger word is to open the card that explains it.
The server enforces the withholding. The field is absent from the payload rather than flagged inside it, which means it is not sitting in the network tab either. Accepting a word banks it, and the next request tells the model which words the writer has already met so it reaches for one they have not.
Most of the work went into knowing whether the model was any good. An eval harness scores both model calls against a committed baseline using deterministic checks and an LLM judge, and the judge was validated against hand-labelled cases before it was trusted, using Cohen’s kappa. It paid for itself by catching a prompt change that raised one score while quietly lowering another.
engineering details
- Enforced the gate on the server: the replacement is dropped from the proposal and released only when a card is opened, so it never reaches the browser early.
- Validated the LLM judge against hand-labelled cases before trusting it, then held every prompt change to a committed baseline at six runs per case.
- Grounded word senses in a local WordNet database rather than a dictionary API, so the part the cards are built on needs no network and no key.
- Set the synthesis output format on the voice path, which cut each spoken word from 15,090 bytes to 3,989.
- Split tests across three runners by filename to keep an ESM-only AI SDK and a CommonJS API in one repository: 498 tests across the three packages.
- Synced the word bank to an account without uploading the writing: the sentence a word was met in has no column on the server and never leaves the browser.
in the product
The Gate
Tier-two suggestions arrive as a mark and a teaser. The replacement stays on the server until the reader opens the card that teaches it.
Judged Against a Baseline
Deterministic scorers and a validated LLM judge run both model calls against a committed baseline, so a prompt change has to prove it helped.
A Bank That Earns Its Reward
Words are banked only when chosen, and the product congratulates reuse, firing only on evidence the writer used the word again unprompted.