The Think Gate

Modern models can "think" before they answer: they work through the problem in a hidden scratchpad first. That makes them much better at maths, trade-offs and diagnosis. It also makes every answer slower, often several times slower.

Most apps put a toggle in front of the user. Most users never touch it, or leave it on and wonder why everything is slow. The better design makes the decision for them.

Psychologists describe two modes of thought: a fast, intuitive one and a slow, deliberate one. A think gate gives your product the fast mode as a separate, very small model. It reads the message, makes one typed decision (direct or think) and routes it. The big model then answers in the right mode.

I expected the model choice to matter most. It mattered less than the definition. My first rubric treated "Can this model run on that machine?" as a simple lookup. The person using the assistant called it hard, and they were right: feasibility and sizing questions get confidently wrong quick answers. Fixing the definition lifted the big model's labelling agreement with the user from 63% to 92%, before any training.

• thinkCan a 120B model run on a single 128 GB box, and how fast?
• thinkThe lawyer sent a new draft. Please check it against the letter of intent.
• thinkCheck this payment split: 5,000 / 10,000 / 15,000.
• thinkShould we move these scripts to the NAS, since the files live there?
• directWhat's the folder path for this project?
• directWhat did you mean by "OTA" earlier?
• directHow do I set up my router as a bridge?
• directCan we add an insurance clause for the contractor?

Rule of thumb from the final rubric: think when a quick answer is likely to be wrong or shallow (feasibility, sizing, decisions, document review, any money arithmetic, diagnosis). Answer directly for lookups, paths, status, "what did you mean", standard how-tos, and single edits.

All gates were tested on the same 148 real questions taken from one person's assistant sessions. Half needed thinking and half didn't. I labelled them, and the user confirmed the 30 most borderline labels. Everything ran locally on one NVIDIA GB10 desktop AI box (128 GB unified memory).

"Needless thinking" is the share of all messages that were simple but got sent to the slow path. It's the cost users feel most, because every one is a simple question answered slowly.

• It never waits in line. Asking the main model "is this hard?" is accurate, but that call queues behind whatever the model is already generating. Many local servers answer one request at a time, so the check can take seconds. A 4 GB gate on its own process stays at ~30 ms.
• It works with any main model. Swap the chat model, or switch engines, and the gate keeps working.
• It's cheap to teach. We fine-tuned it in about 12 minutes on 1,627 example questions, starting from the released checkpoint rather than from scratch (section 06).
• It returns a probability, not prose. Kev-style decision models answer a typed choice question with a probability for each option, so there's nothing to parse and no chance of a rambling reply.

Yes, this is real model training, but only of the small 0.8B gate. The main assistant model is never touched. And it's the light kind of training, called LoRA: the gate's own weights stay frozen, and we train two small add-ons that sit on top of them.

1. Settle the definition. Write the think/direct rubric and check it against real questions the product owner has reviewed (30 here). This step moved results more than anything else.
2. Make the training set locally. The main model wrote about 1,070 new questions in the user's own style and topics, and relabelled 947 older ones under the corrected rubric. Every question was labelled twice, independently, and only agreements were kept: 1,915 questions, split 1,627 for training and 288 for checking. None of the 148 test questions were included.
3. Write each one as a typed decision. The training record is the same shape the gate answers at run time, plus the right label:

   {"state": "Can a 120B model run on one 128 GB box, and how fast?",
    "questions": {"mode": {"type": "choice",
      "instructions": "Should the assistant reason step by step first?",
      "criteria": {"think": "feasibility, sizing, decisions, money...",
                   "direct": "lookups, paths, status, how-tos..."},
      "label": "think"}}}

4. Train from the released checkpoint, not from scratch. One command, on the same GB10 box, beside the running assistant:

   python -m kev.train --data train.jsonl   --base Qwen/Qwen3.5-0.8B-Base --init_from jaredpalmer/kev-0.8b   --epochs 2 --lr 2e-5 --batch 1 --accum 8   --dtype bf16 --checkpointing 1 --device cuda --out runs/think-gate

   Starting from the released model keeps what it already knows about decisions. Kev's README reports a from-scratch run on another task falling to 0.33, against 0.83 when starting from the checkpoint.
5. Serve it the same way. python -m kev.serve --run runs/think-gate exposes the same /v1/systemone API, so the product's code doesn't change between the stock and tuned gate.

What it is not: it isn't training the main chat model, it isn't full fine-tuning (the 0.8B base stays frozen), and it isn't only prompt writing, although the rubric mattered most. For contrast, the Needle gate in the results table had no model training at all: we trained only a simple linear classifier on top of Needle's frozen embeddings, which is why better labels couldn't lift it.

1. Collect 150 real messages from your users or your own logs, and label them think or direct. Have the product owner check the borderline ones. This set is your test; never train on it.
2. Write the rubric from those labels, with concrete examples of each side. Expect your first version to be wrong in a specific direction.
3. Generate training data locally with your main model: a few thousand questions in your users' own style and topics. Label each one twice, independently, and keep only the agreements.
4. Fine-tune a small decision model from a released checkpoint. Kev (Apache-2.0, 0.8B to 27B, Jev-compatible API) worked best for us. Test it against your real set, not the generated one: our generated-set scores ran about 10 points higher.
5. Ship it behind two safety nets: a few high-precision rules first, and a "Rethink deeply" button on every fast answer. If the gate is down or slow, answer directly; never block the user on the check.

• Synthetic test scores flatter. The trained gates scored 93–96% on generated questions but 76–86% on real ones. Real messages are longer, mix several requests, and use your own jargon.
• An embedding with a simple classifier on top hit a ceiling. Better labels didn't move it (78% to 76%). It can't tell "how do I set up X" (a how-to) from "can X run on Y" (a judgement).
• Untuned small models hedge. Kev-0.8B as released put most messages near 50/50. After tuning, it was decisive, sometimes overconfident, which is why the Rethink button stays.
• On shared-memory AI boxes, "available memory" lies to the GPU. Training the 4B model failed with an out-of-memory error at 20 GB "available", because most of that was file cache the GPU allocator can't use.

The idea of small, fast "decision" models comes from TypeSafe's System One models and Jev, a hosted API. Open alternatives include Kev (used here), Respan's Span-02 (a much larger 26B model), Convai's Laya (421M, English-only, needs fine-tuning), and Cactus's Needle (a tiny tool-calling model, tested here as an embedding source).