Small Models, Big Impact: The Quiet Shift From Frontier Models to Specialized Ones

Your cost dashboard says 78% of your LLM spend goes on three production tasks. Extracting fields from documents, classifying tickets, translating product descriptions. None of those need GPT-4. You knew that. You're paying for it anyway, because nobody got around to actually evaluating the smaller models.

You're not alone. I've been that team. It's how I learned the lesson.

What's actually changed

A year ago, "smaller model" meant "stupider model." That stopped being true. Models in the 3-8B parameter range now match (sometimes beat) frontier models on narrow tasks they've been fine-tuned for. The shift isn't about raw capability. It's about specialisation paying off again.

Think of the difference between a GP and a cardiologist. The GP is impressive across thousands of conditions, but you don't want them doing heart surgery. The cardiologist isn't smarter. They're aimed.

A practical specialisation pattern

# Step 1: collect 500-2000 examples from your production traffic
examples = pull_logged_traffic(task="ticket_classification", limit=2000)

# Step 2: fine-tune a small model on those examples
fine_tune_job = client.fine_tunes.create(
    base_model="phi-3-mini-4k-instruct",
    training_file=upload(format_as_jsonl(examples)),
    n_epochs=3,
    suffix="ticket-classifier-v1",
)

# Step 3: route by task at request time
def classify(ticket):
    if uses_legacy_path(ticket):
        return frontier_model.classify(ticket)   # fallback for edge cases
    return small_model.classify(ticket, model="ticket-classifier-v1")

Why it works

The fine-tuned small model sees a narrow distribution. Your tickets, your taxonomy, your edge cases. It is no longer trying to also write code, plan trips, and tell jokes. The parameters it does have are aimed entirely at your problem. Cost drops 5-20x per token, latency drops 3-10x, and accuracy often goes up because the model isn't distracted. Once you see the numbers it's hard to go back.

When to specialise

Any production task with high volume, a stable shape, and access to enough real examples. About 500 minimum to start. Classification, extraction, routing, summarisation with a fixed structure. Those pay for the fine-tuning project on cost alone within a quarter. I've seen the maths work out within weeks.

When to stay on the frontier model

Low-volume tasks where engineering time costs more than the model bill. Tasks with high variance in input distribution where you'd need to retrain frequently. Anything novel or exploratory where you don't yet know the contract. The frontier model is your prototyping environment. Use it as one.

The routing pattern

flowchart LR
    Req[Incoming request] --> R{Task type<br/>in scope?}
    R -- Yes --> Small["Fine-tuned small model<br/>cheap + fast"]
    R -- No --> Big[Frontier model<br/>generalist]
    Small --> Q{Confidence<br/>> threshold?}
    Q -- Yes --> Done[Return]
    Q -- No --> Big
    Big --> Done

Conclusion

Pull last week's LLM spend and group it by task. The top three are almost certainly candidates for specialisation. Pick the one with the most stable inputs and start collecting training examples this sprint. You're already producing them. They're just sitting unused in your logs.