From Model Selection to First Run

TL;DR

• Model selection is not about finding the "best" model on a public leaderboard; it is about finding the first model worth testing against your specific production constraints.
• Never start with fine-tuning or complex multi-agent frameworks; start with a 20-example golden baseline and a single, unadorned prompt.
• Evaluate candidates across three tiers: frontier commercial APIs for rapid capability ceilings, mid-tier models for balanced production workloads, and compact open-weight models for zero-egress local serving.
• Use the first run to diagnose the root constraint: is the bottleneck missing domain knowledge (solve with RAG), inconsistent formatting (solve with fine-tuning/grammar), or reasoning failure (solve with a larger model)?

Abstract

Selecting an AI model in modern software engineering often resembles falling into an academic research trap.

Engineers spend weeks browsing Hugging Face leaderboards, reading arXiv preprints, comparing MMLU benchmarks, debating quantization algorithms, and analyzing synthetic arena scores. Every week brings a new model release claiming state-of-the-art performance on a composite benchmark.

None of that replaces running a candidate model on your actual task.

Public leaderboards measure general capabilities across broad academic tests. They do not tell you whether a model can extract line items from your company's idiosyncratic billing PDFs, adhere to your strict JSON schema, or respond within your 800-millisecond latency budget on an AWS g5.2xlarge instance.

The objective of early model selection is not to find your final production architecture. It is to find the first candidate worth testing, execute a disciplined baseline run, inspect the failure modes, and determine whether the task requires prompting, retrieval, fine-tuning, or a different model entirely.

This guide outlines an actionable, code-first engineering workflow to navigate from task definition and candidate filtering to running your first baseline evaluation.

Table of Contents

• The Leaderboard Trap
• Step 1: Define the Operational Envelope
• Step 2: The Three-Tier Candidate Filtering Matrix
• Step 3: Curating the 20-Example Diagnostic Test Set
• Step 4: The Minimal Baseline Test Harness (Code Implementation)
• Step 5: Output Inspection & Error Taxonomy
• Step 6: The Architectural Decision Fork (Prompting vs. RAG vs. Fine-Tuning)
• Summary
• Next Steps

The Leaderboard Trap

Leaderboards create the illusion that model selection is a one-dimensional ranking problem. If Model X scores 84.2 on HumanEval and Model Y scores 81.8, teams assume Model X is the correct choice.

In production, models fail for reasons leaderboards never measure:

+---------------------------------------------------------------+
|                 Production Constraints vs. Evals              |
|                                                               |
|   What Leaderboards Measure     What Production Requires      |
|   -------------------------     ------------------------      |
|   - Synthetic reasoning (MMLU)  - Strict JSON schema validity |
|   - Multi-choice accuracy       - p95 latency < 1,000ms       |
|   - Raw Python generation       - Zero data egress (HIPAA/SOC)|
|   - Massive context window      - Predictable cost per query  |
|   - General academic knowledge  - Deterministic tool calling  |
+---------------------------------------------------------------+

A 405-billion-parameter frontier model may achieve a slightly higher reasoning score, but if it takes 8 seconds to stream a response and costs $0.04 per call, it is unviable for an interactive autocomplete feature. Conversely, a 7B model running on local GPUs at 120 tokens per second may solve the problem completely if properly formatted.

Stop optimizing for leaderboards. Optimize for your operational envelope.

Step 1: Define the Operational Envelope

Before opening a terminal or generating an API key, write down the non-negotiable operational boundaries of the task:

# Operational Envelope Specification

1. Input / Output Schema:
   - Input: Unstructured customer support transcript (avg 800 tokens).
   - Output: Strict JSON containing intent_category (enum), sentiment_score (float), and action_items (list[str]).

2. Latency Budget:
   - Target Time-to-First-Token (TTFT): < 600ms.
   - Total Completion Time: < 2,000ms.

3. Privacy & Compliance Boundaries:
   - Tier 1: Zero third-party data egress allowed (Requires on-prem / VPC open-weight deployment).
   - Tier 2: Commercial enterprise API permitted with Zero Data Retention (ZDR) agreement.

4. Volume & Financial Budget:
   - Expected Traffic: 250,000 requests / day.
   - Max Allowable Cost: $0.0015 per transaction ($375 / day cap).

Notice how these constraints immediately eliminate 90% of model options. If your financial cap is $0.0015 per transaction, frontier models like GPT-5.4 or Claude 4.5 Sonnet are financially impossible at 250k calls/day without extreme caching. You are immediately steered toward compact open-weight models (e.g., Qwen3.5-4B/9B, Qwen3.5-35B-A3B, or Llama 4 8B) hosted on dedicated compute.

Step 2: The Three-Tier Candidate Filtering Matrix

Filter candidates into three distinct functional tiers to run a comparative baseline:

+--------------------------------------------------------------------------+
|                     The Candidate Filtering Triad                        |
|                                                                          |
|  Tier 1: Frontier Cloud API   --> Establish the Capability Ceiling       |
|          (Claude 4.5 Sonnet / GPT-5.4)                                    |
|                                                                          |
|  Tier 2: Mid-Tier Efficient   --> Balanced Production Baseline           |
|          (Claude 4 Haiku / GPT-5-mini / Gemini 2.5 Flash)             |
|                                                                          |
|  Tier 3: Compact Open-Weight  --> Zero-Egress / Self-Hosted Option       |
|          (Qwen3.5 9B-35B / Llama 4 70B / Gemma 3 12B on vLLM)         |
+--------------------------------------------------------------------------+

1. The Frontier Benchmark (The Capability Ceiling)

Always test one frontier model (such as Claude 4.5 Sonnet or GPT-5.4) first.

The goal is not necessarily to deploy it; the goal is to establish whether the task is solvable at all using pure prompting. If a frontier model with 100+ billion parameters cannot solve your task with a clear prompt, a smaller 7B model will not solve it either without specialized fine-tuning or retrieval.

2. The Mid-Tier Efficient API

Select one lightweight, highly optimized cloud model (e.g., GPT-5-mini, Claude 4 Haiku, Gemini 2.5 Flash). These models cost 1/10th the price of frontier models and respond with sub-second latencies, serving as the commercial cost-efficiency baseline.

3. The Open-Weight Self-Hosted Candidate

Select one modern open-weight checkpoint that fits your deployment hardware:

• 1x Consumer / Cloud GPU (16GB–24GB VRAM): Qwen3.5-4B / Qwen3.5-9B or Llama 4 8B quantized (FP8 / AWQ / AutoRound) running on vLLM.
• 1x Enterprise GPU (40GB–80GB VRAM): Qwen3.5-35B-A3B MoE or Qwen3.6-27B running unquantized or in FP8.
• Multi-GPU Cluster (2x–4x H100): Llama 4 70B running tensor-parallel with EAGLE-3 speculative decoding.

Step 3: Curating the 20-Example Diagnostic Test Set

Do not start by evaluating 10,000 synthetic rows. Assemble a curated, human-inspected diagnostic dataset of exactly 20 examples:

### The 20-Example Composition

- 10 Standard Happy-Path Examples: Typical production queries with clear, unambiguous answers.
- 4 Complex / Verbose Edge Cases: Extremely long documents, messy formatting, multi-intent prompts.
- 3 Adversarial / Malformed Inputs: Empty strings, prompt injection attempts, out-of-domain noise.
- 3 Strict Constraint Boundary Cases: Inputs designed to test exact schema limits (e.g., zero-length lists, numerical boundaries).

Save these in a plain JSON Lines file (baseline_test_set.jsonl):

{"id": 1, "input": "Customer: Hi, I was double charged $49 on invoice #8841. Agent: Let me refund that for you.", "expected_category": "BILLING", "expected_action": "REFUND"}
{"id": 2, "input": "Customer: How do I export my data to Parquet format? Agent: Go to settings, click export, choose Parquet.", "expected_category": "TECHNICAL_SUPPORT", "expected_action": "DOCUMENTATION"}

Step 4: The Minimal Baseline Test Harness (Code Implementation)

Build a single, unadorned Python evaluation script. Avoid LangChain, LlamaIndex, or agent frameworks for this step. Use direct client calls with structured Pydantic validation:

import json
import time
from typing import List, Literal, Optional
from pydantic import BaseModel, Field
from openai import OpenAI

class TicketClassification(BaseModel):
    category: Literal["BILLING", "TECHNICAL_SUPPORT", "ACCOUNT_ACCESS", "GENERAL"]
    sentiment_score: float = Field(..., ge=-1.0, le=1.0)
    primary_action: str = Field(..., max_length=100)

def run_baseline_evaluation(
    client: OpenAI,
    model_name: str,
    test_file: str = "baseline_test_set.jsonl"
):
    results = []
    
    with open(test_file, "r") as f:
        test_cases = [json.loads(line) for line in f]

    print(f"
--- Running Baseline Evaluation: {model_name} ---")
    
    for case in test_cases:
        start_time = time.perf_counter()
        schema_valid = False
        error_msg = None
        parsed_output = None
        
        try:
            completion = client.beta.chat.completions.parse(
                model=model_name,
                messages=[
                    {
                        "role": "system",
                        "content": "Analyze the customer support conversation. Return strict JSON matching the schema."
                    },
                    {"role": "user", "content": case["input"]}
                ],
                response_format=TicketClassification,
                temperature=0.0,
            )
            elapsed_ms = (time.perf_counter() - start_time) * 1000
            parsed_output = completion.choices[0].message.parsed
            schema_valid = True
            prompt_tokens = completion.usage.prompt_tokens
            completion_tokens = completion.usage.completion_tokens
            
        except Exception as e:
            elapsed_ms = (time.perf_counter() - start_time) * 1000
            error_msg = str(e)
            prompt_tokens = 0
            completion_tokens = 0

        results.append({
            "test_id": case["id"],
            "model": model_name,
            "latency_ms": round(elapsed_ms, 1),
            "schema_valid": schema_valid,
            "prompt_tokens": prompt_tokens,
            "completion_tokens": completion_tokens,
            "output": parsed_output.model_dump() if parsed_output else None,
            "error": error_msg
        })

    return results

# Run across candidate models
client = OpenAI()
models = ["gpt-5-mini", "gpt-5.4"]
summary = {}

for m in models:
    res = run_baseline_evaluation(client, m)
    valid_count = sum(1 for r in res if r["schema_valid"])
    avg_latency = sum(r["latency_ms"] for r in res) / len(res)
    summary[m] = {
        "schema_pass_rate": f"{valid_count}/{len(res)}",
        "avg_latency_ms": round(avg_latency, 1)
    }

print("
=== Baseline Results Summary ===")
print(json.dumps(summary, indent=2))

This 60-line script provides immediate, empirical facts: exactly how long each model took, whether the output adhered to your Pydantic schema, and how many tokens were consumed.

Step 5: Output Inspection & Error Taxonomy

Do not just look at the aggregate pass rate. Open the JSON outputs and classify the failures into root categories:

# Diagnostic Error Taxonomy

1. SCHEMA_FAILURE:
   - Output contained markdown fences (` ```json `), trailing commas, or omitted required fields.

2. REASONING_FAILURE:
   - Model correctly parsed the text but arrived at the wrong logical conclusion (e.g., categorized a billing dispute as general feedback).

3. HALLUCINATION:
   - Model invented information not present in the input text (e.g., generated a fictional invoice ID).

4. CONTEXT_OVERFLOW / TRUNCATION:
   - Long documents caused the model to lose track of early instructions or truncate output mid-sentence.

5. LATENCY_SPIKE:
   - Model took more than 3x the average latency on complex inputs.

Step 6: The Architectural Decision Fork

Once you inspect the error taxonomy from your baseline run, you face an architectural fork. Choose your next move based on the specific failure mode observed:

+--------------------------------------------------------------------------+
|                       The Optimization Decision Fork                     |
|                                                                          |
|   Failure Mode Observed                     Engineering Remedy           |
|   ---------------------                     ------------------           |
|   - Inconsistent formatting or enum keys -> Structured Outputs / Zod     |
|   - Missing factual / proprietary knowledge -> RAG (Vector / Lexical)    |
|   - Inconsistent reasoning despite clear prompt -> Step-Up Model Tier   |
|   - Accurate reasoning, but cost/latency too high -> LoRA Fine-Tuning    |
|   - Task fundamentally too ambiguous       -> Human-in-the-Loop Review  |
+--------------------------------------------------------------------------+

Fork 1: The Prompt / Schema Path

If the model gets the right answer 80% of the time but occasionally misspells an enum key or drops a required field, do not fine-tune. Enforce constrained decoding via native structured output APIs or provide three clear few-shot examples in the system prompt.

Fork 2: The Retrieval (RAG) Path

If the model hallucinates internal company policies, cites outdated pricing, or struggles with specific documentation, your issue is not model intelligence. It is information deficiency. Build a tenant-isolated RAG pipeline to inject the ground-truth text chunks directly into the prompt context.

Fork 3: The Fine-Tuning Path

If a mid-tier model (e.g., Qwen3.5-9B or Llama 4 8B) understands the task but struggles with specialized domain vocabulary, or if you need to eliminate 1,500 tokens of system prompt instructions to reduce latency, fine-tune. Collect 500 verified input-output pairs and train a LoRA adapter. See From Prompts to Practice: Instruction Tuning Qwen3 with Transformers.

Fork 4: The Step-Up Path

If your candidate model repeatedly fails basic logical deductions on complex multi-step instructions, you have hit the reasoning ceiling of that parameter scale. Step up to a larger model tier before writing hundreds of lines of fragile prompt engineering hacks.

Summary

Model selection is not an academic competition; it is an engineering discipline driven by operational constraints.

By defining your latency, cost, and privacy boundaries upfront, curating a tight 20-example diagnostic dataset, executing a direct baseline evaluation harness, and systematically diagnosing failure modes, you make defensible architectural choices backed by real telemetry.

Start simple. Measure cleanly. Inspect the failures. Optimize only when empirical evidence demands it.

Next Steps

1. Write down your task's Operational Envelope (schema, latency budget, privacy tier, max cost per transaction).
2. Curate a 20-example diagnostic JSONL dataset containing edge cases and adversarial inputs.
3. Run the baseline evaluation script against one frontier model and one compact open-weight model today.

If you want an experienced team to benchmark candidate models on your data, optimize self-hosted vLLM throughput, or build automated evaluation suites, talk to us. We provide AI engineering and model evaluation consulting that turns research uncertainty into shipped production systems.

Frequently asked questions

How should an engineering team choose its first candidate AI model for a new task?

Model selection is not about finding the universally optimal model on public leaderboards; it is about finding the fastest candidate worth testing against specific operational constraints. Define the input-output schema, p95 latency budget, privacy boundaries, and expected token volume first, then test two or three models representing distinct capability tiers (e.g., one frontier cloud model and one compact open-weight model).

What should a baseline test harness for an AI model look like?

The first run should be minimalist: a 20-example evaluation set, zero complex multi-agent frameworks, default temperature parameters (0.0 to 0.2), and strict output logging that records input prompts, raw completions, prompt/completion token counts, wall-clock latency, and deterministic validation flags.

When should an AI engineering team move from prompt engineering to RAG or fine-tuning?

Add RAG when the model lacks access to private, rapidly changing, or proprietary domain knowledge. Fine-tune when the model already understands the knowledge but repeatedly fails on specialized output syntax, strict JSON schemas, or when you need to compress a 70B model's task performance into a low-latency 7B or 14B model. Switch models entirely if the base reasoning capability cannot handle the task logic.

What are the trade-offs between frontier cloud APIs and self-hosted open-weight models?

Frontier APIs (Claude 4.5 Sonnet, GPT-5.4) offer maximum general reasoning and zero infrastructure maintenance, but introduce variable network latency, recurring per-token SaaS costs, and third-party data egress. Self-hosted open-weight models (Qwen3.5, Llama 4 running on vLLM with speculative decoding) ensure complete data privacy, zero egress, and deterministic throughput, but require GPU memory management and hardware provisioning.

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.