When to Fine-Tune vs Use Prompting vs RAG: A Decision Framework

The Three Approaches to LLM Customization

Every team building with LLMs eventually hits the same question: the base model is close but not quite right for our use case. The answer is not always fine-tuning. In fact, fine-tuning is often the most expensive and least necessary option. The three primary approaches — prompt engineering, retrieval-augmented generation (RAG), and fine-tuning — each solve different problems, and choosing wrong wastes months of engineering time.

Prompt engineering modifies the model's behavior through instructions alone. RAG augments the model's knowledge by retrieving external documents at query time. Fine-tuning changes the model's weights by training on custom data. Understanding where each approach excels is the key to shipping on time and on budget.

The Decision Tree

Start with the simplest approach and escalate only when necessary.

flowchart LR
    Q(["User query"])
    EMB["Embed query<br/>text-embedding-3"]
    VEC[("Vector DB<br/>pgvector or Pinecone")]
    RET["Top-k retrieval<br/>k = 8"]
    PROMPT["Augmented prompt<br/>system plus context"]
    LLM["LLM generation<br/>Claude or GPT"]
    CITE["Inline citations<br/>and page anchors"]
    OUT(["Grounded answer"])
    Q --> EMB --> VEC --> RET --> PROMPT --> LLM --> CITE --> OUT
    style EMB fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style VEC fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style OUT fill:#059669,stroke:#047857,color:#fff

Step 1 — Can prompt engineering solve it? If you need the model to follow a specific output format, adopt a particular tone, or handle a well-defined task, prompt engineering is almost always sufficient. Few-shot examples in the prompt can teach surprisingly complex patterns.

Step 2 — Does the model lack domain knowledge? If the model needs access to proprietary data, recent information, or a large knowledge base, RAG is the right choice. RAG does not change the model — it feeds relevant context into the prompt at query time.

Step 3 — Does the model need a fundamentally different behavior pattern? If you need the model to consistently produce a specific style, follow complex domain-specific reasoning patterns, or achieve latency that a long prompt cannot deliver, fine-tuning is justified.

def recommend_approach(
    needs_custom_knowledge: bool,
    knowledge_base_size: str,
    needs_behavior_change: bool,
    labeled_examples_available: int,
    latency_sensitive: bool,
    budget: str,
) -> str:
    # Step 1: Try prompting first
    if not needs_custom_knowledge and not needs_behavior_change:
        return "prompt_engineering"

    # Step 2: Knowledge gap → RAG
    if needs_custom_knowledge and knowledge_base_size in ("medium", "large"):
        if not needs_behavior_change:
            return "rag"
        if labeled_examples_available < 100:
            return "rag_with_prompt_engineering"

    # Step 3: Behavior change with sufficient data → fine-tune
    if needs_behavior_change and labeled_examples_available >= 100:
        if latency_sensitive:
            return "fine_tune_smaller_model"
        return "fine_tune"

    # Hybrid approaches
    if needs_custom_knowledge and needs_behavior_change:
        return "fine_tune_plus_rag"

    return "prompt_engineering"

Cost Comparison

The cost differences are dramatic. Here is a rough comparison for a customer support assistant use case processing 10,000 queries per month.

Prompt Engineering: Zero upfront cost. Per-query cost depends on prompt length. A 2,000-token system prompt with few-shot examples costs roughly $0.01-0.03 per query with GPT-4o. Monthly total: $100-300.

RAG: Infrastructure cost for a vector database (Pinecone, pgvector, or Qdrant) plus embedding generation. Typical monthly cost: $50-200 for infrastructure plus $150-400 for query costs with retrieved context. Monthly total: $200-600.

Fine-Tuning: Training cost of $5-50 per run depending on dataset size and model. But inference on a fine-tuned model is often cheaper per query because you eliminate long prompts. Training: $5-50 per iteration. Monthly inference: $50-200. Monthly total: $100-250 after initial training.

When Fine-Tuning Actually Wins

Fine-tuning delivers clear ROI in three scenarios.

Latency reduction. A fine-tuned model that has internalized your formatting rules does not need a 2,000-token system prompt. Eliminating that prompt reduces time-to-first-token by 30-60%.

Consistent style and tone. When you need every output to match a brand voice or follow a precise clinical documentation format, fine-tuning encodes that pattern into the weights rather than relying on instructions the model might occasionally ignore.

Cost at scale. If you process millions of queries, the per-query savings from shorter prompts compound. A fine-tuned GPT-4o-mini can replace a heavily prompted GPT-4o at a fraction of the cost.

# Example: calculating break-even point for fine-tuning
def fine_tuning_break_even(
    prompt_cost_per_query: float,   # e.g., $0.025
    ft_cost_per_query: float,       # e.g., $0.008
    training_cost: float,           # e.g., $25.00
    queries_per_month: int,
) -> dict:
    savings_per_query = prompt_cost_per_query - ft_cost_per_query
    if savings_per_query <= 0:
        return {"recommendation": "Do not fine-tune", "reason": "No cost savings"}

    break_even_queries = training_cost / savings_per_query
    months_to_break_even = break_even_queries / queries_per_month

    return {
        "savings_per_query": f"${savings_per_query:.4f}",
        "break_even_queries": int(break_even_queries),
        "months_to_break_even": round(months_to_break_even, 1),
    }

# Example: 10K queries/month
result = fine_tuning_break_even(0.025, 0.008, 25.0, 10_000)
# break_even_queries: 1471, months_to_break_even: 0.1

Common Mistakes

Mistake 1: Fine-tuning when RAG would suffice. If the model gives correct answers when you paste the relevant document into the prompt, you need RAG — not fine-tuning. Fine-tuning does not reliably inject factual knowledge.

Mistake 2: Using RAG when prompting works. If the information the model needs can fit in a few-shot prompt and does not change frequently, a vector database adds unnecessary complexity.

Mistake 3: Skipping evaluation. Whichever approach you choose, you need a test set to measure whether it actually improved performance. Build the eval first, then choose the approach.

FAQ

How many training examples do I need for fine-tuning?

OpenAI recommends a minimum of 10 examples but suggests 50-100 for noticeable improvements. For complex domain-specific tasks, 500-1,000 high-quality examples typically produce strong results. Quality matters far more than quantity — 200 carefully curated examples usually outperform 2,000 noisy ones.

Can I combine fine-tuning and RAG?

Yes, and this is often the optimal approach for complex applications. Fine-tune the model to learn your output format and reasoning style, then use RAG to inject current knowledge at query time. The fine-tuned model processes retrieved context more effectively because it already understands your domain patterns.

Should I fine-tune an open-source model or use the OpenAI fine-tuning API?

If you need data privacy, full control over the model, or plan to run inference on your own infrastructure, fine-tune an open-source model like Llama or Mistral. If you want the fastest path to production with managed infrastructure, use the OpenAI API. The API handles training, hosting, and scaling — you just provide data and pay per token.

---

#FineTuning #RAG #PromptEngineering #LLM #DecisionFramework #AgenticAI #LearnAI #AIEngineering