Self-Consistency Prompting: Sampling Multiple Answers for Higher Accuracy

The Problem with Single-Sample Answers

When you ask an LLM a reasoning question once, you get one answer. That answer might be correct, or it might reflect a reasoning misstep that the model happened to take on that particular generation. The stochastic nature of language models means that running the same prompt multiple times with temperature above zero produces different reasoning chains — and sometimes different final answers.

Self-consistency prompting exploits this property deliberately. Instead of trusting a single output, you sample multiple responses, extract the final answer from each, and take a majority vote. The intuition is simple: correct reasoning paths tend to converge on the same answer, while incorrect paths scatter across different wrong answers.

How Self-Consistency Works

The technique has three steps:

flowchart TD
    SPEC(["Task spec"])
    SYSTEM["System prompt<br/>role plus rules"]
    SHOTS["Few shot examples<br/>3 to 5"]
    VARS["Variable injection<br/>Jinja or f-string"]
    COT["Chain of thought<br/>or scratchpad"]
    CONSTR["Output constraint<br/>JSON schema"]
    LLM["LLM call"]
    EVAL["Offline eval<br/>LLM as judge plus regex"]
    GATE{"Score over<br/>threshold?"}
    COMMIT(["Promote to prod<br/>version pinned"])
    REVISE(["Revise prompt"])
    SPEC --> SYSTEM --> SHOTS --> VARS --> COT --> CONSTR --> LLM --> EVAL --> GATE
    GATE -->|Yes| COMMIT
    GATE -->|No| REVISE --> SYSTEM
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style EVAL fill:#f59e0b,stroke:#d97706,color:#1f2937
    style COMMIT fill:#059669,stroke:#047857,color:#fff

1. Sample — generate N responses to the same chain-of-thought prompt using temperature > 0
2. Extract — parse the final answer from each response
3. Aggregate — select the answer that appears most frequently

Research from Google Brain showed that this approach improves accuracy on arithmetic, commonsense, and symbolic reasoning benchmarks by 5 to 15 percentage points over standard chain-of-thought, with no changes to the prompt itself.

Python Implementation

import openai
from collections import Counter

client = openai.OpenAI()

def self_consistency_query(
    question: str,
    n_samples: int = 5,
    temperature: float = 0.7,
    model: str = "gpt-4o",
) -> dict:
    """Query an LLM with self-consistency voting."""
    prompt = (
        "Think step by step, then provide your final answer "
        "on the last line in the format: ANSWER: <your answer>\n\n"
        f"Question: {question}"
    )

    responses = []
    for _ in range(n_samples):
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=temperature,
        )
        responses.append(response.choices[0].message.content)

    # Extract final answers
    answers = []
    for resp in responses:
        for line in resp.strip().split("\n")[::-1]:
            if "ANSWER:" in line.upper():
                answer = line.split(":", 1)[1].strip()
                answers.append(answer)
                break

    # Majority vote
    if not answers:
        return {"answer": None, "confidence": 0.0, "samples": responses}

    vote_counts = Counter(answers)
    best_answer, best_count = vote_counts.most_common(1)[0]
    confidence = best_count / len(answers)

    return {
        "answer": best_answer,
        "confidence": confidence,
        "vote_distribution": dict(vote_counts),
        "total_samples": len(answers),
    }

result = self_consistency_query(
    "If a train travels 120 km in 2 hours, then stops for 30 minutes, "
    "then travels 90 km in 1.5 hours, what is its average speed for "
    "the entire journey including the stop?"
)
print(f"Answer: {result['answer']}")
print(f"Confidence: {result['confidence']:.0%}")
print(f"Votes: {result['vote_distribution']}")

Confidence Scoring and Thresholds

The vote distribution gives you a natural confidence metric. If all five samples agree, confidence is 100 percent and you can trust the answer. If votes split 3-2, confidence is 60 percent and you might want to escalate to a human or sample more responses.

def adaptive_self_consistency(
    question: str,
    confidence_threshold: float = 0.8,
    initial_samples: int = 5,
    max_samples: int = 15,
) -> dict:
    """Adaptively sample until confidence threshold is met."""
    all_answers = []
    batch_size = initial_samples

    while len(all_answers) < max_samples:
        result = self_consistency_query(
            question, n_samples=batch_size, temperature=0.7
        )
        # Accumulate answers from this batch
        for ans in result["vote_distribution"]:
            all_answers.extend(
                [ans] * result["vote_distribution"][ans]
            )

        vote_counts = Counter(all_answers)
        best_answer, best_count = vote_counts.most_common(1)[0]
        confidence = best_count / len(all_answers)

        if confidence >= confidence_threshold:
            return {
                "answer": best_answer,
                "confidence": confidence,
                "total_samples": len(all_answers),
            }

        batch_size = 3  # smaller incremental batches

    # Return best answer even if threshold not met
    vote_counts = Counter(all_answers)
    best_answer, best_count = vote_counts.most_common(1)[0]
    return {
        "answer": best_answer,
        "confidence": best_count / len(all_answers),
        "total_samples": len(all_answers),
        "threshold_met": False,
    }

This adaptive approach starts with 5 samples and only generates more if the confidence is below the threshold. It avoids wasting tokens on easy questions where 5 samples all agree.

When Self-Consistency Helps Most

Self-consistency shines on tasks with a single correct answer — math problems, factual questions, classification tasks, and logical puzzles. It is less useful for open-ended generation like creative writing, where there is no single "correct" output to converge on.

The technique also works best when combined with chain-of-thought prompting. Without reasoning steps, the model tends to produce the same answer repeatedly regardless of temperature, making voting trivial. The reasoning chain introduces the variation that self-consistency needs to be effective.

FAQ

How many samples should I use for self-consistency?

Five samples is a strong starting point for most tasks. Research shows diminishing returns beyond 10 to 15 samples. For production systems, the adaptive approach — starting small and only adding samples when confidence is low — gives the best balance between accuracy and cost.

Does self-consistency work with low temperature settings?

It requires temperature above zero to produce diverse reasoning paths. Temperature 0.5 to 0.8 is the sweet spot. Too low and all samples produce identical outputs. Too high and the reasoning quality degrades, introducing noise into the voting process.

Can I combine self-consistency with other prompting techniques?

Yes. Self-consistency is a meta-technique that wraps around any prompt strategy. You can combine it with few-shot prompting, role prompting, or retrieval-augmented prompting. The underlying prompt determines the quality of individual samples, and self-consistency improves the reliability of the final answer selection.

---

#PromptEngineering #SelfConsistency #Accuracy #LLM #Python #AgenticAI #LearnAI #AIEngineering