Building a Hypothesis-Testing Agent: Scientific Method Applied to Data Analysis

Why Data Analysis Needs the Scientific Method

Most data analysis follows a dangerous pattern: look at the data, notice something interesting, and declare it a finding. This is a recipe for false discoveries. The scientific method — hypothesis first, then test — is the antidote.

A hypothesis-testing agent formalizes this process. It generates hypotheses about the data, designs tests to evaluate them, runs statistical analyses, interprets results, and iterates. This structured approach reduces false positives and produces reliable, actionable insights.

The Hypothesis Lifecycle

Every hypothesis moves through these stages:

flowchart LR
    PR(["PR opened"])
    UNIT["Unit tests"]
    EVAL["Eval harness<br/>PromptFoo or Braintrust"]
    GOLD[("Golden set<br/>200 tagged cases")]
    JUDGE["LLM as judge<br/>plus regex graders"]
    SCORE["Aggregate score<br/>and per slice"]
    GATE{"Score regress<br/>more than 2 percent?"}
    BLOCK(["Block merge"])
    MERGE(["Merge to main"])
    PR --> UNIT --> EVAL --> GOLD --> JUDGE --> SCORE --> GATE
    GATE -->|Yes| BLOCK
    GATE -->|No| MERGE
    style EVAL fill:#4f46e5,stroke:#4338ca,color:#fff
    style GATE fill:#f59e0b,stroke:#d97706,color:#1f2937
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style MERGE fill:#059669,stroke:#047857,color:#fff

from pydantic import BaseModel
from enum import Enum
from typing import Any

class HypothesisStatus(str, Enum):
    PROPOSED = "proposed"
    TESTING = "testing"
    SUPPORTED = "supported"
    REJECTED = "rejected"
    INCONCLUSIVE = "inconclusive"

class Hypothesis(BaseModel):
    id: str
    statement: str          # e.g., "Users who complete onboarding convert at 2x rate"
    null_hypothesis: str    # "Onboarding completion has no effect on conversion"
    variables: dict[str, str]  # independent and dependent variables
    test_method: str        # statistical test to use
    significance_level: float  # alpha, typically 0.05
    status: HypothesisStatus = HypothesisStatus.PROPOSED
    p_value: float | None = None
    effect_size: float | None = None
    conclusion: str | None = None

class ExperimentResult(BaseModel):
    hypothesis_id: str
    test_statistic: float
    p_value: float
    effect_size: float
    sample_size: int
    confidence_interval: tuple[float, float]
    interpretation: str

Hypothesis Generation

The agent observes data and generates testable hypotheses. The key constraint: hypotheses must be falsifiable and specific:

from openai import OpenAI
import json

client = OpenAI()

def generate_hypotheses(
    data_description: str,
    domain_context: str,
    num_hypotheses: int = 5,
) -> list[Hypothesis]:
    """Generate testable hypotheses from data observation."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are a data scientist generating hypotheses.
Requirements for each hypothesis:
1. Must be FALSIFIABLE — there must be a possible outcome that would disprove it
2. Must be SPECIFIC — state the expected direction and approximate magnitude
3. Must include a null hypothesis (the "no effect" baseline)
4. Must specify the appropriate statistical test
5. Must identify independent and dependent variables

Do NOT generate vague hypotheses like "X is related to Y".
DO generate specific ones like "X increases Y by at least 15% (p < 0.05)".

Return JSON array of hypothesis objects."""},
            {"role": "user", "content": (
                f"Data description: {data_description}\n"
                f"Domain context: {domain_context}\n"
                f"Generate {num_hypotheses} testable hypotheses."
            )},
        ],
        response_format={"type": "json_object"},
    )
    data = json.loads(response.choices[0].message.content)
    return [Hypothesis(**h) for h in data["hypotheses"]]

Experiment Design

Before running a test, the agent designs the experiment — specifying sample requirements, test parameters, and success criteria:

def design_experiment(hypothesis: Hypothesis, available_data: dict) -> dict:
    """Design a statistical experiment to test the hypothesis."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are an experimental design expert.
Design a rigorous test for the given hypothesis. Specify:
1. Required sample size (use power analysis)
2. The exact statistical test (t-test, chi-squared, ANOVA, regression, etc.)
3. Control variables to account for
4. Potential confounding factors
5. Pre-registration: what result would support vs reject the hypothesis?

Return JSON with: test_type, required_sample_size, control_variables,
confounders, support_criteria, rejection_criteria."""},
            {"role": "user", "content": (
                f"Hypothesis: {hypothesis.statement}\n"
                f"Null hypothesis: {hypothesis.null_hypothesis}\n"
                f"Variables: {hypothesis.variables}\n"
                f"Available data: {available_data}\n"
                f"Significance level: {hypothesis.significance_level}"
            )},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

Running Statistical Tests

The agent executes the appropriate statistical test using Python's scipy or statsmodels:

import numpy as np
from scipy import stats

def run_statistical_test(
    test_type: str,
    group_a: list[float],
    group_b: list[float],
    alpha: float = 0.05,
) -> ExperimentResult:
    """Execute a statistical test and return structured results."""
    a = np.array(group_a)
    b = np.array(group_b)

    if test_type == "t_test_independent":
        stat, p_value = stats.ttest_ind(a, b)
        effect_size = (a.mean() - b.mean()) / np.sqrt(
            (a.std()**2 + b.std()**2) / 2
        )  # Cohen's d

    elif test_type == "mann_whitney":
        stat, p_value = stats.mannwhitneyu(a, b, alternative="two-sided")
        effect_size = stat / (len(a) * len(b))  # rank-biserial

    elif test_type == "chi_squared":
        contingency = np.array([group_a, group_b])
        stat, p_value, _, _ = stats.chi2_contingency(contingency)
        n = contingency.sum()
        effect_size = np.sqrt(stat / n)  # Cramer's V

    else:
        raise ValueError(f"Unsupported test: {test_type}")

    # Confidence interval for the difference in means
    diff = a.mean() - b.mean()
    se = np.sqrt(a.var()/len(a) + b.var()/len(b))
    ci = (diff - 1.96*se, diff + 1.96*se)

    return ExperimentResult(
        hypothesis_id="",
        test_statistic=float(stat),
        p_value=float(p_value),
        effect_size=float(effect_size),
        sample_size=len(a) + len(b),
        confidence_interval=ci,
        interpretation=interpret_result(p_value, effect_size, alpha),
    )

def interpret_result(p_value: float, effect_size: float, alpha: float) -> str:
    """Generate a plain-language interpretation."""
    significant = p_value < alpha
    practical = abs(effect_size) > 0.2  # small effect threshold

    if significant and practical:
        return "Statistically significant with practical importance"
    elif significant and not practical:
        return "Statistically significant but effect size is trivially small"
    elif not significant:
        return "Not statistically significant — cannot reject null hypothesis"

The Iteration Loop

After testing, the agent does not stop. It examines results, generates follow-up hypotheses, and tests those:

def hypothesis_testing_loop(
    data_description: str,
    domain: str,
    data: dict,
    max_iterations: int = 3,
) -> list[Hypothesis]:
    """Full scientific method loop: hypothesize, test, iterate."""
    all_hypotheses: list[Hypothesis] = []

    for iteration in range(max_iterations):
        # Generate hypotheses (informed by prior findings)
        prior_findings = [
            f"{h.statement}: {h.status.value} (p={h.p_value})"
            for h in all_hypotheses if h.status != HypothesisStatus.PROPOSED
        ]
        context = f"{domain}\nPrior findings: {prior_findings}"

        new_hypotheses = generate_hypotheses(data_description, context, num_hypotheses=3)

        for hyp in new_hypotheses:
            experiment = design_experiment(hyp, data)
            # Execute test (simplified — real version fetches actual data)
            # result = run_statistical_test(...)
            # hyp.status = determine_status(result)
            # hyp.p_value = result.p_value
            all_hypotheses.append(hyp)

        print(f"Iteration {iteration + 1}: tested {len(new_hypotheses)} hypotheses")

    return all_hypotheses

Guarding Against Common Pitfalls

The agent must avoid: p-hacking (testing many hypotheses without correction — apply Bonferroni or FDR correction), HARKing (hypothesizing after results are known — pre-register hypotheses before testing), and ignoring effect size (a statistically significant but tiny effect is often meaningless in practice).

FAQ

How does the agent handle multiple hypothesis testing?

It applies multiple comparison corrections. For a small number of hypotheses (under 20), Bonferroni correction divides alpha by the number of tests. For larger sets, the Benjamini-Hochberg procedure controls the false discovery rate. The agent tracks how many tests it has run and adjusts significance thresholds automatically.

Can this agent work with non-tabular data?

Yes. For text data, the agent generates hypotheses about word frequencies, sentiment distributions, or topic prevalence, then uses appropriate tests (chi-squared for categorical, permutation tests for complex metrics). For image or time-series data, it first extracts numerical features, then applies standard statistical tests to those features.

How do you handle insufficient sample sizes?

The agent performs a power analysis before testing. If the available sample is too small to detect the hypothesized effect size at the desired significance level, it reports this explicitly rather than running an underpowered test. It may suggest: collecting more data, testing a larger effect size, or using a Bayesian approach that handles small samples more gracefully.

---

#HypothesisTesting #ScientificMethod #DataAnalysis #StatisticalTesting #AgenticAI #PythonAI #DataScience #ExperimentDesign