Building a Compliance Checking Agent: Policy Validation and Regulatory Mapping

Why Compliance Checking Is a Perfect Fit for AI Agents

Regulatory compliance is an exercise in mapping — matching an organization's policies and controls against a constantly evolving set of rules. SOC 2, GDPR, HIPAA, PCI DSS, and dozens of other frameworks each contain hundreds of individual requirements. Manually cross-referencing company policies against these requirements is tedious, error-prone, and expensive. An AI agent can load regulatory text, parse your internal policies, identify gaps, and generate a compliance report in minutes.

Agent Design

The compliance checking agent uses three core components:

flowchart LR
    REQ(["Inbound request"])
    PII["PII detection<br/>regex plus NER"]
    POL{"Policy engine<br/>OPA or rules"}
    REDACT["Redact or mask"]
    LLM["LLM call"]
    OUT["Response"]
    AUDIT[("Append only<br/>audit log")]
    BLOCK(["Block plus<br/>notify DPO"])
    REQ --> PII --> POL
    POL -->|Allow| REDACT --> LLM --> OUT --> AUDIT
    POL -->|Deny| BLOCK
    style POL fill:#4f46e5,stroke:#4338ca,color:#fff
    style AUDIT fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style BLOCK fill:#dc2626,stroke:#b91c1c,color:#fff
    style OUT fill:#059669,stroke:#047857,color:#fff

1. Regulatory Framework Loader — structured storage of regulatory requirements
2. Policy Analyzer — parses internal policies and maps them to requirements
3. Gap Analysis Engine — identifies missing or weak coverage and generates reports

Step 1: Modeling Regulatory Requirements

First, define structured models for regulations and policies.

from pydantic import BaseModel
from enum import Enum

class ComplianceStatus(str, Enum):
    COMPLIANT = "compliant"
    PARTIAL = "partial"
    NON_COMPLIANT = "non_compliant"
    NOT_ASSESSED = "not_assessed"

class Requirement(BaseModel):
    req_id: str  # e.g., "GDPR-Art-25"
    framework: str  # e.g., "GDPR"
    title: str
    description: str
    category: str  # e.g., "Data Protection by Design"

class PolicyDocument(BaseModel):
    doc_id: str
    title: str
    content: str
    last_updated: str
    owner: str

class ComplianceMapping(BaseModel):
    requirement: Requirement
    status: ComplianceStatus
    matched_policy: str | None
    coverage_score: float  # 0.0 to 1.0
    gaps: list[str]
    recommendations: list[str]

Step 2: Regulatory Framework Loader

Store requirements in a structured format. In production you would load these from a database, but a JSON file works for demonstration.

import json
from pathlib import Path

def load_framework(framework_path: str) -> list[Requirement]:
    """Load regulatory requirements from a JSON file."""
    data = json.loads(Path(framework_path).read_text())
    return [Requirement(**req) for req in data["requirements"]]

# Example framework JSON structure:
# {
#   "framework": "SOC2",
#   "version": "2024",
#   "requirements": [
#     {
#       "req_id": "CC6.1",
#       "framework": "SOC2",
#       "title": "Logical Access Security",
#       "description": "The entity implements logical access ...",
#       "category": "Common Criteria"
#     }
#   ]
# }

Step 3: Policy Analysis with LLM

The agent reads each internal policy document and determines which regulatory requirements it addresses.

from openai import OpenAI

client = OpenAI()

class PolicyAnalysis(BaseModel):
    addressed_requirements: list[str]  # List of req_ids
    key_controls: list[str]
    weaknesses: list[str]

def analyze_policy(
    policy: PolicyDocument, requirements: list[Requirement]
) -> PolicyAnalysis:
    """Analyze a policy document against requirements."""
    req_list = "\n".join(
        f"- {r.req_id}: {r.title} — {r.description[:200]}"
        for r in requirements
    )

    response = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a compliance analyst. Analyze the policy "
                    "document and determine which regulatory requirements "
                    "it addresses. Identify key controls implemented and "
                    "any weaknesses in coverage."
                ),
            },
            {
                "role": "user",
                "content": (
                    f"Policy: {policy.title}\n\n"
                    f"{policy.content}\n\n"
                    f"Requirements to check against:\n{req_list}"
                ),
            },
        ],
        response_format=PolicyAnalysis,
    )
    return response.choices[0].message.parsed

Step 4: Gap Analysis and Report Generation

Compare analyzed policies against the full set of requirements to find gaps.

class GapReport(BaseModel):
    framework: str
    total_requirements: int
    compliant: int
    partial: int
    non_compliant: int
    compliance_percentage: float
    mappings: list[ComplianceMapping]

def run_gap_analysis(
    requirements: list[Requirement],
    policies: list[PolicyDocument],
) -> GapReport:
    """Run full gap analysis across all requirements and policies."""
    # Analyze each policy
    all_analyses = {}
    for policy in policies:
        analysis = analyze_policy(policy, requirements)
        all_analyses[policy.doc_id] = analysis

    # Map requirements to policy coverage
    mappings = []
    for req in requirements:
        best_match = None
        best_score = 0.0
        gaps = []
        recommendations = []

        for doc_id, analysis in all_analyses.items():
            if req.req_id in analysis.addressed_requirements:
                # Found coverage — score it
                score = score_coverage(req, analysis)
                if score > best_score:
                    best_score = score
                    best_match = doc_id

        if best_score >= 0.8:
            status = ComplianceStatus.COMPLIANT
        elif best_score >= 0.4:
            status = ComplianceStatus.PARTIAL
            gaps.append("Partial coverage — review for completeness")
            recommendations.append(
                f"Strengthen {req.title} controls in policy"
            )
        else:
            status = ComplianceStatus.NON_COMPLIANT
            gaps.append(f"No policy addresses {req.req_id}")
            recommendations.append(
                f"Create or update policy to cover {req.title}"
            )

        mappings.append(
            ComplianceMapping(
                requirement=req,
                status=status,
                matched_policy=best_match,
                coverage_score=best_score,
                gaps=gaps,
                recommendations=recommendations,
            )
        )

    compliant = sum(
        1 for m in mappings if m.status == ComplianceStatus.COMPLIANT
    )
    partial = sum(
        1 for m in mappings if m.status == ComplianceStatus.PARTIAL
    )
    non_compliant = sum(
        1 for m in mappings if m.status == ComplianceStatus.NON_COMPLIANT
    )

    return GapReport(
        framework=requirements[0].framework,
        total_requirements=len(requirements),
        compliant=compliant,
        partial=partial,
        non_compliant=non_compliant,
        compliance_percentage=(compliant / len(requirements)) * 100,
        mappings=mappings,
    )

def score_coverage(req: Requirement, analysis: PolicyAnalysis) -> float:
    """Score how well a policy covers a requirement (0.0 to 1.0)."""
    if req.req_id not in analysis.addressed_requirements:
        return 0.0
    weakness_penalty = len(analysis.weaknesses) * 0.1
    return max(0.0, 1.0 - weakness_penalty)

Running the Agent

requirements = load_framework("soc2_requirements.json")
policies = [
    PolicyDocument(
        doc_id="POL-001",
        title="Information Security Policy",
        content="... policy text ...",
        last_updated="2026-01-15",
        owner="CISO",
    ),
]

report = run_gap_analysis(requirements, policies)
print(f"Compliance: {report.compliance_percentage:.1f}%")
print(f"Gaps found: {report.non_compliant} requirements uncovered")

FAQ

How often should the compliance agent be run?

Run it on every policy change and at least quarterly. Set up CI hooks so that when policy documents are updated in your document management system, the agent automatically re-evaluates compliance status and notifies the compliance team of any regressions.

Can one agent handle multiple regulatory frameworks simultaneously?

Yes. Load multiple frameworks and run gap analysis against all of them in parallel. The structured output model supports a framework field on each requirement, so the report can show SOC 2, GDPR, and HIPAA compliance side by side.

How do you keep the regulatory requirements database up to date?

Subscribe to regulatory update feeds from organizations like NIST, the EU Data Protection Board, or PCI SSC. Periodically scrape published requirement updates and have the LLM parse changes into your structured format. Always flag new or amended requirements for human review before adding them to the active database.

---

#Compliance #Regulatory #PolicyValidation #RiskManagement #AIAgent #AgenticAI #LearnAI #AIEngineering