Building an AI Agent Admin Dashboard: User Management, Analytics, and Configuration

What an AI Agent Admin Dashboard Must Do

An enterprise AI agent platform without an admin dashboard is like a database without a query console. Platform teams resort to SSH sessions and raw API calls to manage agents, check usage, and troubleshoot issues. This does not scale past two or three agents and creates a bus factor of one — the single engineer who knows the CLI commands.

A well-designed admin dashboard covers four areas: user and role management, agent configuration, real-time analytics, and operational health monitoring. Each area maps to a set of API endpoints that the dashboard frontend consumes.

User and Role Management API

The user management layer synchronizes with the enterprise identity provider and adds agent-specific role assignments. Users are imported from SSO, and administrators assign them to agent roles through the dashboard.

flowchart LR
    INPUT(["User intent"])
    PARSE["Parse plus<br/>classify"]
    PLAN["Plan and tool<br/>selection"]
    AGENT["Agent loop<br/>LLM plus tools"]
    GUARD{"Guardrails<br/>and policy"}
    EXEC["Execute and<br/>verify result"]
    OBS[("Trace and metrics")]
    OUT(["Outcome plus<br/>next action"])
    INPUT --> PARSE --> PLAN --> AGENT --> GUARD
    GUARD -->|Pass| EXEC --> OUT
    GUARD -->|Fail| AGENT
    AGENT --> OBS
    style AGENT fill:#4f46e5,stroke:#4338ca,color:#fff
    style GUARD fill:#f59e0b,stroke:#d97706,color:#1f2937
    style OBS fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style OUT fill:#059669,stroke:#047857,color:#fff

from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel, EmailStr
from datetime import datetime
from uuid import uuid4
from enum import Enum

class Role(str, Enum):
    VIEWER = "viewer"
    USER = "user"
    CONFIGURATOR = "configurator"
    ADMIN = "admin"

class UserCreate(BaseModel):
    email: EmailStr
    display_name: str
    role: Role = Role.USER
    department: str
    allowed_agents: list[str] = []

class UserResponse(BaseModel):
    id: str
    email: str
    display_name: str
    role: Role
    department: str
    allowed_agents: list[str]
    last_active: datetime | None
    total_requests: int

app = FastAPI(title="Agent Admin API")

@app.post("/admin/users", response_model=UserResponse)
async def create_user(payload: UserCreate, db=Depends(get_db)):
    existing = await db.fetchrow(
        "SELECT id FROM users WHERE email = $1", payload.email
    )
    if existing:
        raise HTTPException(409, "User already exists")

    user_id = str(uuid4())
    await db.execute(
        """
        INSERT INTO users (id, email, display_name, role, department, allowed_agents)
        VALUES ($1, $2, $3, $4, $5, $6)
        """,
        user_id, payload.email, payload.display_name,
        payload.role.value, payload.department, payload.allowed_agents,
    )
    return UserResponse(
        id=user_id, email=payload.email, display_name=payload.display_name,
        role=payload.role, department=payload.department,
        allowed_agents=payload.allowed_agents, last_active=None, total_requests=0,
    )

@app.get("/admin/users")
async def list_users(
    page: int = 1, per_page: int = 25,
    role: Role | None = None, db=Depends(get_db)
):
    offset = (page - 1) * per_page
    query = "SELECT * FROM users"
    params = []

    if role:
        query += " WHERE role = $1"
        params.append(role.value)

    query += f" ORDER BY display_name LIMIT ${len(params) + 1} OFFSET ${len(params) + 2}"
    params.extend([per_page, offset])

    rows = await db.fetch(query, *params)
    total = await db.fetchval("SELECT COUNT(*) FROM users")

    return {"users": rows, "total": total, "page": page, "per_page": per_page}

Agent Configuration Management

Agents need configurable parameters: system prompts, model selection, temperature, available tools, and guardrails. The admin dashboard provides a form interface for these settings, backed by a versioned configuration store.

class AgentConfig(BaseModel):
    agent_id: str
    display_name: str
    description: str
    model: str = "gpt-4o"
    system_prompt: str
    temperature: float = 0.7
    max_tokens: int = 4096
    tools_enabled: list[str] = []
    guardrails: dict = {}
    is_active: bool = True

@app.put("/admin/agents/{agent_id}/config")
async def update_agent_config(
    agent_id: str, config: AgentConfig, db=Depends(get_db)
):
    current = await db.fetchrow(
        "SELECT version FROM agent_configs WHERE agent_id = $1 "
        "ORDER BY version DESC LIMIT 1",
        agent_id,
    )
    new_version = (current["version"] + 1) if current else 1

    await db.execute(
        """
        INSERT INTO agent_configs (
            agent_id, version, display_name, description,
            model, system_prompt, temperature, max_tokens,
            tools_enabled, guardrails, is_active, created_by
        ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)
        """,
        agent_id, new_version, config.display_name, config.description,
        config.model, config.system_prompt, config.temperature,
        config.max_tokens, config.tools_enabled, config.guardrails,
        config.is_active, "admin",
    )
    return {"agent_id": agent_id, "version": new_version, "status": "updated"}

Real-Time Analytics Endpoints

Dashboard analytics show usage trends, cost breakdowns, and agent performance. Aggregate metrics by hour or day and cache the results so that dashboard page loads are fast even with millions of audit events.

@app.get("/admin/analytics/usage")
async def get_usage_analytics(
    days: int = 30, agent_id: str | None = None, db=Depends(get_db)
):
    query = """
        SELECT
            date_trunc('day', timestamp) AS day,
            agent_id,
            COUNT(*) AS request_count,
            AVG(latency_ms)::int AS avg_latency_ms,
            SUM(token_count) AS total_tokens,
            COUNT(DISTINCT user_id) AS unique_users
        FROM agent_requests
        WHERE timestamp > NOW() - INTERVAL '%s days'
    """
    params = [days]

    if agent_id:
        query += " AND agent_id = $2"
        params.append(agent_id)

    query += " GROUP BY day, agent_id ORDER BY day DESC"

    rows = await db.fetch(query % days if not agent_id else query, *params[1:])
    return {"period_days": days, "data": [dict(r) for r in rows]}

FAQ

Should the admin dashboard use the same database as the agents?

No. Keep a dedicated admin database or at minimum separate schemas. Agents need low-latency read/write access to their operational data. Admin queries, especially analytics aggregations, can be expensive and should not compete for the same connection pool. Use read replicas or materialized views for analytics data.

How do you handle configuration changes that could break running agent sessions?

Configuration changes should take effect on new sessions only. Running sessions continue with the configuration they started with. Store the active config version in the session metadata at creation time. This prevents mid-conversation behavior changes that confuse users.

What access controls should the admin dashboard itself have?

Implement tiered admin access. Viewers can see analytics and logs but cannot change anything. Configurators can update agent settings but not manage users. Full admins can do everything. Use the same SSO integration and role mapping that governs agent access, so there is one unified permission model.

#EnterpriseAI #AdminDashboard #UserManagement #Analytics #Configuration #AgenticAI #LearnAI #AIEngineering