Building Your First AI Agent from Scratch in Python (No Framework)

Why Build Without a Framework?

Frameworks like LangChain, CrewAI, and the OpenAI Agents SDK are excellent tools — but if you use them without understanding what they do underneath, you are building on a foundation you cannot debug, optimize, or extend. Building an agent from scratch once teaches you the mechanics that every framework abstracts away.

This tutorial builds a fully functional agent with tools, conversation management, and error handling using only the OpenAI Python SDK and the standard library. No LangChain, no Agents SDK, no abstractions. Just the raw loop.

What We Are Building

A personal assistant agent that can:

flowchart LR
    CORPUS[("Pre-training corpus<br/>trillions of tokens")]
    FILTER["Quality filter and<br/>dedupe"]
    TOK["BPE tokenizer"]
    SHARD["Shard plus<br/>data parallel"]
    GPU{"GPU cluster<br/>FSDP or DeepSpeed"}
    CKPT[("Checkpoints<br/>every N steps")]
    LOSS["Loss curve plus<br/>eval gates"]
    SFT["SFT phase"]
    DPO["DPO or RLHF"]
    BASE([Base model])
    INSTR([Instruct model])
    CORPUS --> FILTER --> TOK --> SHARD --> GPU
    GPU --> CKPT --> LOSS
    LOSS --> BASE --> SFT --> DPO --> INSTR
    style GPU fill:#4f46e5,stroke:#4338ca,color:#fff
    style LOSS fill:#f59e0b,stroke:#d97706,color:#1f2937
    style INSTR fill:#059669,stroke:#047857,color:#fff

• Look up the current time in any timezone
• Do math calculations
• Read and summarize web page content (simulated)
• Maintain a conversation across multiple turns

Step 1: Project Setup

Create a single file, agent.py. No dependencies beyond the OpenAI SDK:

# agent.py
import json
import math
from datetime import datetime, timezone, timedelta
from openai import OpenAI

client = OpenAI()  # Uses OPENAI_API_KEY env variable
MODEL = "gpt-4o"

Step 2: Define Tools as Python Functions

Each tool is a regular Python function. We will also create the tool definitions that the OpenAI API requires.

def get_current_time(timezone_offset: int = 0) -> dict:
    """Get the current time with an optional UTC offset."""
    tz = timezone(timedelta(hours=timezone_offset))
    now = datetime.now(tz)
    return {
        "time": now.strftime("%H:%M:%S"),
        "date": now.strftime("%Y-%m-%d"),
        "timezone": f"UTC{'+' if timezone_offset >= 0 else ''}{timezone_offset}",
    }

def calculate(expression: str) -> dict:
    """Safely evaluate a mathematical expression."""
    # Whitelist allowed names for safety
    allowed_names = {
        "abs": abs, "round": round, "min": min, "max": max,
        "sum": sum, "pow": pow, "sqrt": math.sqrt,
        "pi": math.pi, "e": math.e, "log": math.log,
        "sin": math.sin, "cos": math.cos, "tan": math.tan,
    }
    try:
        result = eval(expression, {"__builtins__": {}}, allowed_names)
        return {"expression": expression, "result": result}
    except Exception as e:
        return {"error": f"Cannot evaluate '{expression}': {str(e)}"}

def read_webpage(url: str) -> dict:
    """Simulate reading a webpage. Replace with real HTTP calls in production."""
    # In a real agent, you would use httpx or requests here
    return {
        "url": url,
        "title": f"Page at {url}",
        "content": f"This is simulated content from {url}. "
                   f"In production, use an HTTP client to fetch real content.",
        "status": "simulated",
    }

Step 3: Create Tool Definitions for the API

The OpenAI API needs JSON Schema definitions for each tool. This is the contract that tells the LLM what tools exist and how to call them.

TOOL_DEFINITIONS = [
    {
        "type": "function",
        "function": {
            "name": "get_current_time",
            "description": (
                "Get the current date and time. Optionally specify a timezone "
                "as a UTC offset (e.g., -5 for EST, +9 for JST)."
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "timezone_offset": {
                        "type": "integer",
                        "description": "UTC offset in hours. Default is 0 (UTC).",
                    },
                },
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "calculate",
            "description": (
                "Evaluate a mathematical expression. Supports basic arithmetic "
                "(+, -, *, /, **), functions (sqrt, log, sin, cos, tan), "
                "and constants (pi, e). Example: 'sqrt(144) + pi'"
            ),
            "parameters": {
                "type": "object",
                "properties": {
                    "expression": {
                        "type": "string",
                        "description": "The math expression to evaluate.",
                    },
                },
                "required": ["expression"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "read_webpage",
            "description": "Read and return the content of a webpage given its URL.",
            "parameters": {
                "type": "object",
                "properties": {
                    "url": {
                        "type": "string",
                        "description": "The full URL to read.",
                    },
                },
                "required": ["url"],
            },
        },
    },
]

# Map tool names to their Python functions
TOOL_MAP = {
    "get_current_time": get_current_time,
    "calculate": calculate,
    "read_webpage": read_webpage,
}

Step 4: Build the Tool Executor

This function takes a tool call from the API and executes the corresponding Python function:

def execute_tool_call(tool_call) -> str:
    """Execute a single tool call and return the result as a JSON string."""
    function_name = tool_call.function.name

    # Parse arguments
    try:
        arguments = json.loads(tool_call.function.arguments)
    except json.JSONDecodeError:
        return json.dumps({"error": f"Invalid arguments: {tool_call.function.arguments}"})

    # Look up and execute the function
    func = TOOL_MAP.get(function_name)
    if not func:
        return json.dumps({"error": f"Unknown tool: {function_name}"})

    try:
        result = func(**arguments)
        return json.dumps(result, default=str)
    except TypeError as e:
        return json.dumps({"error": f"Wrong arguments for {function_name}: {str(e)}"})
    except Exception as e:
        return json.dumps({"error": f"{function_name} failed: {str(e)}"})

Step 5: The Agent Loop

This is the core. The agent loop sends messages to the API, checks if the response contains tool calls, executes them, and continues until the model returns a plain text response.

def agent_turn(messages: list, max_iterations: int = 10) -> str:
    """Run the agent loop for a single user turn. Returns the final text response."""

    for iteration in range(max_iterations):
        try:
            response = client.chat.completions.create(
                model=MODEL,
                messages=messages,
                tools=TOOL_DEFINITIONS,
            )
        except Exception as e:
            return f"API error: {str(e)}"

        choice = response.choices[0]
        assistant_message = choice.message

        # Add the assistant's message to history
        messages.append(assistant_message)

        # Check if the model wants to call tools
        if not assistant_message.tool_calls:
            # No tool calls — this is the final response
            return assistant_message.content or "(No response generated)"

        # Execute each tool call
        print(f"  [Step {iteration + 1}] Calling {len(assistant_message.tool_calls)} tool(s)...")

        for tool_call in assistant_message.tool_calls:
            print(f"    → {tool_call.function.name}({tool_call.function.arguments})")

            result = execute_tool_call(tool_call)

            print(f"    ← {result[:100]}{'...' if len(result) > 100 else ''}")

            # Add the tool result to the conversation
            messages.append({
                "role": "tool",
                "tool_call_id": tool_call.id,
                "content": result,
            })

    return "I was not able to complete the task within the step limit. Please try a simpler request."

Step 6: Conversation Manager

The conversation manager maintains state across multiple user turns and handles the system prompt:

class Conversation:
    """Manages a multi-turn conversation with an agent."""

    def __init__(self, system_prompt: str = None):
        self.messages = []
        if system_prompt:
            self.messages.append({"role": "system", "content": system_prompt})

    def chat(self, user_input: str) -> str:
        """Send a message and get the agent's response."""
        self.messages.append({"role": "user", "content": user_input})
        response = agent_turn(self.messages)
        return response

    def get_history(self) -> list[dict]:
        """Return conversation history (for debugging)."""
        return [
            {
                "role": m["role"] if isinstance(m, dict) else m.role,
                "content": (m.get("content", "") if isinstance(m, dict)
                           else m.content or "[tool call]"),
            }
            for m in self.messages
        ]

Step 7: Put It All Together

def main():
    """Interactive agent REPL."""
    print("AI Agent (type 'quit' to exit, 'history' to see conversation)")
    print("-" * 50)

    conversation = Conversation(
        system_prompt=(
            "You are a helpful personal assistant. You can check the current time "
            "in any timezone, perform calculations, and read webpages. "
            "Be concise and helpful. When using tools, explain what you found."
        )
    )

    while True:
        user_input = input("\nYou: ").strip()

        if not user_input:
            continue
        if user_input.lower() == "quit":
            print("Goodbye!")
            break
        if user_input.lower() == "history":
            for msg in conversation.get_history():
                print(f"  [{msg['role']}] {msg['content'][:80]}")
            continue

        response = conversation.chat(user_input)
        print(f"\nAgent: {response}")

if __name__ == "__main__":
    main()

Run it:

python agent.py

Try conversations like:

You: What time is it in Tokyo?
Agent: [calls get_current_time with offset +9]

You: What is the square root of that hour number times pi?
Agent: [calls calculate — uses context from previous turn]

You: Read https://example.com and summarize it
Agent: [calls read_webpage, then summarizes]

What You Just Built

This 150-line agent has every fundamental capability of a production agent. It has a tool loop that continues until the task is done. It has proper error handling at every layer — argument parsing, tool execution, API failures. It maintains conversation state across turns. And it is completely transparent — you can see every tool call and result.

Frameworks add value on top of this foundation: streaming, tracing, type safety, multi-agent handoffs, guardrails. But every framework is ultimately running this same loop underneath. Now that you have built it from scratch, you will understand exactly what those frameworks are doing — and you will know how to debug them when they do not behave as expected.

FAQ

How would I add a new tool to this agent?

Three steps: write a Python function that implements the tool, add its JSON Schema definition to TOOL_DEFINITIONS, and register it in TOOL_MAP. That is it. The agent loop and tool executor handle the rest automatically because they are generic — they work with any tool that follows the function calling contract.

How do I handle tools that take a long time to execute?

Wrap tool execution in an async pattern with a timeout. Use Python's asyncio.wait_for() with a reasonable timeout (10-30 seconds). If a tool times out, return an error message to the LLM so it can try a different approach. For very long operations (minutes), consider a polling pattern where the tool returns a job ID and a separate check_job_status tool lets the agent poll for completion.

Should I use this raw approach or a framework in production?

Use a framework for production. This from-scratch approach is for learning. Frameworks like the OpenAI Agents SDK add streaming (critical for user experience), tracing (critical for debugging), guardrails (critical for safety), and structured outputs (critical for reliability). The value of building from scratch is that you understand what the framework does, so you can configure it correctly and debug it when needed.

---

#AIAgent #Python #OpenAIAPI #Tutorial #FromScratch #AgenticAI #LearnAI #AIEngineering