Code-Writing Agents in 2026: Execution-Based Evaluation Beats Everything Else
Why LLM-as-judge is the wrong tool for code agents — and how to build an execution-based eval pipeline that actually catches broken code.
13 min read
Agentic AI & LLMs
Agentic AI & LLMs archive page 147 of 631
Agentic AI & LLMs
Agentic AI, LLM engineering, and the models behind modern automation — multi-agent systems, LLM evaluation and comparisons, RAG, fine-tuning, AI infrastructure, security, and production AI engineering.
From the blog
Latest articles
Continuous Evaluation: Wiring LangSmith into Your CI/CD for Agent Releases
Run offline evals as a CI gate. GitHub Actions wiring, threshold gates, LangSmith Experiments, and how to block merges on agent regression — with real YAML.
12 min read
Cost-Aware Agent Evaluation: Putting Token Spend, Latency, and Quality on the Same Dashboard
Eval scores alone mislead. Here is how we build a Pareto view across cost, latency, and quality so agent releases ship on signal, not vibes.
12 min read
Evaluating Agent Memory: Recall, Precision, and the Eval Pipeline Most Teams Don't Build
Memory is supposed to make agents better — but does it? Build a memory eval pipeline that measures recall, precision, contradiction rate, and the freshness/staleness tradeoff.
13 min read
Evaluating Multi-Step Tool-Using Agents: Why End-to-End Metrics Lie
A 'did the agent answer correctly?' pass/fail hides broken tool calls, wasted tokens, and silent retries. Here is how to evaluate intermediate steps.
13 min read
Evaluating Agent Reasoning Traces: Measuring Thought Quality Beyond Final Answers
Final-answer accuracy hides broken reasoning. Build an eval pipeline that scores the reasoning trace itself — coherence, faithfulness to tools, dead-end detection.
13 min read
How to Build a Golden Dataset for Production AI Agents
A principal engineer's playbook for curating, versioning, and growing a golden dataset for an agent — from production trace mining to annotation queues in LangSmith.
12 min read
Building Reasoning Agents with GPT-5 and o3 in 2026: When to Reach for the Big Brain
When reasoning models actually help inside an agent loop — and when they're an expensive mistake. Architecture patterns, code, and the cost/quality tradeoffs that matter.
13 min read
Input and Output Guardrails in the OpenAI Agents SDK: A Production Pattern (2026)
Stop the agent BEFORE it does the wrong thing. How to wire input and output guardrails in the OpenAI Agents SDK with cheap classifiers and an eval suite that proves they work.
13 min read