Building AI Agents That Work: Architecture, Memory, and Frameworks

Calling a language model API takes about ten minutes to learn. Getting an agent to finish a five-step task without looping, forgetting its own context, or quietly doing something you never approved takes a lot longer. We found that out the slow way: our first research agent ran a web search 41 times on a single question, then gave us a confident summary built from the first three results.

This article is the roadmap we wish we had before that happened. It is aimed at developers, software engineers, and technical product managers who already know how to prompt a model and now want to build something stateful and reliable. We cover the anatomy of an agent, five workflow patterns ordered from simplest to riskiest, the memory, tool, and state subsystems, a framework decision matrix, a production hardening checklist, and a small project we built and tested end to end. The code is illustrative blueprint code, meant to show the shape of each pattern rather than to be pasted into production unchanged.

The Anatomy of an AI Agent: Beyond the LLM Wrapper

The formula we use with every team we talk to is simple:

Agent = LLM + Prompt + Memory + Tools + Orchestration + Observability

A raw model talks. It takes text in and returns text out, and nothing in the world changes. An agent does. It decides to call a tool, reads the result, updates what it knows, and chooses the next step. That decide, act, observe cycle is what makes an agent useful. It is also where almost every failure we have seen comes from.

Each term in the formula is a separate engineering problem:

• LLM: the reasoning engine. It decides what to do next but holds no durable state between calls.
• Prompt: the standing instructions, role, rules, and output format.
• Memory: what the agent can recall, both inside one run and across runs.
• Tools: the functions the agent may call, such as search, file read and write, database queries, or APIs.
• Orchestration: the code that decides the order of steps, handles errors, and decides when to stop.
• Observability: traces, logs, and cost counters that let you see what the agent actually did.

Most failed agent projects we have reviewed were missing the last two. The model was fine. Nobody had written the control logic, and nobody could see what the agent was doing when it went wrong.

If the underlying concept is new to you, we covered the shift from chatbots to planning systems in Agentic AI Explained: How AI Systems Plan, Act, and Adapt. This article assumes you want to go one level deeper and actually build.

Workflows vs. Agents: The Distinction That Saves You Money

Before the patterns, one distinction matters more than any framework choice.

• Deterministic workflows are paths your code defines. The model fills in steps, but your program decides the order. Prompt chaining and routing belong here.
• Autonomous loops are paths the model defines at runtime. It decides which tool to call, how many times, and when it is done. The ReAct pattern belongs here.

Autonomy buys flexibility and costs predictability, tokens, and safety margin. In our testing, the cheapest, most debuggable solution to a problem was almost never the most autonomous one. Our rule of thumb is to use the least autonomy that solves the problem.

The 4 Canonical Workflow Patterns (Start Simple, Scale Only When Needed)

Pattern 1: The Augmented LLM

A single model call with access to a few tools, such as search or retrieval. There is no loop and no graph. It is fast, cheap, and easy to debug because every run is one request and one response, possibly with a tool call in the middle.

Use it when the task fits in one reasoning pass, such as answering a question from a document or extracting fields from text. In our tests this pattern handled more tasks than we expected, and teams routinely skip it on the way to something fancier.

Pattern 2: Prompt Chaining and Routing

Prompt chaining splits a task into a fixed sequence: step one output feeds step two, which feeds step three. You can validate between steps, which is the real benefit. A drafted summary can be checked against a schema before it moves on.

Routing adds a classifier at the front. It reads the input and sends it down one of several specialised paths, for example billing questions to one prompt and technical questions to another. Routing keeps each prompt short and focused instead of one giant prompt trying to handle everything.

Use it when the steps are known in advance. This is still a deterministic workflow, just a longer one.

Pattern 3: Orchestrator-Workers and Evaluator-Optimizer

In orchestrator-workers, a coordinating model breaks a task into subtasks you did not know about ahead of time, hands them to worker calls, and merges the results. The number and type of subtasks are decided at runtime, which is what separates it from a chain.

In evaluator-optimizer, one call generates and a second call critiques against explicit criteria. If the critique fails, the feedback goes back to the generator for another pass. This is the writer-and-editor loop, and it is the single most valuable pattern we used in the project below.

If you want to see how several cooperating agents behave at larger scale, our look at how multiple AI agents work together goes into coordination and failure modes in more depth. Our advice here is to earn that complexity first.

Pattern 4: Autonomous ReAct Loops

ReAct stands for Reason, Act, Observe. The model thinks about the goal, picks a tool, sees the result, and repeats until it decides it is done. It is the most flexible pattern and the riskiest. Three things go wrong repeatedly:

• Infinite or near-infinite loops when the model keeps retrying a step that cannot succeed.
• Token explosion because every iteration re-sends a growing history.
• Silent drift where the agent slowly wanders from the original goal.

Every ReAct loop we ship has three guards: a hard step cap, a token budget, and a stop condition that does not depend on the model deciding it is finished. The minimal version looks like this:

MAX_STEPS = 8
TOKEN_BUDGET = 40000

def run_agent(goal, tools, llm):
    history, spent = [goal], 0
    for step in range(MAX_STEPS):
        action = llm.decide(history, tools)      # reason
        if action.type == "final_answer":
            return action.content
        result = tools.execute(action)           # act (sandboxed)
        history.append(summarize(result))        # observe, kept compact
        spent += action.tokens_used
        if spent > TOKEN_BUDGET:
            return escalate_to_human(history, reason="budget")
    return escalate_to_human(history, reason="step_cap")

The important detail is the last two lines. When the agent hits a limit, it hands off instead of failing silently or guessing.

We also recorded a short video version of this article, covering how to choose the right agent pattern and what to put in place before an agent touches real work.

What the video covers:

• The anatomy of an agent and the difference between workflows and autonomous loops
• The workflow patterns, from the simplest augmented LLM to ReAct loops
• Memory, tools, and framework choices, and when plain code is enough
• The hardening checklist that keeps agents from looping or burning tokens

Designing the Core Subsystems

Memory Systems

We separate memory into three layers, because mixing them is a common source of bugs:

• Short-term working context: the current conversation and recent tool results, held in the prompt. It is fast but limited by the context window, so it needs summarising as it grows.
• Episodic vector memory: past runs or documents stored as embeddings and retrieved by similarity. It is good for "what did we learn last time" and fuzzy recall, and poor for exact facts.
• SQL or structured state: exact, queryable facts such as task status, user settings, and step counters. If a value must be right, it belongs here rather than in a vector store.

The mistake we made early was storing task status in a vector store. Retrieval returned a similar but outdated status, and the agent happily repeated a finished step. Exact state should live in a database. Deciding what goes into the prompt at each step is a discipline of its own, which we cover in Context Engineering for AI: How to Give AI the Right Information at the Right Time.

Tool Integration

Tool descriptions matter as much as the prompt. The model chooses tools by reading their names and descriptions, so a vague description produces wrong calls. We rewrote one tool description from "searches stuff" to a precise sentence naming its inputs, outputs, and when not to use it, and wrong-tool calls dropped noticeably in our next batch of test runs.

Execution boundaries matter just as much. Tools should run with the narrowest permissions that work: read-only by default, write access only where needed, and anything destructive behind approval. Exposing tools through Model Context Protocol (MCP) servers helps because it gives you one defined place to enforce what each tool can touch, rather than scattering permission checks through agent code.

State and Orchestration

Once a workflow has branches, retries, or waits for a human, you need three things: a queue or step record of what has happened, error handling that distinguishes retryable failures from fatal ones, and durable checkpoints so a crashed run resumes from the last good step instead of restarting. Checkpointing is the capability that most often justifies a heavier framework, which brings us to the stack.

Choosing the Underlying Model

The orchestration layer decides how an agent behaves, but the model underneath decides how well each individual step reasons. For multi-step tool use, we prioritise reliable structured output, consistent tool-call formatting, and the ability to stay on task across long contexts over raw benchmark headlines. When selecting a high-capability base model for complex reasoning, developers can look at the frontier options examined in our breakdown of the Alibaba Zhenwu V900 10-Trillion Qwen ecosystem. Whichever model you pick, keep it behind a thin client interface so you can swap it without rewriting orchestration code. We changed models twice during our own project, and that abstraction saved us a lot of time.

Choosing Your Stack: Framework Decision Matrix

Frameworks are tools for specific bottlenecks, not badges of seriousness. Our approach is to start with plain code and adopt a framework only when a concrete problem appears. Here is how we match problems to tools:

Framework	Primary Architecture	Best For	Learning Curve	State Management
LangGraph	Explicit graph of nodes and edges	Durable state, branching, retries, human approval steps	Steeper; you design the graph yourself	Strong; built-in checkpointing and resume
CrewAI	Role-based agents with delegated tasks	Fast prototypes where roles map cleanly to tasks	Gentle; productive quickly	Lighter; less control over fine-grained recovery
LlamaIndex Workflows	Event-driven steps around data and retrieval	RAG-centric agents over documents and knowledge bases	Moderate; easy if you already use its retrieval tools	Event context; good for retrieval pipelines
Plain Python	Your own functions and loops	Patterns 1 to 3 with known steps	Lowest; you already know it	Whatever you build, usually a database table

The Architectural Reality Test is the quick check we run before adopting anything. Ask three questions:

• Does the workflow branch or retry in ways that need saved state? If yes, a graph runtime like LangGraph starts earning its keep.
• Is the core job retrieval over your own documents? If yes, look at LlamaIndex Workflows first.
• Do you mainly need a quick demo with clear roles? CrewAI gets you there fastest.

If you answered no to all three, write plain code. We would rather see a clean 80-line chain than a framework carrying features nobody uses.

Project Blueprint: A Market and Technical Research Assistant

To check that these ideas hold together, we built a small research assistant that takes a broad question, gathers material, and returns a structured report.

Design

The system has two cooperating roles. A retrieval worker pulls web search results and our local notes. A generator-critic pair then drafts and reviews the synthesis, an evaluator-optimizer loop capped at three iterations. Inputs and tools were an LLM client, a web search tool, local file storage, and a JSON schema validator.

Workflow

• The user submits a broad query, such as "summarise the options for running small models locally and when each makes sense."
• The orchestrator triggers the retrieval worker, which collects sources and relevant internal notes, then compresses them into a bounded context.
• The generator drafts a report in a fixed JSON schema: summary, key findings, risks, and open questions.
• The validator checks the schema first. Then the critic checks the draft against written quality rules: every claim traceable to a retrieved source, no unsupported numbers, and no missing sections.
• If either check fails, the feedback returns to the generator. After three failed rounds, the run stops and escalates to a human with the best draft attached.

What We Measured

We ran 10 test prompts through three configurations and counted two things: reports that passed schema and quality checks on the first complete run, and runs that hit a hard cap. These are our results from this one small project, not a general benchmark:

Configuration	Reports Passing Checks	Runs Hitting a Cap	Our Observation
Single augmented LLM call	4 of 10	0	Cheap and fast, but it skipped sections and invented figures on broad prompts
Open ReAct loop, no critic	6 of 10	3	Better coverage, but token use varied wildly and some runs wandered
Retrieval worker + evaluator-optimizer, capped	9 of 10	1	Most stable output; the capped run escalated cleanly to a human

The pattern that won was not the most autonomous one. The critic with explicit rules and a hard iteration cap gave us stability that open-ended autonomy did not.

Failure Scenarios We Hit

• Infinite retry loops: in an early version, the critic and generator disagreed about one rule and bounced back and forth. A max-iteration cap and a requirement that feedback be specific fixed it.
• Context overflow: long source documents pushed the prompt past useful size. A summarisation buffer that compresses each source to a few key points before synthesis solved it.
• Stale memory: covered above, and solved by moving exact state into a database table.

Production Hardening: Keeping Your Agent From Burning Tokens and Breaking Systems

This is the checklist we apply before any agent touches real work.

• Hard step caps and budgets. Every loop needs a maximum step count and a token or cost ceiling that stops the run regardless of what the model wants. Hitting a cap should trigger a handoff, never a silent stop.
• Structured outputs between steps. Validate each step's output against a JSON schema before passing it on. A malformed output caught at step two costs a retry, while the same output caught at step six costs a corrupted report.
• Sandboxed, least-privilege tools. Read-only by default, scoped credentials, and no tool that can do more than its task requires.
• Human-in-the-loop for high-stakes actions. Anything that sends, deletes, pays, or publishes should wait for explicit approval. The cost of one extra click is small next to the cost of one wrong action.
• Tracing from day one. Log every model call, tool call, input, output, and cost so you can replay a failed run. Retrofitting observability after an incident is painful.
• A small, fixed test set. Keep around ten representative prompts and rerun them after every change. It catches regressions that eyeballing one example will not.

These controls line up with the broader practices in AI Safety Engineering: A Practical Guide to Guardrails, Monitoring, and AI Risk, and the governance side of running agents in real organizations is covered in Production AI Agents: How to Build Reliable, Secure, and Governed AI Systems.

A Simple Way to Decide What to Build

When someone asks us whether they need an agent, we walk down this ladder and stop at the first rung that works:

• One call with tools solves it: use an augmented LLM.
• The steps are known: use a chain, with routing if inputs vary.
• The steps are unknown but the output is checkable: use orchestrator-workers or an evaluator-optimizer loop.
• The path genuinely cannot be predicted: use an autonomous loop, with caps, sandboxing, and human approval.

Moving down the ladder should be a decision you can justify, not a default.

Frequently Asked Questions

What is the difference between an AI workflow and an AI agent?

In a workflow, your code defines the sequence of steps and the model fills in each one. In an agent, the model decides at runtime which steps to take and when to stop. Workflows are more predictable and cheaper. Agents are more flexible and riskier.

Do I need LangGraph or CrewAI to build an agent?

No. Many useful agents are plain code with a model client, a few tools, and a loop with a step cap. Frameworks help when you need durable state and recovery (LangGraph), rapid role-based prototyping (CrewAI), or retrieval-centric pipelines (LlamaIndex Workflows).

How do I stop an agent from looping forever?

Set a hard maximum number of steps and a token budget enforced by your own code rather than by the model. When either limit is reached, return the best result so far or escalate to a human.

Which type of memory should an agent use?

Use all three for different jobs: working context for the current run, vector memory for fuzzy recall of past material, and a database for exact facts like status and settings. Do not store facts that must be exact in a vector store.

When should a human approve an agent's action?

Any time the action is hard to undo or affects other people or money, such as sending messages, deleting data, making payments, or publishing content. Read-only actions can usually run without approval.

Why do tool descriptions matter so much?

The model chooses tools by reading their descriptions. Vague or overlapping descriptions lead to wrong or unnecessary tool calls, so describe what each tool does, what it needs, and when not to use it.

Final Thoughts

Building a production-ready agent is mostly an exercise in restraint. The model is the exciting part, but reliability comes from everything around it: the pattern you choose, the memory you separate, the tools you constrain, the limits you enforce, and the traces you keep. In our own project, the winning design was a modest two-role workflow with a rule-driven critic and a hard cap, not an open-ended autonomous swarm.

Start with one augmented call, add structure only when a real bottleneck appears, and treat every step toward autonomy as something that must be justified, capped, and observable. That discipline is what separates an agent that impresses in a demo from one you can trust on a Tuesday afternoon with real work on the line.