A prospective client once described what they wanted as "an AI agent that handles customer support." Twenty minutes into scoping, the actual requirement turned out to be: answer questions from the help centre, and hand off anything it can't answer. No record needed updating. No system needed to change state. What they wanted was a chatbot that doesn't make things up — and calling it an agent nearly sent the project toward tool integrations, approval gates, and audit logging it never needed.
This mislabelling is common enough that it's worth being precise about, because the word "agent" has stopped doing useful work in most conversations. It gets applied to a scripted decision tree, to a system that answers questions from documents, and to a system that actually goes and does things on your behalf — three architectures with wildly different build costs, risk profiles, and failure modes, flattened into one buzzword. Getting the label right before you scope anything is the difference between a project that ships on time and one that either underdelivers on what was promised or ships far more risk than the problem warranted.
Three things being called "AI" that are not the same thing
Start with the plainest distinction, because the rest of this piece depends on it holding.
Scripted / rules-based chatbot — a decision tree or intent-matching system. It follows pre-written branches ("if the user says X, show response Y") and has no real language understanding. Cheap, predictable, and brittle the moment a question falls outside its tree.
RAG-grounded Q&A chatbot — an LLM that retrieves relevant text from your actual content at the moment of the question, then answers using that retrieved context. It understands open-ended phrasing and can cite what it drew from. It answers; it does not act. Nothing changes anywhere else in your systems because of what it said.
AI agent — a system that plans a sequence of steps, calls tools or APIs to carry them out, evaluates the results, and decides what to do next. It doesn't just tell you the answer — it can look something up, then act on what it found, then act again based on that outcome. It has some degree of autonomy, and it needs guardrails precisely because of that autonomy.
The first two only ever produce text. The third produces consequences. That single fact is why the rest of this framework exists — a wrong answer is embarrassing, but a wrong action can be a refund issued twice, a record updated incorrectly, or an email sent to the wrong person.
Why "we want an AI agent" usually means something narrower
When a business asks for an AI agent, what they're pointing at is almost always one of two things: they've seen the phrase used to describe any AI product with a chat interface, or they've had a bad experience with a rules-based bot and are reaching for the most capable-sounding term available to describe "please make it actually understand what people are asking." Neither of those is a request for autonomy. Both are a request for better grounding.
This matters because the honest answer, most of the time, is that a well-built conversational bot — one that retrieves the right passage from your documentation and answers from it — solves the actual complaint. The complaint was never "our system lacks the ability to take multi-step actions across our CRM and billing platform." It was "our old chatbot gives generic, wrong, or unhelpfully vague answers, and customers can tell." That is a retrieval-quality problem, not an autonomy problem, and the fix is grounding responses in real content rather than building a system that can independently decide to do things — the same architecture question we walk through in more depth when comparing retrieval against fine-tuning as ways of getting a model to actually know your business.
There's a reasonable instinct behind reaching for "agent" anyway: it sounds more capable, more future-facing, more like the thing competitors are supposedly deploying. But capability that isn't needed isn't free — it's cost and risk sitting idle, and in the case of an agent with real tool access, it's cost and risk that has to be actively defended against even while idle.
What a genuine agent use case actually looks like
The cases where an agent earns its complexity share a specific shape: multiple steps across more than one system, where later steps depend on what earlier steps found, and where a human doing the same job would need judgement rather than a fixed script.
A concrete example: a customer asks about a delayed order. Answering well means looking up the order status, checking whether it falls inside the refund policy for that shipping delay, and then — if it qualifies — actually updating the record to issue a partial credit. The first step is a lookup. The second is a policy check that depends on the lookup's result. The third is an action that changes something real. A pure Q&A bot can do the first two and then has to stop and tell a human to do the third, which is often exactly the friction the business is trying to remove. An agent, properly scoped, can complete all three end to end.
What makes this a sound agent use case, rather than just an ambitious one, is that the third step — the action — has a bounded and reversible cost. A partial credit that was issued incorrectly can be reversed. Compare that to an agent empowered to permanently delete records, change account permissions, or issue full refunds with no upper bound: the action itself is what should determine whether autonomy is appropriate, not whether the conversation "sounds complex." The operational reality of giving a system real actions is that every tool you hand it is something a bad instruction — through misuse, misunderstanding, or an injected prompt buried in something the agent reads — can trigger. A chatbot that gets confused produces a bad sentence. An agent that gets confused produces a bad API call, and API calls have side effects that don't undo themselves just because you're sorry.
A rough decision framework
None of this requires an elaborate rubric. Three questions cover most of the ground:
Does the system need to change anything, anywhere, or does it only need to tell someone something? If the entire job is answering, stop there — you want a grounded chatbot, not an agent. Resist the pull toward agent architecture just because the questions are varied or the domain is complex; complexity of the question is not the same as complexity of the action.
If it does need to act, how many systems does that action touch, and does step two depend on what step one found? A single deterministic action — "create a ticket with these fields" — is a function call behind a chatbot, not a reason to build an agent. Real agent architecture earns its cost when the path branches on intermediate results: look up, then decide, then act differently depending on what was found.
If it's genuinely multi-step, is the cost of a wrong action bounded and reversible? Tagging a record, drafting a message, adding an internal note — low blast radius, safe to automate with good logging. Refunding money, deleting data, changing permissions — put a human in the loop regardless of how good your evaluation numbers look, because "we tested it a lot" is not the same guarantee as "this can be undone."
A short comparison makes the trade-off visible in one place:
| | Scripted bot | RAG chatbot | AI agent | | ------------------------------------ | --------------------------- | ------------------------- | ----------------------------------- | | Handles novel phrasing | No | Yes | Yes | | Answers from real content | No (hardcoded) | Yes, with citations | Yes, when it retrieves | | Takes actions on other systems | No | No | Yes | | Needs approval gates / audit logging | No | Minimal | Yes, by design | | Build and operating cost | Low | Moderate | Highest | | Right fit when... | Questions are few and fixed | The job is answering well | The job includes acting, reversibly |
The cost of guessing wrong in either direction
Underbuilding is the more visible failure, because it shows up immediately: a business is told it's getting "an AI agent," discovers three weeks in that it can only answer questions and can't actually process the return it promised customers, and now has a support experience that overpromises and underdelivers in the worst possible place — the moment a customer is already frustrated. That failure is a scoping failure dressed up as a technical one. The system does what it was built to do; it just wasn't what was described.
Overbuilding is quieter and more expensive over time. An agent architecture brought in for a job a chatbot could handle means paying for tool integrations that see limited use, running an evaluation suite and audit logging pipeline sized for autonomous action when nothing autonomous is happening, and carrying an attack surface — every tool is a thing that can be misused — for capability nobody asked the system to exercise. It also means the team maintaining it has to keep reasoning about failure modes (compounding error across steps, idempotency on retries, prompt injection through tool-readable content) that only exist because the system can act, when a text-only answer would have closed the ticket just as well.
Both failures trace back to the same root: nobody separated "this needs to understand and answer well" from "this needs to autonomously do something to a system of record." The distinction takes ten minutes to work through properly. Skipping it costs a rebuild.
The honest way to scope this
If you're not sure which of these your business actually needs, that uncertainty is normal, and it's worth resolving before any code gets written rather than after. The test that usually settles it: write down the exact steps a human currently follows to handle the request end to end. If every step is "find the right piece of information and say it," you want a well-grounded chatbot, and you should stop worrying about whether that sounds impressive enough. If any step is "and then update something" and the steps genuinely branch on what came before, you have an agent use case — provided the actions involved are ones you'd be comfortable seeing reversed if the system gets one wrong. Scoping that honestly, before committing to an architecture, is a shorter conversation than it sounds like, and we're happy to have it before either underbuilding or overbuilding is locked in.