When an AI Agent Outgrows Its Prompt: Tools, Skills, and Sub-Agents
A growing prompt and a pile of bolted-on tools can make a once-reliable AI agent slower, costlier, and less accurate. Here is how to decide what belongs in a tool, a skill, or a sub-agent, and how to measure the improvement.
Many teams ship an AI agent that does one job well. Then the business asks for forecasting, supplier selection, and weekly reports. Before long the prompt runs to hundreds of lines, the agent has a dozen tools and several sub-agents, and work that used to be reliable starts to slip.
This is usually a design problem, not a model problem. How you split work across a prompt, tools, skills, and sub-agents matters as much as the model.
Why bolted-on agents start to fail
The extra capabilities are rarely hard on their own. The trouble comes from stacking them into one ever-growing agent without revisiting the architecture.
Familiar failure patterns follow. The agent reaches the right answer only after a long, inefficient path. A sub-agent does its part correctly, then the handoff back to the main agent goes wrong. Rules added at different times contradict each other. Or the agent loads the right numbers and still produces the wrong calculation because the surrounding instructions conflict.
That last case is telling. An inventory agent might pull the correct baseline forecast and the correct promotion multiplier, then apply a completely different figure in the final step. The model did not forget how to multiply. It was surrounded by too much context, including rules that did not agree. In operations work, even a modest failure rate is expensive.
Measure first, then change the design
Before you rewrite anything, write evaluations for the jobs you actually care about. Mix simple, realistic tasks with harder multi-step scenarios, and track both hard numbers and softer qualities: steps taken, run time, token use, and judged scores for tone, structure, and output quality.
Establish a baseline, change one part of the design, and run the same suite again. That is how you know a shorter prompt or a smaller tool set is helping, not just looking cleaner. As the product grows, the evaluations have to grow with it. If you add forecasting, you need tests for forecasting, including awkward cases such as a promotion month.
Keep the system prompt short; put procedures in skills
A system prompt should hold only what the agent needs on every turn. Role, standing constraints, and how to ask for help belong there. Detailed policies do not.
Skills — packaged instructions the agent can load when a task requires them — are a better place for procedures that apply only some of the time. Forecasting rules should appear when the agent is asked to forecast, not when it is checking stock or drafting a weekly report.
This is progressive disclosure: a short standing brief, plus the relevant playbook only when it is needed. The context window stays cleaner, and the model has more room for the actual work. If your prompt has been accumulating every new business rule, move those rules into skills and shrink the standing prompt to the essentials.
Start with simple tools, then add custom ones only when needed
It is tempting to wrap every action in a custom tool: one to fetch data, one to analyze it, one to write a file. A better starting point is the same basic capabilities a person has at a desk: read and write files, run code, search the web if the job requires it, and keep a short to-do list for longer tasks.
Those primitives go a long way. If the agent needs to work across spreadsheets or CSV exports, let it write a short script, run it, and read the result, rather than pouring the entire file into the prompt. That shift can cut token use and cost sharply, and it often shortens the run as well.
Add a custom tool when a primitive is not enough, such as a governed action against your inventory system or a carefully scoped way to file a purchase order. Remove primitives you do not need. A purely internal agent may not need web search.
Be cautious about rushing to a large set of shared tool servers. Overlapping collections can crowd the context and confuse the agent. Start with simple primitives, add local tools that only this agent needs, and publish a shared, governed layer only when several agents need the same actions. In some cases, the agent can call an API or command-line interface through code instead of loading a broad tool catalog into the prompt.
Use sub-agents sparingly
Sub-agents are useful in two situations. The first is when the work benefits from several passes in parallel: deep research, a wide search, or a large exploration of records. The second is when you need a fresh context. The same agent that has been chatting with a user should not also be the only reviewer of its own work. A forecast is another good example: you may want that calculation isolated so earlier conversation does not color the numbers.
Outside those cases, many teams can fold work back into the main agent. Models can now handle more in one loop, and every extra sub-agent creates a handoff that can fail. Communication between the orchestrator and a sub-agent is a common breaking point: the specialist may be right, and the result still gets lost on the way back. If you keep a sub-agent, make the handoff and the traces first-class, not an afterthought wrapped inside a vague tool.
What to take away
Start with one agent loop and a small set of human-like tools, then grow from there. Keep the standing prompt short, and package procedures as skills the agent can load when needed. Add custom tools only when a primitive cannot do the job safely. Reach for a sub-agent when you need parallelism or a clean, separate context, not as the default way to add a feature. Write evaluations for the work that matters, get a baseline, and rerun them after every design change, updating the suite as the product grows.
A simpler design is often faster, cheaper, and more accurate, because the model is no longer wading through conflicting instructions and unused tools. For demanding work such as forecasting, a slightly longer run can still be worthwhile if quality rises and cost falls.
If an agent in production has picked up too many jobs, this is a good time to redesign it rather than adding another line to the prompt. Xprolabs helps teams put agentic AI into production and keep it reliable as the work grows. If you want a second look at an agent that is starting to drift, we would be glad to talk.