A convincing agent demo takes an afternoon. A production agent that customers can't break takes months, and the gap between the two is where most AI projects quietly die. Here is where the real work actually goes, learned the expensive way.
Tool design eats more time than prompting. An agent is only as good as the actions it can take, and every tool needs crisp inputs, unambiguous failure modes, and (this is the part teams skip) a description written for the model, not for your API docs. Rewriting tool descriptions alone, touching no code, is routinely the highest-leverage change available.
Guardrails are product decisions wearing engineering clothes. What may the agent never say? When must it hand off to a human? What happens when a user tries to talk it out of its instructions? Decide these explicitly, in writing, before launch, because the alternative is deciding them in an incident channel after.
Evals are the part everyone knows about and almost nobody does. Ours are unglamorous: a set of real transcripts, graded on 'did it actually resolve the thing,' re-run on every prompt change. The single most common cause of agent regressions is a prompt tweak that fixed one conversation and silently broke forty others. Evals catch that; vibes don't.
The UI carries more weight than the model. Streaming responses, visible tool calls, a clear 'here's what I can help with' opening, and a one-click path to a human. These set the expectations that determine whether users experience the same model as magical or broken.
Start narrower than feels impressive. Ship an agent that handles three intents well before one that handles thirty badly, and make each expansion earn its place with data. Iris, the concierge in the corner of this site, follows exactly those rules: scoped, streaming, honest about its limits. Ask it something outside its lane and watch it decline politely; that refusal took longer to build than the answers did.
