Building Production-Ready AI Agents

An AI agent becomes operationally significant when model output can cause external effects. Reading records, sending messages, or changing tickets introduces permissions, partial failure, duplicate execution, and recovery requirements. Treat the model as an untrusted planner inside a control system that owns state, validates actions, limits authority, and records each transition.

Architecture and control flow

A practical agent separates probabilistic decisions from deterministic execution. An orchestrator loads durable state and presents the model with the current objective plus a small set of permitted tool schemas. The model proposes one action. A policy layer validates that proposal against identity, state, budget, and risk rules. An executor performs an approved action, stores the result, and advances an explicit state machine.

1. Accept a request, authenticate the principal, and create a run with a unique identifier and bounded objective.
2. Load the current state, prior observations, remaining budgets, and permitted tools.
3. Ask the model for a structured proposal, not unrestricted code or direct system access.
4. Validate arguments, authorization, preconditions, rate limits, and approval requirements.
5. Execute through a controlled adapter using an idempotency key and a deadline.
6. Persist the action, sanitized result, state transition, and accounting data atomically where possible.
7. Continue, pause for approval, complete, or enter a recoverable failure state.

Keep business records outside model context as the source of truth. Conversation history is not durable workflow state. Persist the state, revision, principal, policy version, budgets, deadlines, approvals, and tool result references.

Model the workflow as a state machine

An explicit state machine prevents the agent from improvising lifecycle rules. States might include planning, awaiting approval, executing, verifying, completed, failed, and cancelled. Each transition has allowed origins, validated inputs, and durable effects. The orchestrator should reject a tool result that arrives after cancellation or belongs to an older state revision.

Use optimistic concurrency or row locking when several workers can resume the same run. A revision number allows an update only if the caller observed the current version. This guards against two workers both executing the next action. Long operations should use leases with expiration, but lease expiry does not prove that the remote action failed. Recovery must inspect the external system before deciding to repeat an uncertain call.

Bound tools by capability

Expose narrow tools with typed schemas and precise semantics. A tool named set_ticket_priority is easier to authorize and audit than a generic HTTP client. Validate every model supplied value on the server, including identifiers, enum values, lengths, destinations, and resource ownership. Resolve sensitive identifiers from trusted context rather than allowing the model to choose an account or tenant freely.

Tools should return structured, size limited results. Remove secrets and irrelevant fields before placing data in model context. Separate read capabilities from write capabilities, and issue short lived credentials with the least authority needed for one adapter. The model must not construct shell commands, database queries, or arbitrary URLs unless an isolated use case explicitly requires that power and supplies additional controls.

Idempotency, approvals, and retries

Every effectful action needs a stable operation identifier derived from the run and logical action, not from a retry attempt. Store the intended request before execution. The adapter should send the same idempotency key on retry and retain the external operation identifier in its receipt. If the provider lacks idempotency support, create a local operation ledger and reconcile external state before another attempt, while recognizing that local deduplication cannot eliminate every network ambiguity.

Approval is a workflow state, not a chat question. Persist the proposed action, normalized arguments, reason, policy version, approver identity, decision, and expiration. After approval, execute exactly the approved payload. If arguments change, request approval again. Use stronger approval requirements for irreversible, financial, public, or cross tenant actions.

Retry only errors classified as transient, such as selected timeouts or service unavailable responses. Use capped exponential backoff with jitter and a total deadline. Validation errors, permission denials, and policy rejections should not be retried. When an outcome is unknown, move to reconciliation rather than blindly issuing the same action.

Concrete orchestration example

def advance_run(run_id, expected_revision):
    run = store.lock(run_id, expected_revision)
    assert run.state in {"planning", "retry_ready"}
    assert run.action_count < run.max_actions
    assert now() < run.deadline

    proposal = planner.propose(
        objective=run.objective,
        state=run.safe_context,
        tools=policy.allowed_tool_schemas(run.principal)
    )
    action = policy.validate(proposal, run)

    if action.requires_approval:
        store.pause_for_approval(run, action)
        return

    operation_key = stable_key(run.id, run.next_action_number)
    store.record_intent(run, action, operation_key)

    try:
        receipt = tools.execute(
            action,
            idempotency_key=operation_key,
            timeout_ms=action.timeout_ms
        )
        verified = tools.verify(action, receipt)
        store.record_result_and_transition(run, action, verified)
    except TransientError as error:
        store.schedule_bounded_retry(run, action, error)
    except UnknownOutcome as error:
        store.mark_for_reconciliation(run, action, error)

The example gives the planner no direct executor. Policy validation and durable intent recording occur before the external call. Verification occurs afterward because an accepted request is not proof that the intended state exists.

Observability and recovery

Record a structured event for every state transition, proposal, policy decision, approval, tool attempt, retry, verification, and terminal outcome. Include run and action identifiers, revisions, durations, model and prompt versions, token counts, tool status categories, and redacted error details. Do not rely on free form model reasoning for audit. Capture concise decision summaries and observable inputs without requesting or storing hidden reasoning.

Useful metrics include completion by task type, approval frequency, policy rejection rate, tool error rate, retry count, unknown outcome count, recovery age, action count, cost, and time in each state. Distributed traces should connect model calls with policy checks and tool adapters. Alerts should focus on stuck runs, repeated effects, elevated denials, exhausted budgets, and growing reconciliation queues.

Recovery begins from durable state. A worker can resume an expired lease, inspect the last recorded intent, query the provider by idempotency key or external identifier, and choose a valid transition. Define compensating actions for reversible steps, but do not describe compensation as rollback when an external observer may already have seen the effect. Some failures require manual resolution. Provide operators with pause, cancel, retry, reconcile, and mark resolved controls, each protected and audited.

Security considerations

Treat user content, retrieved documents, web pages, tool results, and prior messages as untrusted data. They may contain prompt injection that asks the model to reveal secrets or call a powerful tool. The policy layer must enforce authority independently of prompt instructions. Never place long lived credentials in model context. Restrict network destinations, validate redirects, isolate code execution, and apply output size and resource limits.

Bind each run to an authenticated principal and tenant. Recheck authorization at execution time because permissions can change while approval is pending. Encrypt sensitive state, minimize retention, redact logs, and separate operator access from application access.

Testing and evaluation

Test the orchestrator as a distributed workflow before evaluating model quality. Unit tests should cover every legal and illegal state transition, policy rule, argument validator, budget, and retry classification. Adapter contract tests should verify idempotency behavior, timeout handling, redaction, and reconciliation. Fault injection should simulate worker crashes before and after an external call, duplicate queue delivery, stale revisions, delayed approvals, malformed tool results, and provider outages.

Scenario evaluations should measure whether the planner selects permitted tools, supplies valid arguments, respects stopping conditions, requests clarification, and avoids unnecessary actions. Include adversarial content in every untrusted channel. Track severe failures separately from average task completion, and review traces to distinguish planning errors from policy, adapter, or infrastructure defects. A shadow or simulation environment can exercise workflows without allowing real effects.

Failure modes and compromises

Common mistakes include giving the model a generic network tool, encoding workflow state only in chat history, retrying all exceptions, assuming a timeout means no effect, and asking for approval without freezing the payload. Other failures include unlimited planning loops, tool responses that overflow context, credentials shared across tenants, and logs that capture sensitive arguments.

Narrow tools reduce flexibility but make policy and testing tractable. More approval steps reduce autonomy and increase delay, while too few approvals raise the consequence of a model error. Verification and reconciliation add latency and storage, but they address ambiguity that prompts cannot solve. Durable state machines require more engineering than an open loop agent, yet they create clear boundaries for retries, cancellation, observability, and recovery.

Conclusion

A production ready agent is a controlled workflow system with a model inside it. Bounded tools limit capability, explicit states govern progress, idempotency reduces duplicate effects, and persisted approvals protect consequential actions. Conservative retries, verification, observability, and reconciliation make partial failure manageable. When authority and lifecycle rules remain in deterministic code, the model can contribute flexible planning without becoming the system's security boundary or source of truth.