How to host AI agents in production without a science project

Most agent demos die in the gap between a notebook and a URL that someone else can hit. Hosting is not “put the model behind a load balancer.” It is isolation, secrets, queues, health, and a way to undo a bad prompt at 2 a.m.

Give every agent an identity

Treat an agent like a small service: a name, a version, a region, and an endpoint. Health should be a dedicated route that does not invoke the expensive path. If the health check triggers a full LLM call, you will page yourself into bankruptcy.

• Versioned endpoints so clients pin a known-good release
• Secrets injected at runtime, never baked into prompts in git
• Timeouts and retries at the tool boundary, not only at HTTP
• A kill switch that returns a safe fallback, not a 500

Separate the thinking path from the living path

The living path is heartbeat, queue depth, and last successful tool call. The thinking path is the LLM. Host them so a model outage does not look like a total blackout — and so you can see which one actually failed.

If you cannot answer “is it up?” in under a second, you do not host the agent. You host a mystery.

Ship a rollback, not a rewrite

Prompt and tool changes should roll forward and back like code. Keep the last two versions warm. Production hosting is enablement: your team can revert without waiting on a lab.