A demo works once. A product works every day, for many users, within a budget.
Architecture
- Keep AI calls on the server; the front end only talks to your API.
- Put the model behind a provider abstraction so you can change model or vendor without rewriting features.
- Store prompts in files with versions.
- Use background jobs and queues for slow work, and stream results where users wait.
- Keep conversation and session data in your database.
Control cost
- Log tokens and cost for each request, user and feature.
- Set per-user quotas and plan limits.
- Cache repeated answers and use prompt caching for repeated prefixes.
- Model routing: send easy tasks to a small model and hard tasks to a larger one.
Reliability
- Retries with exponential backoff and timeouts.
- Circuit breakers to stop calling a failing service.
- A fallback model or provider for critical features.
- Concurrency limits so a spike does not exhaust rate limits.
Monitor
- Latency (including time to first token), error rates and token usage.
- Quality signals: user feedback, thumbs up or down, and sample reviews.
- Alerts for sudden cost or error spikes.
Ship safely
- Test prompts and models with your evaluation set before release.
- Roll out to a small share of users first and run A/B tests.
- Watch model deprecation notices and plan migrations.
- Keep secrets in a secrets manager, never in code.
Incident response
Decide in advance how you will turn an AI feature off (a feature flag), who is told and how you will communicate with users.