Build the company around one valuable workflow, portable model access, controlled customer context, and measurable reliability. Treat models as replaceable dependencies while owning the business logic, integrations, evaluation data, and customer experience.
1. Start with a narrow business outcome
Choose one recurring task with a clear buyer, accessible data, and a verifiable result—for example, preparing a support resolution, reviewing a contract against company policy, or implementing a small code change.
Before selecting models, establish:
| Question | Required decision |
|---|---|
| Who uses and buys it? | Named user role and budget owner |
| What work improves? | One end-to-end workflow |
| What constitutes success? | A result users can verify |
| What actions are consequential? | Actions requiring approval |
| What is the current baseline? | Time, cost, error rate, and manual effort |
| What data is permitted? | Approved sources, tenants, regions, and providers |
Measure completed work and user corrections—not conversation volume. Begin with assisted execution and expand autonomy as evidence supports it.
2. Separate the product from models and agent runtimes
Use this architecture:
```mermaid
flowchart TB
Experience["Workflow interface and integrations"]
Policy["Identity, permissions, and data policy"]
Workflow["Workflow orchestration and task state"]
Context["Context and retrieval"]
Tools["Validated business tools"]
Runtime["Agent/runtime adapters"]
Gateway["Model gateway and routing"]
Models["Approved hosted or local models"]
Evaluation["Evaluation and operational telemetry"]
Experience --> Policy --> Workflow
Workflow <--> Context
Workflow <--> Tools
Workflow --> Runtime --> Gateway --> Models
Workflow --> Evaluation
Gateway --> Evaluation
```Keep domain logic independent of provider SDKs. Define stable internal interfaces for model requests, structured results, streaming events, errors, usage, and tool calls. Provider-specific features belong in adapters.
Portability must cover agent runtimes as well as models. Conversation formats, tool semantics, compaction, and approval behavior can create dependencies even when model endpoints are interchangeable.
Start with one primary provider and one tested alternative. Use a gateway where it simplifies routing and governance; avoid building a complex routing platform before the first workflow works.
Route in this order:
Customer policy → Required capabilities → Quality threshold → Latency → CostNever let a cheaper model or automatic fallback bypass a customer’s approved region, provider, or retention requirements. OpenRouter, for example, exposes provider-selection and retention-related routing controls that require explicit configuration.
3. Make enterprise controls part of the architecture
Maintain an approved deployment matrix for each customer:
| Control | What to establish |
|---|---|
| Identity and authorization | Tenant membership, roles, and access to individual resources |
| Data location | Processing, storage, backups, telemetry, and support access |
| Provider approval | Approved endpoint, model, region, and subprocessors |
| Retention | Rules for prompts, responses, tool outputs, traces, and application records |
| Credentials | Managed credentials or BYOK, with rotation and revocation |
| Execution | Tool permissions, workspace isolation, network access, and approval requirements |
Do not equate U.S. hosting, a private network, open weights, or BYOK with compliance. Each addresses only part of the problem.
Likewise, distinguish no training, no retention, and regional processing. They are separate commitments. OpenAI’s retention controls include eligibility and endpoint-specific conditions; AWS Bedrock distinguishes geographic from global cross-region routing. Validate the exact service configuration and contract.
Offer deployment choices progressively:
- Standard managed service for early customers.
- Customer-owned credentials and approved enterprise endpoints.
- Dedicated/private deployments when contracts justify their operating cost.
- Local execution where hardware, model quality, and support requirements make it viable.
4. Build defensibility through context and reliable execution
Invest in three product assets.
| Asset | Engineering focus |
|---|---|
| Workflow integration | Read and write through the systems users already depend on |
| Context management | Permission-aware retrieval, freshness, provenance, and explicit memory |
| Reliable orchestration | Durable task state, validated actions, recovery, and human review |
Start context management with authoritative connectors, structured records, and searchable documents. Add knowledge graphs only when relationships and multi-step queries demonstrably improve the workflow.
RAG is not inherently deterministic or accurate. Evaluate retrieval quality separately from answer quality, and enforce authorization before context enters a prompt. Treat retrieved documents and tool outputs as untrusted content.
Use deterministic controls around model behavior:
- Validate structured outputs at runtime; TypeScript types alone do not validate incoming JSON.
- Check business rules before executing tools.
- Require approvals for defined consequential actions.
- Bound retries, execution time, and spending.
- Use idempotency and recorded action state to prevent duplicate writes.
- Provide cancellation, escalation, and recovery paths.
Retrying a model response and retrying a payment, message, or database update require different policies.
5. Establish evaluation and economics before scaling
Create an initial evaluation set from expert-written examples and permissioned, appropriately sanitized real usage. Include normal tasks, ambiguous requests, missing context, malicious instructions, tool failures, and corrected outputs.
Keep a held-out set separate from examples used to tune prompts. Combine automated checks, human review, and calibrated model-based grading.
Every change to models, prompts, retrieval, tools, or orchestration should pass release gates:
| Dimension | Measure |
|---|---|
| Business quality | Verified task completion and correction rate |
| Grounding | Unsupported claims and source accuracy |
| Execution | Tool correctness, duplicate actions, and recovery |
| Security | Unauthorized access/action cases and data-boundary violations |
| Performance | End-to-end p50/p95 latency and time to first useful result |
| Economics | Total cost per successful workflow |
| Portability | Equivalent workflow results through the alternative provider |
Define thresholds per workflow and risk level. Require all critical authorization tests to pass and compare quality against the current production baseline.
Continuous evaluation should incorporate new production failures and user corrections over time.
Track costs beyond tokens: retrieval, retries, tool execution, storage, infrastructure, and human review. Optimize cost per successful outcome, rather than cost per request.
6. Deliver through a staged 90-day programme
| Phase | Deliverables | Exit criterion |
|---|---|---|
| Weeks 1–2: Validate | Workflow definition, baseline measurements, design partners, initial evaluation set | Users confirm the problem and acceptance criteria |
| Weeks 3–6: Build | One complete workflow, primary/alternative adapters, context connectors, approvals, tracing | Workflow passes agreed quality and security gates |
| Weeks 7–10: Pilot | Tenant controls, approved deployment configuration, recovery, support procedures | Real users demonstrate measurable value |
| Weeks 11–12: Optimize | Routing experiments, cost analysis, provider failover tests, controlled rollout | Sustainable economics and reliable operation |
Assign explicit ownership for workflow quality, platform reliability, security/data governance, and evaluation. A small team can combine roles, but each outcome needs an owner.
Deployment trade-offs
| Approach | Best fit | Principal constraint |
|---|---|---|
| Frontier APIs | Rapid validation and demanding reasoning | Variable cost, external dependencies, and feature portability |
| Managed open-weight hosting | Workloads benefiting from model/deployment control | Quality evaluation, capacity planning, licensing, and operations |
| Client-side/hybrid | Offline or privacy-sensitive workflows | Hardware variability, distribution, update management, and model limits |
Local inference can reduce network dependence; it does not guarantee low latency or zero data exposure.
For the existing Codex powered projects, retain the desktop interface as a workflow surface. Add application-owned context, task records, evaluation, and policy boundaries around the runtime. Route models through supported Codex provider configuration, while keeping a separate runtime adapter so the product is not permanently coupled to Codex’s protocol or storage.