How to Productionise a Stateful LLM Platform Without Rewriting the Product
Stateful LLM platform production architecture is the set of boundaries that makes an AI application safe to operate repeatedly for several users. It covers authoritative data, identity, object access, expensive background work, external interfaces, deployment, recovery and verification. The model call is only one component inside that operating system.
This article owns one specific question: how do you move a feature-rich, stateful LLM application from local research use into a controlled shared pilot? It does not decide whether you need a knowledge graph, which is covered by our graph engineering decision guide. It also does not repeat the generic prototype-to-production checklist. The focus here is the dependency order for a platform that already has valuable product logic.
What makes a stateful LLM application different from a demo?
A demo can keep context in a browser session, reload fixtures from files and wait synchronously for a model response. A shared platform cannot. Users expect their data to survive a deployment, their permissions to apply through every interface and a long-running operation to remain understandable after a timeout, refresh or failure.
The starting application in this engagement already had substantial logic for structured domain data, graph operations, document ingestion, grounded conversations, interviews, simulations and evaluation. Its gap was not a missing feature list. The gap was operational ownership across the paths that connect those features.
| Research-friendly assumption | Shared-platform requirement | Failure if left implicit |
|---|---|---|
| Files are the live state | One authoritative transactional store | Deployments or concurrent users produce divergent data. |
| One trusted operator | Server-validated identity and object access | A valid user can reach another user's data. |
| The request waits for the model | Owned, observable and recoverable jobs | Timeouts look like failures and retries duplicate work. |
| The browser is the only client | Consistent policy across browser, API and MCP | A new interface bypasses assumptions enforced only in the UI. |
| A running process is healthy | Dependency-aware readiness and rollback | Traffic reaches an application that cannot serve correct results. |
| Manual testing proves the release | Automated checks plus representative flow QA | Cross-layer regressions recur after every change. |
The seven-layer productionisation sequence
The order matters. Polishing the interface before persistence and access control only makes unstable behavior easier to demonstrate. Adding an external API before central authorization creates a second security model. The programme therefore established each foundation before expanding the surface above it.
1. Define the operating envelope before choosing infrastructure
The target was a controlled initial release, not a claim of globally redundant enterprise infrastructure. That decision fixed the immediate acceptance criteria: a repeatable HTTPS deployment, persistent data, controlled onboarding, recoverable releases, useful logs and an explicit list of remaining scale and availability work.
This wording is more than risk management. It prevents architecture theatre. A single application task and a single-zone database may be reasonable for a design-partner pilot when their failure modes, backups and escalation paths are documented. They are not high availability. Naming the boundary lets the team buy the next reliability step when usage justifies it.
2. Move authoritative runtime state into PostgreSQL
The application had portable seed files and local data flows that were useful during research. Wavect retained those files as controlled bootstrap material while moving live subjects, ontologies, user records, organisations, access grants, sessions and machine credentials into PostgreSQL-backed stores.
The important design choice was ownership, not the database brand. Each domain received a store or repository contract, pooled connections and explicit create, read, update and delete behavior. Startup could seed an empty environment deterministically, but a running container no longer treated mutable files as the source of truth. Database integration tests then exercised the same persistence behavior the shared deployment used.
3. Centralize identity, tenant membership and object authorization
Authentication proves who sent a request. It does not prove that person may open a particular subject, graph or job. The platform therefore combined revocable server-side browser sessions, administrative roles, organisation membership and per-user object grants behind one access decision path. Browser routes, JSON APIs and external clients had to ask the same question.
This is the boundary described by OWASP API1:2023 Broken Object Level Authorization: every endpoint that receives an object identifier must verify that the logged-in user may perform the requested action on that object. UUIDs, a valid JWT or a hidden button do not replace that check.
The practical test matrix crossed roles, organisations, objects and transports. A regular user should not gain access by changing an identifier. An organisation administrator should remain inside that organisation. A revoked session or API key should stop working without waiting for a deployment.
4. Treat long-running LLM work as owned jobs
Document extraction, interviews, profile generation and simulations can outlive a normal HTTP request. Extending a web-server timeout reduces one symptom but does not answer who owns the work, whether a retry is safe or how the user learns what happened.
Wavect introduced a backend-independent job contract with typed states, handler registration, per-user concurrency limits, user-scoped idempotency keys, cooperative cancellation, stored results and errors, completion expiry and graceful shutdown. Polling endpoints provided a simple baseline. Server-Sent Events added application progress and heartbeat updates without making the browser connection the owner of the job.
| Job property | Question it must answer | Release evidence |
|---|---|---|
| Ownership | Which authenticated user can observe or cancel it? | Cross-user access tests fail closed. |
| Idempotency | What happens when the same action is submitted twice? | The duplicate maps to one intended execution. |
| Progress | Can the user distinguish active, stalled, failed and complete? | Polling and event paths expose the same state. |
| Cancellation | Can expensive work stop at a safe boundary? | The handler cooperates and records the terminal state. |
| Shutdown | What happens during a deployment? | The queue drains or marks unfinished work for recovery. |
An in-process executor was an intentional pilot trade-off, not the final scaling story. The contract made a later move to a durable queue possible without rewriting every product flow. The key is to separate the job lifecycle from the first backend that implements it.
5. Give browser, API and MCP access explicit security contracts
The external API was reorganised into modular route groups with shared dependencies, standard errors, scope checks, per-object authorization and throttling. Human sessions, JWTs and hashed API keys served different clients, but all of them converged on the same domain permissions. Administrative settings could disable external access for a specific object instead of exposing every capability by default.
MCP needs the same discipline. A tool call is not trusted because an agent initiated it. Identifier validation, read and write scope separation, rate limits and sanitized errors still apply. The current Model Context Protocol authorization specification requires resource-bound authorization requests and token audience validation. New integrations should follow the current protocol rather than copying the session or transport assumptions of an older implementation.
6. Make deployment health depend on the service users need
Infrastructure as code defined the network, container service, private database, load balancer, TLS, DNS, image registry, logs and permissions. A commit-triggered workflow built a versioned image, updated the service and waited for stability. The application health endpoint checked database connectivity, so a live process with an unavailable authoritative store did not count as ready.
Deployment controls need a failure action. AWS documents deployment circuit breakers and CloudWatch alarms as mechanisms that can detect failed ECS deployments and roll them back to the last known good state. In this programme, health-aware deployment and rollback were part of the release design, not an operator's improvised response.
7. Combine automated tests with repeated browser QA
Unit tests covered isolated graph, retrieval, grounding, safety and job behavior. A separate integration track used a real PostgreSQL service for persistence, authentication, API and MCP paths. That separation kept fast feedback fast without mocking away the boundaries most likely to fail in the shared environment.
Automation did not replace exploratory QA. Repeated browser passes exposed interaction failures across login expiry, selection state, long-running progress, graph editing, calculations, theme synchronization, errors and loading behavior. Each defect was fixed at the lowest stable owner, then covered by a regression check where practical.
This risk-ordered approach aligns with the NIST Secure Software Development Framework, which organizes secure development around preparing the organisation, protecting software, producing well-secured releases and responding to residual vulnerabilities. NIST explicitly frames the practices as an adaptable basis for risk-based improvement, not a universal checklist.
What evidence makes a controlled pilot credible?
Production readiness is not a binary property of the repository. It is a release claim supported by evidence across the operating envelope. For this class of platform, the minimum useful evidence set looks like this:
| Boundary | Evidence before release | Residual risk to state |
|---|---|---|
| Data | Persistence and seed tests against PostgreSQL, plus a backup and restore procedure | Migration and recovery maturity |
| Access | Role, tenant and object tests across session and token clients | Administrative misuse and future policy growth |
| LLM jobs | Duplicate, cancellation, timeout, error and shutdown behavior | Process-local queue durability and capacity |
| External interfaces | Scope, rate-limit, invalid-input and sanitized-error tests | Protocol evolution and third-party client behavior |
| Deployment | Health-aware rollout, observable failure and tested rollback | Single-task and single-zone availability |
| User journeys | Representative browser flows with recovery from expired or partial states | Unseen combinations in a large interactive interface |
Why incremental productionisation beat a rewrite
The existing application already encoded hard-won product behavior. A rewrite would have exchanged visible technical debt for hidden product regression risk. The faster path was to preserve validated flows, install stable ownership below them and repair defects against the new foundation.
That does not mean every old choice survived. File-backed runtime state stopped being authoritative. Access checks moved out of individual routes. Long work stopped belonging to a request. API dependencies became explicit. Shared interface patterns replaced local fixes. Incremental productionisation is selective replacement with an evidence trail, not endless patching.
If your product has the same shape, Wavect's AI enablement and architecture service covers evaluation, system boundaries, production hardening and handover. The anonymized analytics platform case study provides a separate example of stabilizing a complex data product without naming the client. For a wider keep-versus-rebuild decision, use the prototype-to-production guide, or request a production architecture review with your current operating constraints.
Stateful LLM platform production FAQ
What is a stateful LLM platform?
What should be productionised first in an LLM application?
Should long-running LLM work stay inside an HTTP request?
How should an MCP server access platform data?
Does a successful pilot mean the platform is highly available?
When is a rewrite justified?
Final thoughts
A stateful LLM platform becomes operable when every important state and authority has an owner. PostgreSQL owns durable runtime data. A central policy owns tenant and object access. The job layer owns expensive work. API and MCP routes reuse those decisions. Deployment health reflects real dependencies, and tests plus browser QA supply release evidence.
Keep the product behavior that users already value. Replace the foundations that cannot support shared operation. Most importantly, name the operating envelope honestly. A controlled pilot with explicit trade-offs is a stronger engineering result than an undefined production claim.