--- title: "Stateful LLM Platform Production Architecture" canonical: https://wavect.io/blog/stateful-llm-platform-production-architecture/ language: en description: "A field-tested architecture for productionising a stateful LLM platform: persistence, tenant access, background jobs, APIs, deployment and continuous QA." image: "https://wavect.io/img/blog/headers/header_stateful-llm-platform-production-architecture.png" --- [**Back**](/blog/overview/) [![Christof Jori](/img/team/christof.webp)](/team/christof-jori/) [Christof Jori](/team/christof-jori/) https://linkedin.com/in/jocr77 11 min read · 14 Aug 2026 Last reviewed August 14, 2026 [**Next**](/blog/fintech-architecture-remediation-without-rewrite/) # How to Productionise a Stateful LLM Platform Without Rewriting the Product TL;DR Productionising a stateful LLM platform is not mainly a model-serving task. The hard part is making state, identity, long-running work, external access, deployment and recovery explicit. In this anonymized field engagement, Wavect kept the working product logic and hardened the foundations in dependency order: define a controlled operating boundary, move authoritative runtime state to PostgreSQL, centralize tenant and object authorization, place expensive LLM work behind owned and idempotent jobs, secure API and MCP access separately, deploy from infrastructure as code with dependency-aware health checks and rollback, then use automated tests and repeated browser QA as one release system. The result was a credible controlled pilot, not a claim of high availability. That distinction matters: production readiness is a documented operating envelope with evidence and residual risks, not a green build or polished interface. **Stateful LLM platform production architecture is the set of boundaries that makes an AI application safe to operate repeatedly for several users.** It covers authoritative data, identity, object access, expensive background work, external interfaces, deployment, recovery and verification. The model call is only one component inside that operating system. This article owns one specific question: how do you move a feature-rich, stateful LLM application from local research use into a controlled shared pilot? It does not decide whether you need a knowledge graph, which is covered by our [graph engineering decision guide](/blog/graph-engineering-ai-agents/). It also does not repeat the generic [prototype-to-production checklist](/software-development-guide/vibe-coded-prototype-to-production/). The focus here is the dependency order for a platform that already has valuable product logic. **Anonymization boundary:** This is a field case from Wavect delivery records, not a composite. We omit the client name, product category, geography, internal feature names, production hostnames, exact topology and unresolved security details. The technology families, productionisation sequence, failure patterns and trade-offs are real. No client outcome or industry benchmark is implied. ## What makes a stateful LLM application different from a demo? A demo can keep context in a browser session, reload fixtures from files and wait synchronously for a model response. A shared platform cannot. Users expect their data to survive a deployment, their permissions to apply through every interface and a long-running operation to remain understandable after a timeout, refresh or failure. The starting application in this engagement already had substantial logic for structured domain data, graph operations, document ingestion, grounded conversations, interviews, simulations and evaluation. Its gap was not a missing feature list. The gap was operational ownership across the paths that connect those features. | Research-friendly assumption | Shared-platform requirement | Failure if left implicit | | --- | --- | --- | | Files are the live state | One authoritative transactional store | Deployments or concurrent users produce divergent data. | | One trusted operator | Server-validated identity and object access | A valid user can reach another user's data. | | The request waits for the model | Owned, observable and recoverable jobs | Timeouts look like failures and retries duplicate work. | | The browser is the only client | Consistent policy across browser, API and MCP | A new interface bypasses assumptions enforced only in the UI. | | A running process is healthy | Dependency-aware readiness and rollback | Traffic reaches an application that cannot serve correct results. | | Manual testing proves the release | Automated checks plus representative flow QA | Cross-layer regressions recur after every change. | ## The seven-layer productionisation sequence The order matters. Polishing the interface before persistence and access control only makes unstable behavior easier to demonstrate. Adding an external API before central authorization creates a second security model. The programme therefore established each foundation before expanding the surface above it. ### 1. Define the operating envelope before choosing infrastructure The target was a controlled initial release, not a claim of globally redundant enterprise infrastructure. That decision fixed the immediate acceptance criteria: a repeatable HTTPS deployment, persistent data, controlled onboarding, recoverable releases, useful logs and an explicit list of remaining scale and availability work. This wording is more than risk management. It prevents architecture theatre. A single application task and a single-zone database may be reasonable for a design-partner pilot when their failure modes, backups and escalation paths are documented. They are not high availability. Naming the boundary lets the team buy the next reliability step when usage justifies it. ### 2. Move authoritative runtime state into PostgreSQL The application had portable seed files and local data flows that were useful during research. Wavect retained those files as controlled bootstrap material while moving live subjects, ontologies, user records, organisations, access grants, sessions and machine credentials into PostgreSQL-backed stores. The important design choice was ownership, not the database brand. Each domain received a store or repository contract, pooled connections and explicit create, read, update and delete behavior. Startup could seed an empty environment deterministically, but a running container no longer treated mutable files as the source of truth. Database integration tests then exercised the same persistence behavior the shared deployment used. ### 3. Centralize identity, tenant membership and object authorization Authentication proves who sent a request. It does not prove that person may open a particular subject, graph or job. The platform therefore combined revocable server-side browser sessions, administrative roles, organisation membership and per-user object grants behind one access decision path. Browser routes, JSON APIs and external clients had to ask the same question. This is the boundary described by [OWASP API1:2023 Broken Object Level Authorization](https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/): every endpoint that receives an object identifier must verify that the logged-in user may perform the requested action on that object. UUIDs, a valid JWT or a hidden button do not replace that check. The practical test matrix crossed roles, organisations, objects and transports. A regular user should not gain access by changing an identifier. An organisation administrator should remain inside that organisation. A revoked session or API key should stop working without waiting for a deployment. ### 4. Treat long-running LLM work as owned jobs Document extraction, interviews, profile generation and simulations can outlive a normal HTTP request. Extending a web-server timeout reduces one symptom but does not answer who owns the work, whether a retry is safe or how the user learns what happened. Wavect introduced a backend-independent job contract with typed states, handler registration, per-user concurrency limits, user-scoped idempotency keys, cooperative cancellation, stored results and errors, completion expiry and graceful shutdown. Polling endpoints provided a simple baseline. Server-Sent Events added application progress and heartbeat updates without making the browser connection the owner of the job. | Job property | Question it must answer | Release evidence | | --- | --- | --- | | Ownership | Which authenticated user can observe or cancel it? | Cross-user access tests fail closed. | | Idempotency | What happens when the same action is submitted twice? | The duplicate maps to one intended execution. | | Progress | Can the user distinguish active, stalled, failed and complete? | Polling and event paths expose the same state. | | Cancellation | Can expensive work stop at a safe boundary? | The handler cooperates and records the terminal state. | | Shutdown | What happens during a deployment? | The queue drains or marks unfinished work for recovery. | An in-process executor was an intentional pilot trade-off, not the final scaling story. The contract made a later move to a durable queue possible without rewriting every product flow. The key is to separate the job lifecycle from the first backend that implements it. ### 5. Give browser, API and MCP access explicit security contracts The external API was reorganised into modular route groups with shared dependencies, standard errors, scope checks, per-object authorization and throttling. Human sessions, JWTs and hashed API keys served different clients, but all of them converged on the same domain permissions. Administrative settings could disable external access for a specific object instead of exposing every capability by default. MCP needs the same discipline. A tool call is not trusted because an agent initiated it. Identifier validation, read and write scope separation, rate limits and sanitized errors still apply. The current [Model Context Protocol authorization specification](https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization) requires resource-bound authorization requests and token audience validation. New integrations should follow the current protocol rather than copying the session or transport assumptions of an older implementation. ### 6. Make deployment health depend on the service users need Infrastructure as code defined the network, container service, private database, load balancer, TLS, DNS, image registry, logs and permissions. A commit-triggered workflow built a versioned image, updated the service and waited for stability. The application health endpoint checked database connectivity, so a live process with an unavailable authoritative store did not count as ready. Deployment controls need a failure action. [AWS documents deployment circuit breakers and CloudWatch alarms](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-failure-detection.html) as mechanisms that can detect failed ECS deployments and roll them back to the last known good state. In this programme, health-aware deployment and rollback were part of the release design, not an operator's improvised response. ### 7. Combine automated tests with repeated browser QA Unit tests covered isolated graph, retrieval, grounding, safety and job behavior. A separate integration track used a real PostgreSQL service for persistence, authentication, API and MCP paths. That separation kept fast feedback fast without mocking away the boundaries most likely to fail in the shared environment. Automation did not replace exploratory QA. Repeated browser passes exposed interaction failures across login expiry, selection state, long-running progress, graph editing, calculations, theme synchronization, errors and loading behavior. Each defect was fixed at the lowest stable owner, then covered by a regression check where practical. This risk-ordered approach aligns with the [NIST Secure Software Development Framework](https://csrc.nist.gov/projects/ssdf), which organizes secure development around preparing the organisation, protecting software, producing well-secured releases and responding to residual vulnerabilities. NIST explicitly frames the practices as an adaptable basis for risk-based improvement, not a universal checklist. ## What evidence makes a controlled pilot credible? Production readiness is not a binary property of the repository. It is a release claim supported by evidence across the operating envelope. For this class of platform, the minimum useful evidence set looks like this: | Boundary | Evidence before release | Residual risk to state | | --- | --- | --- | | Data | Persistence and seed tests against PostgreSQL, plus a backup and restore procedure | Migration and recovery maturity | | Access | Role, tenant and object tests across session and token clients | Administrative misuse and future policy growth | | LLM jobs | Duplicate, cancellation, timeout, error and shutdown behavior | Process-local queue durability and capacity | | External interfaces | Scope, rate-limit, invalid-input and sanitized-error tests | Protocol evolution and third-party client behavior | | Deployment | Health-aware rollout, observable failure and tested rollback | Single-task and single-zone availability | | User journeys | Representative browser flows with recovery from expired or partial states | Unseen combinations in a large interactive interface | ## Why incremental productionisation beat a rewrite The existing application already encoded hard-won product behavior. A rewrite would have exchanged visible technical debt for hidden product regression risk. The faster path was to preserve validated flows, install stable ownership below them and repair defects against the new foundation. That does not mean every old choice survived. File-backed runtime state stopped being authoritative. Access checks moved out of individual routes. Long work stopped belonging to a request. API dependencies became explicit. Shared interface patterns replaced local fixes. Incremental productionisation is selective replacement with an evidence trail, not endless patching. If your product has the same shape, Wavect's [AI enablement and architecture service](/services/ai-enablement/) covers evaluation, system boundaries, production hardening and handover. The anonymized [analytics platform case study](/case-studies/bond-analytics/) provides a separate example of stabilizing a complex data product without naming the client. For a wider keep-versus-rebuild decision, use the [prototype-to-production guide](/software-development-guide/vibe-coded-prototype-to-production/), or [request a production architecture review](/contact/) with your current operating constraints. ## Stateful LLM platform production FAQ ### What is a stateful LLM platform? It is an AI application whose useful behavior depends on durable domain data, user or tenant identity, permissions, job state, history or connected records across requests. The model is one component; the platform must also own persistence, access, execution and recovery. ### What should be productionised first in an LLM application? Define the operating envelope first, then establish authoritative persistence and one authorization path. Background jobs, external APIs, deployment automation and wider interface QA depend on those foundations, so implementing them earlier creates duplicate state and policy models. ### Should long-running LLM work stay inside an HTTP request? Usually not when it can exceed normal request time, be retried, cancelled or observed after a refresh. Put it behind an owned job with explicit states, idempotency, progress, cancellation, result storage and shutdown behavior. The first backend can be simple if the contract allows later replacement. ### How should an MCP server access platform data? Treat MCP as another external interface. Validate identifiers, authenticate the client, bind tokens to the intended resource, enforce object permissions and read or write scopes, rate-limit calls and sanitize errors. Do not rely on an agent or client interface to enforce server-side policy. ### Does a successful pilot mean the platform is highly available? No. A controlled pilot can be production-like and useful with a narrower availability target. Document task count, database topology, queue durability, backup and recovery, rollback and incident ownership, then describe the release using that operating envelope rather than an unqualified production-ready label. ### When is a rewrite justified? A rewrite is justified when the core data model cannot express the product, safe coexistence is impractical or replacing the foundations costs less than repeatedly adapting them. Valuable product logic and user flows should be preserved unless evidence shows that their structure prevents safe operation. ## Final thoughts A stateful LLM platform becomes operable when every important state and authority has an owner. PostgreSQL owns durable runtime data. A central policy owns tenant and object access. The job layer owns expensive work. API and MCP routes reuse those decisions. Deployment health reflects real dependencies, and tests plus browser QA supply release evidence. Keep the product behavior that users already value. Replace the foundations that cannot support shared operation. Most importantly, name the operating envelope honestly. A controlled pilot with explicit trade-offs is a stronger engineering result than an undefined production claim. ## You may also like.. [**Fintech Architecture Remediation Without a Rewrite** A second anonymized field case on ordering architecture, authorization, persistence and release gates around a live money-moving product.](/blog/fintech-architecture-remediation-without-rewrite/) [**Software QA Checklist Before Launch** Turn production readiness into a practical release evidence checklist.](/software-development-guide/software-qa-checklist-before-launch/) Architecture and platforms ## Continue through this cluster Framework, platform and system-design choices that affect delivery over the long term. [Start with the cornerstone**Smart City Architecture Best Practices: MQTT, LoRaWAN, Kubernetes and Terraform**](/blog/smart-city-architecture-best-practices-2026/) - [AI in Grocery Retail: An MPREIS Opportunity Map](/blog/mpreis-ai-grocery-retail-opportunity-map/) - [Tirol Kliniken (TILAK): A Digital Operations Opportunity Map](/blog/tirol-kliniken-digital-opportunity-analysis/) - [TIWAG Digital Energy: A Software and AI Opportunity Map](/blog/tiwag-digital-energy-services-opportunity-map/) - [AI 3D Model Generators for Game Development: 2026 Guide](/blog/ai-3d-model-generators-game-development-2026/) - [Fintech Architecture Remediation Without a Rewrite](/blog/fintech-architecture-remediation-without-rewrite/) Inbox, without the noise ## Follow the work that matters to you Get a short email when we publish something new. Follow the whole blog or only the problems you care about. [**Back**](/blog/overview/) [![Christof Jori](/img/team/christof.webp)](/team/christof-jori/) [Christof Jori](/team/christof-jori/) https://linkedin.com/in/jocr77 11 min read · 14 Aug 2026 Last reviewed August 14, 2026 [**Next**](/blog/fintech-architecture-remediation-without-rewrite/) New posts by email × × Get new posts by email A short email when we publish. Free, no tracking. ## Structured Data ```json { "@context": "https://schema.org", "@graph": [ { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ], "employee": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "founder": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "legalRepresentative": [ { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } } ], "name": "Wavect GmbH", "subjectOf": { "@id": "https://wavect.io/verified-claims.json#dataset", "@type": "Dataset", "creator": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "description": "A machine-readable registry of quantitative and qualitative claims published by Wavect, with review dates, localized page appearances and public third-party citations where available.", "inLanguage": "en", "isAccessibleForFree": true, "license": "https://creativecommons.org/licenses/by/4.0/", "name": "Wavect verified publication claims", "url": "https://wavect.io/verified-claims.json" }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/team/kevin-riedl/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Kevin Riedl", "sameAs": [ "https://www.wikidata.org/wiki/Q139796365", "https://www.linkedin.com/in/wsdt", "https://github.com/wsdt" ], "url": "https://wavect.io/team/kevin-riedl/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "jobTitle": "Managing Director", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/", "worksFor": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] } }, { "@id": "https://wavect.io/#website", "@type": "WebSite", "inLanguage": [ "en", "de", "es", "zh" ], "name": "Wavect", "potentialAction": { "@type": "SearchAction", "query-input": "required name=search_term_string", "target": { "@type": "EntryPoint", "urlTemplate": "https://wavect.io/search/?q={search_term_string}" } }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/" }, { "@id": "https://wavect.io/blog/stateful-llm-platform-production-architecture/#webpage", "@type": "WebPage", "dateModified": "2026-08-14", "inLanguage": "en", "isPartOf": { "@id": "https://wavect.io/#website", "@type": "WebSite" }, "lastReviewed": "2026-08-14", "url": "https://wavect.io/blog/stateful-llm-platform-production-architecture/" } ] } ``` ```json { "@context": "https://schema.org", "@type": "BlogPosting", "abstract": "Productionising a stateful LLM platform is not mainly a model-serving task. The hard part is making state, identity, long-running work, external access, deployment and recovery explicit. In this anonymized field engagement, Wavect kept the working product logic and hardened the foundations in dependency order: define a controlled operating boundary, move authoritative runtime state to PostgreSQL, centralize tenant and object authorization, place expensive LLM work behind owned and idempotent jobs, secure API and MCP access separately, deploy from infrastructure as code with dependency-aware health checks and rollback, then use automated tests and repeated browser QA as one release system. The result was a credible controlled pilot, not a claim of high availability. That distinction matters: production readiness is a documented operating envelope with evidence and residual risks, not a green build or polished interface.", "articleBody": " Blog overview/Delivery and QA/Architecture and platforms How to Productionise a Stateful LLM Platform Without Rewriting the Product TL;DR Productionising a stateful LLM platform is not mainly a model-serving task. The hard part is making state, identity, long-running work, external access, deployment and recovery explicit. In this anonymized field engagement, Wavect kept the working product logic and hardened the foundations in dependency order: define a controlled operating boundary, move authoritative runtime state to PostgreSQL, centralize tenant and object authorization, place expensive LLM work behind owned and idempotent jobs, secure API and MCP access separately, deploy from infrastructure as code with dependency-aware health checks and rollback, then use automated tests and repeated browser QA as one release system. The result was a credible controlled pilot, not a claim of high availability. That distinction matters: production readiness is a documented operating envelope with evidence and residual risks, not a green build or polished interface. Stateful LLM platform production architecture is the set of boundaries that makes an AI application safe to operate repeatedly for several users. It covers authoritative data, identity, object access, expensive background work, external interfaces, deployment, recovery and verification. The model call is only one component inside that operating system. This article owns one specific question: how do you move a feature-rich, stateful LLM application from local research use into a controlled shared pilot? It does not decide whether you need a knowledge graph, which is covered by our graph engineering decision guide. It also does not repeat the generic prototype-to-production checklist. The focus here is the dependency order for a platform that already has valuable product logic. Anonymization boundary: This is a field case from Wavect delivery records, not a composite. We omit the client name, product category, geography, internal feature names, production hostnames, exact topology and unresolved security details. The technology families, productionisation sequence, failure patterns and trade-offs are real. No client outcome or industry benchmark is implied. What makes a stateful LLM application different from a demo? A demo can keep context in a browser session, reload fixtures from files and wait synchronously for a model response. A shared platform cannot. Users expect their data to survive a deployment, their permissions to apply through every interface and a long-running operation to remain understandable after a timeout, refresh or failure. The starting application in this engagement already had substantial logic for structured domain data, graph operations, document ingestion, grounded conversations, interviews, simulations and evaluation. Its gap was not a missing feature list. The gap was operational ownership across the paths that connect those features. Research-friendly assumptionShared-platform requirementFailure if left implicit Files are the live stateOne authoritative transactional storeDeployments or concurrent users produce divergent data. One trusted operatorServer-validated identity and object accessA valid user can reach another user's data. The request waits for the modelOwned, observable and recoverable jobsTimeouts look like failures and retries duplicate work. The browser is the only clientConsistent policy across browser, API and MCPA new interface bypasses assumptions enforced only in the UI. A running process is healthyDependency-aware readiness and rollbackTraffic reaches an application that cannot serve correct results. Manual testing proves the releaseAutomated checks plus representative flow QACross-layer regressions recur after every change. The seven-layer productionisation sequence The order matters. Polishing the interface before persistence and access control only makes unstable behavior easier to demonstrate. Adding an external API before central authorization creates a second security model. The programme therefore established each foundation before expanding the surface above it. 1. Define the operating envelope before choosing infrastructure The target was a controlled initial release, not a claim of globally redundant enterprise infrastructure. That decision fixed the immediate acceptance criteria: a repeatable HTTPS deployment, persistent data, controlled onboarding, recoverable releases, useful logs and an explicit list of remaining scale and availability work. This wording is more than risk management. It prevents architecture theatre. A single application task and a single-zone database may be reasonable for a design-partner pilot when their failure modes, backups and escalation paths are documented. They are not high availability. Naming the boundary lets the team buy the next reliability step when usage justifies it. 2. Move authoritative runtime state into PostgreSQL The application had portable seed files and", "articleSection": "Engineering", "author": { "@id": "https://wavect.io/team/christof-jori/#person", "@type": "Person", "name": "Christof Jori", "sameAs": [ "https://www.wikidata.org/wiki/Q139796367", "https://www.linkedin.com/in/jocr77/", "https://github.com/jo-chris" ], "url": "https://wavect.io/team/christof-jori/" }, "citation": [ { "@type": "WebPage", "name": "OWASP API1:2023 Broken Object Level Authorization", "url": "https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/" }, { "@type": "WebPage", "name": "Model Context Protocol authorization specification", "url": "https://modelcontextprotocol.io/specification/2026-07-28/basic/authorization" }, { "@type": "WebPage", "name": "AWS documents deployment circuit breakers and CloudWatch alarms", "url": "https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-failure-detection.html" }, { "@type": "WebPage", "name": "NIST Secure Software Development Framework", "url": "https://csrc.nist.gov/projects/ssdf" } ], "dateModified": "2026-08-14", "datePublished": "2026-08-14", "description": "Productionising a stateful LLM platform is not mainly a model-serving task. The hard part is making state, identity, long-running work, external access, deployment and recovery explicit. In this anonymized field engagement, Wavect kept the working product logic and hardened the foundations in dependency order: define a controlled operating boundary, move authoritative runtime state to PostgreSQL, centralize tenant and object authorization, place expensive LLM work behind owned and idempotent jobs, secure API and MCP access separately, deploy from infrastructure as code with dependency-aware health checks and rollback, then use automated tests and repeated browser QA as one release system. The result was a credible controlled pilot, not a claim of high availability. That distinction matters: production readiness is a documented operating envelope with evidence and residual risks, not a green build or polished interface.", "headline": "Stateful LLM Platform Production Architecture", "image": "https://wavect.io/img/blog/headers/header_stateful-llm-platform-production-architecture.svg", "inLanguage": "en", "keywords": "LLM Engineering, Software Architecture", "mainEntityOfPage": { "@id": "https://wavect.io/blog/stateful-llm-platform-production-architecture/", "@type": "WebPage" }, "publisher": { "@id": "https://wavect.io/#organization", "@type": [ "Organization", "ProfessionalService", "LocalBusiness" ] }, "url": "https://wavect.io/blog/stateful-llm-platform-production-architecture/", "wordCount": 2521 } ``` ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ { "@type": "ListItem", "item": "https://wavect.io/", "name": "Home", "position": 1 }, { "@type": "ListItem", "item": "https://wavect.io/blog/overview/", "name": "Blog overview", "position": 2 }, { "@type": "ListItem", "item": "https://wavect.io/blog/topics/delivery-qa/", "name": "Delivery and QA", "position": 3 }, { "@type": "ListItem", "item": "https://wavect.io/blog/clusters/architecture-platforms/", "name": "Architecture and platforms", "position": 4 }, { "@type": "ListItem", "item": "https://wavect.io/blog/stateful-llm-platform-production-architecture/", "name": "Stateful LLM Platform Production Architecture | ", "position": 5 } ] } ``` ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "It is an AI application whose useful behavior depends on durable domain data, user or tenant identity, permissions, job state, history or connected records across requests. The model is one component; the platform must also own persistence, access, execution and recovery." }, "name": "What is a stateful LLM platform?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Define the operating envelope first, then establish authoritative persistence and one authorization path. Background jobs, external APIs, deployment automation and wider interface QA depend on those foundations, so implementing them earlier creates duplicate state and policy models." }, "name": "What should be productionised first in an LLM application?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Usually not when it can exceed normal request time, be retried, cancelled or observed after a refresh. Put it behind an owned job with explicit states, idempotency, progress, cancellation, result storage and shutdown behavior. The first backend can be simple if the contract allows later replacement." }, "name": "Should long-running LLM work stay inside an HTTP request?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "Treat MCP as another external interface. Validate identifiers, authenticate the client, bind tokens to the intended resource, enforce object permissions and read or write scopes, rate-limit calls and sanitize errors. Do not rely on an agent or client interface to enforce server-side policy." }, "name": "How should an MCP server access platform data?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "No. A controlled pilot can be production-like and useful with a narrower availability target. Document task count, database topology, queue durability, backup and recovery, rollback and incident ownership, then describe the release using that operating envelope rather than an unqualified production-ready label." }, "name": "Does a successful pilot mean the platform is highly available?" }, { "@type": "Question", "acceptedAnswer": { "@type": "Answer", "text": "A rewrite is justified when the core data model cannot express the product, safe coexistence is impractical or replacing the foundations costs less than repeatedly adapting them. Valuable product logic and user flows should be preserved unless evidence shows that their structure prevents safe operation." }, "name": "When is a rewrite justified?" } ] } ```