Conversational AI · Stateful Backend · Memory Systems
Agentic Chatbot — a memory-aware conversational AI backend
This project explores the engineering required to turn a stateless LLM call into a persistent multi-user assistant: authentication, conversation state, three-layer memory, memory-conflict resolution, streaming responses, LLM quality evaluation, PostgreSQL persistence, rate limiting, and distributed tracing.
Primary challenge
Persistent context without blindly trusting memory updates
Core memory model
Long-term facts + summary + recent messages
Service topology
Auth gateway + chatbot API + 2 PostgreSQL databases + Jaeger
01 · Engineering problem
A useful assistant needs memory, but memory can also become wrong
A stateless chatbot can answer one request at a time, but it cannot preserve stable user context across sessions. Simply storing every extracted fact is also unsafe: people move, goals change, and an LLM can misinterpret a message. The project therefore treats memory as managed application state with confidence, compression, conflict detection, explicit confirmation, and persistence.
02 · System architecture
Split identity from conversation intelligence
The repository is not a LangGraph agent runtime. Its architecture is service-oriented: a dedicated authentication gateway establishes identity, while the chatbot service orchestrates LLM, memory, persistence, evaluation, streaming, and observability concerns.
Auth Gateway
A separate FastAPI service owns authentication and its own PostgreSQL database. The chatbot service verifies JWT identity instead of trusting user IDs supplied in chat payloads.
Chatbot API
FastAPI exposes synchronous and SSE streaming chat endpoints, conversation listing, health/readiness endpoints, rate limits, trace middleware, and input validation.
Conversation Services
MemoryService, LongTermMemoryService, PendingMemoryService, ConfirmationService, EvalService, and LLMService separate conversation behavior from API handlers.
Persistence
PostgreSQL stores users, conversations, messages, summaries, long-term memory, pending memory conflicts, and LLM evaluation results through SQLAlchemy and Alembic migrations.
LLM Boundary
The OpenAI-backed LLM service generates replies, streams tokens, summarizes conversations, extracts stable user facts, and classifies ambiguous memory confirmations.
Observability
Trace IDs flow through requests and responses, while OpenTelemetry exports traces to Jaeger across FastAPI, SQLAlchemy, and custom application spans.
Chatbot API → OpenTelemetry → Jaeger
03 · Chat orchestration
The request path is a stateful workflow
Authenticate
JWT verification resolves the user identity before the chat workflow begins.
Resolve conversation
The backend creates or reuses a conversation scoped to the authenticated user.
Check pending conflicts
Unresolved memory conflicts are handled before ordinary chat so contradictory facts cannot silently enter long-term memory.
Store user message
The incoming message becomes part of the persistent conversation history.
Extract user facts
The LLM extracts stable user facts and the evaluation service records the quality of that extraction.
Update or stage memory
Non-conflicting facts update memory; contradictory facts become pending conflicts requiring explicit confirmation.
Build context
Long-term facts, conversation summary, and recent history are assembled into the prompt context.
Generate response
The LLM returns either a complete reply or an SSE token stream.
Self-evaluate
The generated response is evaluated and stored with trace/user/conversation metadata.
Compress when needed
Long conversations are summarized and older messages are pruned while recent context is retained.
04 · Three-layer memory
Persistent identity without sending the entire history every time
Long-term memory
User-scoped persistent facts survive across conversations. Memories carry confidence and are injected into future context.
Conversation summary
When a message-count threshold is reached, older conversation content is compressed into a summary instead of continually expanding prompt size.
Recent messages
A configurable number of recent role/content messages remains verbatim to preserve immediate conversational context.
Long-term facts are additionally separated by confidence. High-confidence facts, likely facts, and uncertain information are injected with different system framing, while structured memory keys and dynamic free-form facts remain distinguishable.
05 · Memory conflict resolution
Contradictions become a user-visible state transition
If an incoming fact conflicts with stored long-term memory, the system does not silently replace the old value. It stages the update and asks the user to decide.
A newly extracted fact is compared with existing user-scoped memory.
If the value contradicts the stored value, the update is staged as a pending conflict rather than applied.
The assistant asks a short confirmation question and keeps the old value active meanwhile.
Common yes/no phrases use a regex fast path; ambiguous replies fall back to LLM classification.
Confirm applies the new memory; reject preserves the old memory.
Pending conflicts expire after a configurable TTL.
06 · Streaming responses
SSE with disconnect-aware persistence
The streaming endpoint emits token events and checks client connectivity while generation is in progress. The complete assistant response is persisted only after streaming completes, followed by evaluation and a transaction commit. Stream failures roll back database work and emit an explicit SSE error event.
07 · LLM self-evaluation
Model operations are evaluated as application events
EvalService evaluates generated replies, extracted memory facts, and confirmation classification. Results are persisted together with trace ID, user, and conversation context. This creates an inspection path for LLM behavior rather than treating each call as an opaque API result.
08 · Reliability & operations
SSE streaming
POST /chat/stream streams token events and checks whether the client disconnected before persisting a completed response.
Rate limiting
Chat endpoints are limited per IP with SlowAPI rather than allowing unbounded request bursts.
Trace propagation
X-Trace-ID is propagated through request/response handling and included in logs and evaluation records.
Transactions
Chat operations commit on success and roll back on exceptions or stream failures.
Health/readiness
Dedicated health endpoints support deployment-oriented liveness/readiness checks.
Migrations
Alembic manages schema evolution instead of relying on ad-hoc database initialization.
09 · Deployment topology
Docker Compose keeps service ownership explicit
The Compose configuration runs four application/infrastructure concerns: the authentication gateway, its PostgreSQL database, the chatbot API, its separate PostgreSQL database, plus Jaeger for traces. Health checks gate database-dependent startup, and Alembic migrations run before each API service starts.
10 · Engineering decisions
Scope all conversation and memory state to authenticated user identity rather than accepting user_id from request bodies.
Use three memory layers to balance persistence, immediate context, and token efficiency.
Stage contradictory long-term memories and require confirmation instead of silently overwriting them.
Use a deterministic regex fast path for common confirmation phrases before spending an LLM call on classification.
Evaluate multiple LLM operations and persist the results instead of evaluating only final assistant replies.
Keep synchronous and streaming chat paths consistent around memory extraction, conflict handling, evaluation, and persistence.
Use trace IDs, rate limits, transactions, health probes, migrations, and distributed tracing as first-class backend concerns.
Engineering takeaway
The strongest lesson from this project is that conversational memory is an application state problem, not just a prompt-engineering problem. Once memory can persist across sessions, the system also needs ownership, confidence, conflict handling, compression, transactional updates, user confirmation, evaluation, and observability.