Building a Session Aware Local LLM Chat Service with FastAPI, Ollama and Docker

This document describes the design and implementation of a local large language model chat service built on top of Ollama, FastAPI and a minimal browser client. The service maintains per session conversational state, exposes a single HTTP endpoint for interaction and can be deployed either as a native Python process or as a Docker container.

System architecture

The system consists of three primary components and one optional orchestration layer:

A browser based client implemented as a static HTML and JavaScript application.
A FastAPI backend that terminates HTTP requests, manages session level state and invokes Ollama.
An Ollama runtime that hosts one or more local language models and exposes a JSON HTTP API.
An optional Docker or docker compose configuration that encapsulates the runtime environment.

The data path for a single user request is as follows. The browser issues an HTTP POST request containing a single text message in JSON form. The FastAPI backend accepts the request, resolves a logical session identifier from cookies, reconstructs the message history associated with that session and optionally answers simple memory queries from that history. If no rule based answer is available, the backend forwards the full message history to Ollama, which returns a model generated reply. The backend appends that reply to the session history and returns it as JSON. The browser then renders the reply as a chat bubble.

The project directory is organised in the following way:

ollama-chatbot-v3/
  main.py              # FastAPI backend and session logic
  Dockerfile           # Container build definition
  requirements.txt     # Python dependencies
  static/
    index.html         # Chat user interface
    script.js          # Front end logic
    style.css          # Styling for the chat panel

Backend configuration and environment variables

The backend reads configuration parameters from environment variables so that model selection and runtime connectivity can be changed without modifying the code. Near the top of main.py the following definitions appear:

OLLAMA_HOST: str = os.getenv("OLLAMA_HOST", "http://host.docker.internal:11434")
OLLAMA_MODEL: str = os.getenv("OLLAMA_MODEL", "hola")
SYSTEM_PROMPT: Optional[str] = os.getenv(
    "SYSTEM_PROMPT", "You are a helpful assistant."
)

These variables control:

OLLAMA_HOST - the base URL of the Ollama HTTP server, for example http://localhost:11434 in a non Docker environment.
OLLAMA_MODEL - the identifier of the model previously pulled into Ollama, such as llama3 or another local model name.
SYSTEM_PROMPT - an optional system level instruction inserted once at the beginning of each new conversation that establishes assistant behaviour.

Using environment variables in this way decouples deployment time configuration from the application code and simplifies both local and container based setups.

FastAPI application and message representation

The backend is implemented as a single FastAPI application with a minimal message abstraction and an in memory store for conversational state:

app = FastAPI()


class Message(Dict[str, str]):
    """Alias for a chat message object."""


conversation_store: Dict[str, List[Message]] = {}

The FastAPI instance represents the web application. Routes are attached to this object and are later served via Uvicorn. The Message alias is a typed dictionary used to represent chat messages as small JSON compatible objects with keys such as role and content. A typical list of messages contains entries of the form:

[
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Hello"},
  {"role": "assistant", "content": "Hello, how may I assist you?"}
]

The conversation_store dictionary is the central memory structure. Its keys are opaque session identifiers and the values are ordered lists of messages associated with each session. This structure is held purely in process memory, which keeps the implementation simple but implies that all stored conversations are lost when the process terminates.

Rule based memory helper

In addition to forwarding history to the language model, the backend implements a small rule based memory helper that can answer certain queries directly from conversation history. This avoids unnecessary model calls for trivial memory requests and also demonstrates how to extend the system with deterministic logic.

def get_memory_answer(user_message: str, history: List[Message]) -> Optional[str]:
    if not history:
        return None

    lower = user_message.lower()

    if any(phrase in lower for phrase in ["last message", "previous message"]):
        for msg in reversed(history):
            if msg.get("role") == "user":
                return f"Your last message was: '{msg.get('content')}'."
        return "I have not received any messages from you yet."

    if "shirt" in lower and "color" in lower:
        for msg in reversed(history):
            if msg.get("role") == "user" and "shirt" in msg.get("content", "").lower():
                return f"You said: {msg.get('content')}"

    return None

The behaviour can be summarised as follows:

If no history exists for the session, the function returns None immediately.
The current user message is normalised to lower case and inspected for trigger phrases.
For queries involving phrases such as “last message” or “previous message”, the function walks the history in reverse order and selects the most recent user message, which is then quoted back to the user.
For queries that mention both “shirt” and “color”, the function scans for the latest user message mentioning a shirt and echoes that content.
If no trigger condition is met, None is returned and control flows to the model path.

This helper illustrates the extension point for additional deterministic memory behaviours. More complex rules or even vector search over past messages can be incorporated here without altering the basic HTTP interface.

Static file serving and main HTML entry point

The backend serves the chat user interface from the static directory. The root route returns the main HTML file:

@app.get("/", response_class=HTMLResponse)
async def index() -> HTMLResponse:
    index_path = os.path.join(os.path.dirname(__file__), "static", "index.html")
    with open(index_path, "r", encoding="utf-8") as f:
        html_content = f.read()
    return HTMLResponse(content=html_content, status_code=200)

The function determines the path to index.html relative to main.py, reads the file into memory and returns it as an HTMLResponse. Static assets such as JavaScript and CSS are exposed through a mounted directory:

static_dir = os.path.join(os.path.dirname(__file__), "static")
app.mount("/static", StaticFiles(directory=static_dir), name="static")

Any requests whose path begins with /static are served from the static folder. This arrangement isolates presentation files from the backend logic while keeping the deployment model simple.

Ollama HTTP integration

Communication with Ollama is encapsulated in the call_ollama_chat function. This function accepts the complete message history for a session and performs a single JSON HTTP round trip to the Ollama server.

async def call_ollama_chat(messages: List[Message]) -> str:
    url = f"{OLLAMA_HOST}/api/chat"
    payload = {
        "model": OLLAMA_MODEL,
        "messages": messages,
        "stream": False,
    }
    async with httpx.AsyncClient(timeout=None) as client:
        try:
            response = await client.post(url, json=payload)
            response.raise_for_status()
        except httpx.HTTPError as exc:
            raise HTTPException(status_code=502, detail=f"Error calling Ollama API: {exc}")
    data = response.json()
    assistant_reply = data.get("message", {}).get("content")
    if assistant_reply is None:
        raise HTTPException(status_code=500, detail="No content returned from Ollama API")
    return assistant_reply

Important details:

The URL is constructed by combining OLLAMA_HOST with the /api/chat path exposed by Ollama.
The payload includes the selected model name, the full message history and a stream flag to request a non streaming response.
The httpx.AsyncClient is used to send an asynchronous POST request with the payload encoded as JSON.
Any HTTP level failure triggers an HTTP 502 response back to the client via a HTTPException.
The function extracts message.content from the Ollama response and returns that string to the caller.

The rest of the backend remains unaware of the details of the Ollama HTTP interface and simply consumes this function as an asynchronous text to text service.

Chat endpoint request lifecycle

The core of the backend is the /chat endpoint. It is responsible for parsing client input, assigning or reusing a session identifier, managing per session history, invoking the memory helper and calling Ollama when required.

@app.post("/chat")
async def chat_endpoint(
    request: Request,
    response: Response,
    session_id: Optional[str] = Cookie(default=None),
) -> JSONResponse:

Input validation

The endpoint begins by parsing the JSON request body and extracting the user message:

    try:
        body = await request.json()
    except Exception:
        raise HTTPException(status_code=400, detail="Invalid JSON payload")
    user_message = body.get("message")
    if not user_message:
        raise HTTPException(status_code=400, detail="'message' field is required")

The expected payload is a JSON object with a single message field. If parsing fails or the field is missing or empty, the endpoint returns an HTTP 400 error indicating invalid input.

Session identifier handling

Session affinity is implemented with a cookie named session_id. The endpoint accepts this cookie through the session_id parameter. If the cookie is missing, a new identifier is allocated:

    is_new_session = False
    if not session_id:
        session_id = str(uuid.uuid4())
        is_new_session = True

    if session_id not in conversation_store:
        conversation_store[session_id] = []
        if SYSTEM_PROMPT:
            conversation_store[session_id].append(
                {"role": "system", "content": SYSTEM_PROMPT}
            )
    history = conversation_store[session_id]

The logic is:

If no session_id cookie is present, generate a new UUID and mark this as a new session.
If the identifier does not yet exist in conversation_store, initialise an empty message list and optionally insert a system level message containing the configured system prompt.
Bind history to the list stored under this session identifier.

This mechanism isolates message histories across different browsers or devices and supports multiple concurrent users sharing the same backend.

Rule based memory path

Before invoking the model, the endpoint checks whether the new message can be serviced by the rule based helper:

    memory_answer = get_memory_answer(user_message, history)
    if memory_answer is not None:
        history.append({"role": "user", "content": user_message})
        history.append({"role": "assistant", "content": memory_answer})
        if len(history) > 40:
            system_msg = history[0] if history[0].get("role") == "system" else None
            trimmed = history[-40:]
            if system_msg:
                trimmed.insert(0, system_msg)
            history[:] = trimmed
        result = JSONResponse(content={"reply": memory_answer})
        if is_new_session:
            result.set_cookie(key="session_id", value=session_id, httponly=True)
        return result

If the helper returns a non null string, the endpoint:

Appends the user message and the generated answer to the current history.
Applies a trimming policy that keeps at most forty messages per session while preserving a leading system message if present.
Constructs a JSON response where the reply field contains the memory based answer.
Sets the session_id cookie for new sessions with the httponly flag enabled.

No call to Ollama is performed in this branch.

Model invocation path

If the rule based helper returns None, the endpoint falls back to the model based path:

    history.append({"role": "user", "content": user_message})

    max_history = 40
    if len(history) > max_history:
        system_msg = history[0] if history[0].get("role") == "system" else None
        trimmed = history[-max_history:]
        if system_msg:
            trimmed.insert(0, system_msg)
        history[:] = trimmed

    assistant_reply = await call_ollama_chat(history)

    history.append({"role": "assistant", "content": assistant_reply})

    result = JSONResponse(content={"reply": assistant_reply})
    if is_new_session:
        result.set_cookie(key="session_id", value=session_id, httponly=True)
    return result

The sequence of operations is:

Append the incoming user message to the history for this session.
Apply the same trimming policy as in the memory branch, retaining only recent messages plus a potential system prompt.
Invoke call_ollama_chat with the current history and await the assistant reply from Ollama.
Append the assistant reply to the history.
Return a JSON response with the reply text and set the session cookie if necessary.

The trimming operation constrains prompt size and ensures that very long conversations do not grow without bound, which is important when working with models that have limited context windows.

Browser client implementation

The browser side of the application is implemented using plain HTML, CSS and JavaScript. The goal is to provide a functional chat interface with minimal dependencies while exercising the backend endpoint described above.

HTML structure

The index.html file defines three main regions: a header, a message viewport and an input form. The essential markup is similar to the following:

<div id="chat-app">
  <header>Local Ollama Chatbot</header>
  <div id="messages" class="messages"></div>
  <form id="messageForm" class="input-area">
    <input
      type="text"
      id="messageInput"
      class="message-input"
      placeholder="Type your message..."
      autocomplete="off"
    />
    <button type="submit" id="sendButton" class="send-button">
      Send
    </button>
  </form>
</div>

<script src="/static/script.js"></script>

The element with id messages acts as the scrollable container for chat bubbles. The form with id messageForm delivers user input to the JavaScript logic.

Styling considerations

The style.css file configures a centered chat layout, scroll behaviour for the message viewport and distinct visual styles for user and assistant messages by assigning classes such as message user and message assistant. Styling does not interact with the backend logic and can be adjusted independently.

JavaScript behaviour

The script.js file is responsible for wiring the HTML elements to the backend endpoint. The main operations are setting up event handlers, constructing and sending HTTP requests and updating the DOM with responses.

document.addEventListener('DOMContentLoaded', () => {
  const form = document.getElementById('messageForm');
  const input = document.getElementById('messageInput');
  const messagesDiv = document.getElementById('messages');

  function appendMessage(content, sender) {
    const msg = document.createElement('div');
    msg.classList.add('message', sender);
    msg.innerText = content;
    messagesDiv.appendChild(msg);
    messagesDiv.scrollTop = messagesDiv.scrollHeight;
  }

  async function sendMessage(event) {
    event.preventDefault();
    const text = input.value.trim();
    if (!text) return;

    appendMessage(text, 'user');
    input.value = '';

    try {
      const res = await fetch('/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        credentials: 'include',
        body: JSON.stringify({ message: text }),
      });
      if (!res.ok) {
        let detail;
        try {
          const err = await res.json();
          detail = err.detail || res.statusText;
        } catch {
          detail = res.statusText;
        }
        appendMessage(`Error: ${detail}`, 'assistant');
        return;
      }
      const data = await res.json();
      appendMessage(data.reply, 'assistant');
    } catch (err) {
      appendMessage(`Error: ${err}`, 'assistant');
    }
  }

  form.addEventListener('submit', sendMessage);
});

Behaviour summary:

On page load, the script resolves references to the form, input control and message container.
appendMessage builds a new message element with appropriate classes and ensures the viewport scrolls to the most recent message.
sendMessage intercepts form submission, reads and clears the input, renders the user message locally, then issues a POST request to /chat with JSON encoded payload.
The credentials option is set to include so that cookies, including session_id, are transmitted to the backend.
Responses are validated, errors are surfaced as assistant messages and successful replies are appended to the message container.

Local execution without Docker

To run the service directly on a development machine without containers, the following sequence can be used:

Extract the project archive and change into the directory.
```
unzip ollama-chatbot-v3.zip
cd ollama-chatbot-v3
```

Create and activate a Python virtual environment.

python -m venv .venv
# Linux or macOS
source .venv/bin/activate
# Windows
.venv\Scripts\activate

Install the Python dependencies.

pip install --upgrade pip
pip install -r requirements.txt

Ensure that Ollama is installed, running and that the desired model has been pulled, for example ollama pull llama3.

Configure the environment and start Uvicorn.

export OLLAMA_HOST="http://localhost:11434"
export OLLAMA_MODEL="llama3"
export SYSTEM_PROMPT="You are a helpful local assistant."

uvicorn main:app --host 0.0.0.0 --port 8000

On Windows, set can be used instead of export to assign environment variables.

Navigate to http://localhost:8000 in a browser and interact with the chat interface.

Docker based deployment

The Dockerfile in the project root defines an image that encapsulates the FastAPI application and its runtime dependencies. The file is concise:

FROM python:3.11-slim

WORKDIR /app

COPY . /app

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

The directives can be interpreted as follows:

FROM python:3.11-slim selects an official base image with Python 3.11 on a minimal Linux distribution.
WORKDIR /app sets the working directory inside the container to /app.
COPY . /app copies all project files into the container image.
RUN pip install ... installs the Python dependencies specified in requirements.txt.
EXPOSE 8000 documents that the application listens on port 8000 inside the container.
CMD [...] defines the default command that launches the FastAPI app using Uvicorn.

Building and running the image can be done with:

docker build -t local-ollama-chatbot .

docker run --rm -p 8000:8000 \
  -e OLLAMA_HOST="http://host.docker.internal:11434" \
  -e OLLAMA_MODEL="llama3" \
  -e SYSTEM_PROMPT="You are a helpful local assistant." \
  local-ollama-chatbot

The port mapping publishes the container's port 8000 on the host. The environment variables configure the backend to reach Ollama running on the host system. On Linux, a different OLLAMA_HOST may be required, depending on the Docker networking configuration.

Optional docker compose topology

A docker compose configuration can be introduced to manage both the Ollama runtime and the chat service as a single logical stack. The following file can be added as docker-compose.yml in the project root:

version: "3.9"

services:
  ollama:
    image: <official_ollama_image>
    container_name: ollama-server
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama

  chatbot:
    build: .
    container_name: ollama-chatbot
    environment:
      - OLLAMA_HOST=http://ollama:11434
      - OLLAMA_MODEL=llama3
      - SYSTEM_PROMPT=You are a helpful local assistant.
    ports:
      - "8000:8000"
    depends_on:
      - ollama

volumes:
  ollama-data:

Key points:

The ollama service runs the Ollama server, maps its port to the host and attaches a named volume ollama-data to persist model files.
The chatbot service builds the image from the local Dockerfile, exposes port 8000 and configures OLLAMA_HOST to point to the ollama service name inside the compose network.
The depends_on directive ensures that the Ollama service is started before the chatbot service.
The volumes block declares the named volume used to persist Ollama data across container restarts.

With this file present, the following command builds and starts the entire stack:

docker compose up --build

Once all services are running, the chat interface is reachable at http://localhost:8000, and Ollama is reachable at http://localhost:11434.

Conversation memory characteristics

The memory model employed by this service is session oriented and in memory. Its properties are as follows:

Each browser client is associated with a session_id stored in an HTTP only cookie.
The backend maintains a mapping from session_id to an ordered list of message objects.
On every request, the backend either produces a response via the rule based memory helper or delegates to the language model with the full current history.
The history is truncated to forty messages plus a possible system message to control prompt length.
All state is stored in process memory so a process restart resets all conversations.

For local experimentation and single user operation this memory strategy is straightforward and effective. When longer retention, multi process scaling or persistence across restarts is required, the dictionary based store can be replaced by a database keyed on session_id with minimal changes to the surrounding logic.

Search This Blog

Sri Surya S. Vaddhiparthy