Streaming LLM Responses End-to-End: A Spring Boot SSE Backend and a Next.js Frontend

In this tutorial we’ll build a working ChatGPT-style chat app that streams an LLM’s reply token-by-token, end to end, using Spring Boot 4.1, Spring AI, Next.js 16, React 19 and an embedded H2 database. The model comes from OpenAI via Spring AI; the tokens travel over Server-Sent Events; and the part most tutorials skip is the actual spine of the piece: surviving a dropped connection, a clicked Stop button, a buffering proxy and a slow client.

You should be comfortable with Spring Boot and a bit of React. You don’t need any prior Spring AI or SSE experience; we’ll build both from the ground up. You’ll need a JDK (the repo targets Java 25), Node 20+, and an OpenAI API key exported as OPENAI_API_KEY. If you don’t have a key handy you can still read along; the app loads fine without one, and calls just fail until you set it.

All the code is on GitHub: github.com/tucanoo/streaming-llm-spring-next. Clone it, run the backend, run the frontend, and you’ve got the whole thing in front of you. I’d suggest having it open while you read. I’ll introduce each file as a complete unit and explain the why, then leave the line-by-line detail to the commented versions in the repo.

One note before we start. If you’re also interested in Virtual Threads, then this pairs naturally with our virtual threads in Spring Boot 4 tutorial, because every open SSE stream holds a server thread for its whole life. We’ll come back to that.

Why streaming feels faster

A large model can take several seconds to produce a full answer. If you wait for the whole response and then drop it into the page, the user stares at a spinner for those seconds and then gets a wall of text all at once. Stream the same answer token-by-token and the first words appear in a few hundred milliseconds, then keep flowing. The total time is identical. The felt time is completely different, and that gap is the entire reason this UX won.

There’s a real-world wrinkle worth naming up front, because it shaped the design. The default model in the repo is gpt-5-mini, which is a reasoning model. It thinks before it answers, so there’s often a several-second pause before the very first token arrives. During that pause an SSE connection looks, to anything in the middle, exactly like a dead one. We’ll come back to that when we talk about heartbeats, but keep it in mind: “slow to start” and “broken” need to be told apart, and that’s a resilience concern, not a happy-path one.

So the goal isn’t just “stream some tokens to a div”. It’s to stream them and have the stream hold up when the network, the user, or the infrastructure does something inconvenient. That’s the part worth writing about.

Picking the transport: SSE vs WebSockets

You may be wondering why I’m not using WebSockets, since a chat is two-way. But it’s only two-way in the turn-taking sense: you send a prompt, the bot streams a reply back. Within that turn the client sends one thing and then just receives; it isn’t pushing and receiving at the same time. So a turn is really an ordinary request with a streamed response, which is what Server-Sent Events are for. We POST the prompt up over plain HTTP and stream the tokens back down over SSE, with no persistent socket and no upgrade handshake.

SSE has its limits, and it’s only fair to list them: it’s one-way (server to client), text only, and the browser’s built-in EventSource can’t set headers or send a POST body. On HTTP/1.1 you also get only about six connections per origin, so open streams add up. None of that hurts us here. We read the stream ourselves with fetch rather than using EventSource, which sidesteps that limitation and is what we’d want anyway, and in return SSE gives us plain HTTP and the Last-Event-ID reconnection support we rely on later for resume.

I’d only reach for WebSockets when the connection is genuinely two-way and continuous: live collaboration, multiplayer, voice, or anything where the server pushes mid-turn. For streaming one reply at a time, SSE is the simpler fit.

What an SSE stream actually looks like

Before we write any Spring code, it helps to see the wire format, because everything on both sides is just producing or parsing this text. An SSE response is text/event-stream, and it’s a sequence of frames separated by a blank line. Each frame is a few field: value lines. Here’s roughly what one of our streamed replies looks like on the wire:

event: meta
data: {"conversationId":1000,"messageId":1001}

id: 1001:10
event: token
data: Tremendous

id: 1001:19
event: token
data:  question

: keep-alive

id: 1001:412
event: done
data: [DONE]

The fields we care about are event (a name we choose, like token or done), data (the payload), and id (a resumption cursor). The line starting with a colon and nothing before it (: keep-alive) is a comment, which is our heartbeat, and clients ignore it. The id is the important one for resilience: ours is of the form <messageId>:<charOffset>, the database id of the assistant message followed by how many characters we’ve streamed so far. The browser automatically remembers the last id it saw and sends it back as Last-Event-ID if it reconnects, which is precisely how we’ll resume without restarting.

There’s a subtlety in that second token frame, data: question, that cost me an afternoon — I’ll explain it properly when we build the client parser. For now, notice the leading space before the word. It matters more than it looks.

The backend, happy path

Now we can build the Spring side. The whole thing was scaffolded from the Spring Initializr with Spring Boot 4.1.0, Java 25 and Gradle, and here’s the first place an older tutorial will trip you up.

Spring Boot 4.x renamed the starters. The plain spring-boot-starter-web of Boot 3.x is gone. The web starter is now spring-boot-starter-webmvc, and the H2 console ships as its own spring-boot-h2console module rather than being folded into the H2 dependency.

If you’re following a 3.x-era guide and your build can’t resolve spring-boot-starter-web, that’s why. Here’s the complete build.gradle:

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.0'
    id 'io.spring.dependency-management' version '1.1.7'
}

group = 'com.tucanoo'
version = '0.0.1-SNAPSHOT'
description = 'StreamingLLMDemo'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

ext {
    set('springAiVersion', "2.0.0")
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-h2console'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
    implementation 'org.springframework.boot:spring-boot-starter-validation'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'org.springframework.ai:spring-ai-starter-model-openai'
    compileOnly 'org.projectlombok:lombok'
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    runtimeOnly 'com.h2database:h2'
    annotationProcessor 'org.projectlombok:lombok'
    testImplementation 'org.springframework.boot:spring-boot-starter-actuator-test'
    testImplementation 'org.springframework.boot:spring-boot-starter-data-jpa-test'
    testImplementation 'org.springframework.boot:spring-boot-starter-validation-test'
    testImplementation 'org.springframework.boot:spring-boot-starter-webmvc-test'
    testCompileOnly 'org.projectlombok:lombok'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
    testAnnotationProcessor 'org.projectlombok:lombok'
}

dependencyManagement {
    imports {
        mavenBom "org.springframework.ai:spring-ai-bom:${springAiVersion}"
    }
}

tasks.withType(JavaCompile).configureEach {
    options.encoding = 'UTF-8'
}

tasks.named('test') {
    useJUnitPlatform()
}

A couple of things to flag here. The test starters follow the same 4.x split, so you’ll see spring-boot-starter-webmvc-test and friends rather than the single spring-boot-starter-test. Spring AI is pinned through its BOM at 2.0.0. And note the Lombok dependency: I’d check it compiles cleanly on your JDK before going further, because Lombok has historically lagged new Java releases by a few weeks, and Java 25 is recent enough that a stale Lombok will fail annotation processing in ways that look baffling. On the version in the repo it’s fine, but if your @Data and @Builder annotations suddenly stop generating, bump Lombok first before suspecting anything cleverer.

Wiring the chat client

Spring AI’s ChatClient is the abstraction over the model. We configure exactly one, with the demo persona baked in as a default system prompt. Under com.tucanoo.streamingllmdemo.config, here’s ChatClientConfig:

package com.tucanoo.streamingllmdemo.config;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.SimpleLoggerAdvisor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ChatClientConfig {

    static final String PERSONA_SYSTEM_PROMPT = """
            You are The Big Beautiful Bot, the self-proclaimed greatest chat assistant ever built,
            tagline "Making Chat Great Again". Answer every question genuinely, accurately and
            usefully — but in a wildly over-confident, superlative-laden showman's style.
            Congratulate the user on a "tremendous" question and promise "the best answer you've
            ever seen". Sprinkle in "tremendous", "the best", "believe me, nobody answers questions
            better".

            Hard rules, never broken:
            - Never mention real politics, political parties, policy, elections, or any real people.
              Your ego is ONLY about being a fantastic chatbot, app and answer.
            - Stay fun, harmless and strictly family-friendly.
            - Despite the bluster, the actual information must be correct. If you don't know, say so
              (in the most magnificent way possible) rather than inventing facts.
            - Keep answers reasonably concise — big on confidence, not padding.
            """;

    @Bean
    public ChatClient chatClient(ChatClient.Builder builder) {
        return builder
                .defaultSystem(PERSONA_SYSTEM_PROMPT)
                .defaultAdvisors(new SimpleLoggerAdvisor())
                .build();
    }
}

The persona is The Big Beautiful Bot, tagline “Making Chat Great Again”, a deliberate parody archetype of a bombastic showman: every answer is “tremendous”, “the best”, “believe me, nobody answers questions better”. I had fun with it, and the over-confident streamed text makes a good demo GIF. The one hard rule, encoded in the prompt, is that it never touches real politics, parties or real people. The ego is only ever about how great the bot is, and underneath the bluster the facts stay correct and family-friendly. Keeping the joke apolitical is what lets us ship it.

The SimpleLoggerAdvisor logs the prompt and reply (it handles the streaming case too), which is handy while developing. One thing I deliberately did not use is Spring AI’s MessageChatMemoryAdvisor for history. It saves on completion, which means it can’t hand me the assistant row’s id before streaming starts, and it can’t capture the partial text if a stream is cancelled. Both of those are exactly what our resume feature depends on. So we manage the message rows ourselves, which you’ll see in the service.

The streaming controller

The endpoint itself is small, because all the interesting work lives in the service. We’re on Spring MVC, not WebFlux. MVC happily adapts a returned Flux onto its async machinery and writes it out as text/event-stream. Under com.tucanoo.streamingllmdemo.controller, here’s ChatStreamController:

package com.tucanoo.streamingllmdemo.controller;

import com.tucanoo.streamingllmdemo.dto.ChatRequest;
import com.tucanoo.streamingllmdemo.service.ChatService;
import jakarta.servlet.http.HttpServletResponse;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import org.springframework.http.MediaType;
import org.springframework.http.codec.ServerSentEvent;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import reactor.core.publisher.Flux;

@RestController
@RequestMapping("/api/chat")
@RequiredArgsConstructor
public class ChatStreamController {

    private final ChatService chatService;

    @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<ServerSentEvent<String>> stream(
            @Valid @RequestBody ChatRequest request,
            @RequestHeader(value = "Last-Event-ID", required = false) String lastEventId,
            HttpServletResponse response) {

        response.setHeader("X-Accel-Buffering", "no");
        response.setHeader("Cache-Control", "no-cache");

        return chatService.streamReply(request, lastEventId);
    }
}

It’s a POST so the prompt rides in the body, which it has to, since SSE can’t send a request body itself. We accept the standard Last-Event-ID header for resume, and we set two response headers (X-Accel-Buffering: no and Cache-Control: no-cache) to stop intermediaries hoarding the stream. We’ll come back to those headers in the resilience section; they’re cheap insurance.

One configuration detail makes MVC streaming work at all. The OpenAI starter drags the full WebFlux stack onto the classpath because it uses WebClient internally, and left alone the app would try to start as a reactive WebFlux server. We pin it back to the servlet stack in application.yaml with spring.main.web-application-type: servlet, and that same file is where virtual threads get switched on. More on that shortly.

The data model on H2

History genuinely comes from a database here, not a Map: an embedded in-memory H2 instance, so there’s no external infra to stand up but the read and write path is real. Two entities, both under com.tucanoo.streamingllmdemo.entity. First, Conversation:

package com.tucanoo.streamingllmdemo.entity;

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import java.time.Instant;

@Entity
@Table(name = "conversation")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class Conversation {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(nullable = false)
    private String title;

    @Column(name = "demo_user", nullable = false)
    private String demoUser;

    @Column(name = "created_at", nullable = false)
    private Instant createdAt;
}

And Message, which is where the resume design starts to show through:

package com.tucanoo.streamingllmdemo.entity;

import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.EnumType;
import jakarta.persistence.Enumerated;
import jakarta.persistence.FetchType;
import jakarta.persistence.GeneratedValue;
import jakarta.persistence.GenerationType;
import jakarta.persistence.Id;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.Lob;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import java.time.Instant;

@Entity
@Table(name = "message")
@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class Message {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @ManyToOne(fetch = FetchType.LAZY, optional = false)
    @JoinColumn(name = "conversation_id", nullable = false)
    private Conversation conversation;

    @Enumerated(EnumType.STRING)
    @Column(nullable = false, length = 16)
    private Role role;

    @Lob
    @Column(nullable = false)
    @Builder.Default
    private String content = "";

    @Column(nullable = false)
    @Builder.Default
    private boolean partial = false;

    @Column(name = "created_at", nullable = false)
    private Instant createdAt;
}

The two fields that earn their keep are content, which grows as tokens stream in, and partial, which is true while a message is streaming and only cleared on a clean finish. A message left partial = true was cut off, and that flag is how we know, on reload, whether a reply finished or got interrupted. The Role enum is just USER and ASSISTANT; the system prompt lives in configuration, not the database. The repositories are the usual Spring Data interfaces: ConversationRepository with a findAllByOrderByCreatedAtDesc() for the sidebar, and MessageRepository with a findByConversationIdOrderByIdAsc(Long) to read history in turn order. You’ll find both in the repo.

Seeding the database

Our other CRM focused tutorials seed their tables via import.sql, and we do exactly the same here: Hibernate runs src/main/resources/import.sql automatically every time the schema is created. Because we set ddl-auto: create-drop, that’s on every startup, so the demo always boots with a populated sidebar. The seed conversations are owned by computing pioneers (Ada L., Grace H., Linus T.), which felt right for a dev audience, and the topics are deliberately benign so the bot’s voice is the only gag. Here’s import.sql:

-- Seed conversations for the Streaming LLM demo. Hibernate runs this automatically on startup
-- whenever the schema is (re)created (ddl-auto=create-drop), the same trick the CRM tutorials use.
INSERT INTO conversation (id, title, demo_user, created_at) VALUES
    (1, 'Explain Server-Sent Events like I''m five', 'Ada L.', CURRENT_TIMESTAMP),
    (2, 'Write a haiku about virtual threads',        'Grace H.', CURRENT_TIMESTAMP),
    (3, 'What can I cook with chicken and rice?',      'Linus T.', CURRENT_TIMESTAMP),
    (4, 'Three startup name ideas',                    'Ada L.', CURRENT_TIMESTAMP);

INSERT INTO message (id, conversation_id, role, content, partial, created_at) VALUES
    (1, 1, 'USER', 'Explain Server-Sent Events like I''m five.', FALSE, CURRENT_TIMESTAMP),
    (2, 1, 'ASSISTANT',
     'Tremendous question — honestly one of the best SSE questions I have ever seen, believe me. Picture a magic mailbox: you ask me once, and instead of making you check it again and again, I keep popping letters in one word at a time, as fast as I think them. That open, one-way mailbox is a Server-Sent Event. No fancy handshakes, just plain HTTP doing something beautiful. Nobody streams text better. Nobody!',
     FALSE, CURRENT_TIMESTAMP),
    (3, 2, 'USER', 'Write a haiku about virtual threads.', FALSE, CURRENT_TIMESTAMP),
    (4, 2, 'ASSISTANT',
     'Folks, this is the greatest little haiku about virtual threads ever generated — tremendous:

Threads light as a breeze
Thousands bloom where hundreds stalled
Tomcat breathes again

Believe me, no chatbot does poetry like this one.',
     FALSE, CURRENT_TIMESTAMP);
-- (further seed rows in the repo)

ALTER TABLE conversation ALTER COLUMN id RESTART WITH 1000;
ALTER TABLE message ALTER COLUMN id RESTART WITH 1000;

Two gotchas bit me here and are worth your time. The haiku has real line breaks inside the string literal, and Hibernate’s default import.sql extractor treats one physical line as one statement, so it choked. The fix is to switch to the multi-line extractor in application.yaml (hbm2ddl.import_files_sql_extractor set to MultiLineSqlScriptExtractor), which terminates on the semicolon and tolerates newlines inside strings. Separately, the em-dashes in the seed text came out as mojibake on Windows until I set the import charset to UTF-8 explicitly, because the default is the platform charset. The ALTER TABLE ... RESTART WITH 1000 at the end hands the identity counters back to the app well clear of the seeded ids, so new conversations don’t collide.

The application configuration

Here’s the relevant slice of src/main/resources/application.yaml, which ties the above together:

spring:
  main:
    web-application-type: servlet
  threads:
    virtual:
      enabled: true
  datasource:
    url: jdbc:h2:mem:streamingllm;DB_CLOSE_DELAY=-1
    driver-class-name: org.h2.Driver
    username: sa
    password: ""
  jpa:
    hibernate:
      ddl-auto: create-drop
    properties:
      hibernate:
        hbm2ddl:
          import_files_sql_extractor: org.hibernate.tool.schema.internal.script.MultiLineSqlScriptExtractor
          charset_name: UTF-8
    open-in-view: false
  ai:
    openai:
      api-key: ${OPENAI_API_KEY:sk-placeholder-set-OPENAI_API_KEY}
      chat:
        model: gpt-5-mini

app:
  sse:
    heartbeat-interval: 15s
    timeout: 5m

Notice spring.threads.virtual.enabled: true. This is the synergy with our virtual threads tutorial, and it’s not a contrivance. Every open SSE stream pins a request thread for the connection’s whole lifetime. On the standard Tomcat platform-thread pool you exhaust at roughly 200 concurrent streams, which for a streaming chat app is not a lot. Flip virtual threads on and you hold thousands of those blocked-on-write threads cheaply, because a parked virtual thread costs almost nothing. For a workload that is mostly threads sitting idle waiting to write the next token, it’s close to ideal, and we expose a live gauge of active connections (sse.connections.active, via Micrometer and Actuator) so you can watch the number climb.

The streaming logic itself lives in ChatService, but it’s so entangled with resilience that I’ll introduce the whole class in the resilience section rather than show a stripped-down version here. For now, know that the controller calls chatService.streamReply(...) and gets back a Flux<ServerSentEvent<String>>.

The frontend, happy path

The frontend is Next.js 16 with the App Router and React 19, scaffolded with npx create-next-app. There are no streaming dependencies at all; the package.json is just next, react and react-dom plus Tailwind for styling. That’s deliberate. Reading an SSE stream is about fifteen lines of real code, and writing those lines yourself teaches you the wire format better than any library would. We’ll look at the optional AI SDK route at the very end for teams who’d rather not.

A quick word on Next.js 16: a few App Router APIs have shifted from what older tutorials show, so I checked the bundled docs rather than going from memory. The most visible change is that route handler params is now a Promise you await. You’ll see that in the conversation route.

Direct call versus a proxy route

There are two ways the browser can reach Spring. It can call the Spring API directly, which means dealing with CORS and cross-origin cookies, or it can call a Next.js route handler that proxies through to Spring. I recommend the proxy for any real app: it hides the backend origin, makes CORS a non-issue, and gives auth a natural home later (the route handler is server-side, so it can attach credentials the browser never sees). The repo includes a small WebCorsConfig on the Spring side so the direct variant works in dev too, but everything below goes through the proxy.

Here’s the proxy itself, the most important file on the frontend, at app/api/chat/route.ts:

import { BACKEND_URL } from "@/app/lib/backend";

// Node runtime, not Edge: we pass the upstream body straight through and rely on undici aborting
// the in-flight request when the client disconnects.
export const runtime = "nodejs";

export async function POST(request: Request): Promise<Response> {
  const body = await request.text();

  let upstream: Response;
  try {
    upstream = await fetch(`${BACKEND_URL}/api/chat/stream`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Accept: "text/event-stream",
      },
      body,
      signal: request.signal,
    });
  } catch (error) {
    if (request.signal.aborted) {
      return new Response(null, { status: 499 });
    }
    return Response.json(
      { error: `Could not reach the chat backend: ${(error as Error).message}` },
      { status: 502 },
    );
  }

  if (!upstream.ok || !upstream.body) {
    const detail = await upstream.text().catch(() => "");
    return new Response(detail || "Upstream error", { status: upstream.status });
  }

  return new Response(upstream.body, {
    status: 200,
    headers: {
      "Content-Type": "text/event-stream; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      "X-Accel-Buffering": "no",
    },
  });
}

Two things make this work as a streaming proxy rather than a buffering one. We hand upstream.body straight back as the response body without reading it into memory, so tokens flow through as they arrive. And we forward request.signal to the upstream fetch, so when the browser aborts, that abort cascades through the proxy and on to Spring. That’s the whole cancellation story, and we’ll trace it end to end in its own section. The no-transform and X-Accel-Buffering: no headers are repeated here so Next’s own dev infrastructure doesn’t buffer either. The conversation-list and history routes are thinner proxies in the same app/api tree.

The SSE parser, and the bug that cost me an afternoon

The reader is dependency-free, in app/lib/sse.ts. It reads the response body as a stream, splits it on the blank-line frame boundary, and pulls out the id, event and data fields:

export interface SseEvent {
  id?: string;
  event?: string;
  data: string;
}

export async function* parseSse(body: ReadableStream<Uint8Array>): AsyncGenerator<SseEvent> {
  const reader = body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  try {
    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      buffer += decoder.decode(value, { stream: true }).replace(/\r/g, "");

      let boundary: number;
      while ((boundary = buffer.indexOf("\n\n")) !== -1) {
        const frame = buffer.slice(0, boundary);
        buffer = buffer.slice(boundary + 2);
        const event = parseFrame(frame);
        if (event) yield event;
      }
    }

    buffer += decoder.decode().replace(/\r/g, "");
    if (buffer.trim().length > 0) {
      const event = parseFrame(buffer);
      if (event) yield event;
    }
  } finally {
    reader.releaseLock();
  }
}

function parseFrame(frame: string): SseEvent | null {
  let id: string | undefined;
  let event: string | undefined;
  const dataLines: string[] = [];

  for (const line of frame.split("\n")) {
    if (line === "" || line.startsWith(":")) continue;

    const colon = line.indexOf(":");
    const field = colon === -1 ? line : line.slice(0, colon);
    // Note: we do NOT strip a leading space after the colon. Spring writes `data:<token>` verbatim,
    // so for a token like " question" that space is the gap between words — stripping it (as the
    // SSE spec and EventSource do) mashes the text together.
    const value = colon === -1 ? "" : line.slice(colon + 1);

    if (field === "id") id = value;
    else if (field === "event") event = value;
    else if (field === "data") dataLines.push(value);
  }

  if (id === undefined && event === undefined && dataLines.length === 0) return null;
  return { id, event, data: dataLines.join("\n") };
}

That comment in the middle is the war story. The SSE spec says a parser should strip one leading space after the colon in a field line, and the browser’s built-in EventSource dutifully does exactly that. But OpenAI streams tokens with their leading spaces: you get a chunk that is literally " question", space included, because that’s how the words are meant to join. Spring writes that verbatim as data: question. If you strip the leading space, as a spec-compliant parser does, you delete the gap between words, and the rendered text comes out as Tremendoustohearfromyou, every word mashed against the next. I stared at that mangled output for far too long before the penny dropped. The fix is the single most counterintuitive line in the whole project: do not strip the leading space. This, incidentally, is a concrete reason hand-rolling the reader beats reaching for a stock EventSource: you can’t fix this if the parser isn’t yours.

The streaming hook and the UI

The hook that drives it all is app/hooks/useChat.ts. It’s the longest file in the project because it also handles resume and cancellation, so I’ll show the core runStream loop here and leave the full hook to the repo. The shape is: POST the prompt, async-iterate the frames off response.body, and fold each one into React state.

const res = await fetch("/api/chat", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    conversationId: activeIdRef.current,
    message: text,
    lastEventId: resumeFrom,
  }),
  signal: controller.signal,
});

if (!res.ok || !res.body) {
  throw new Error(`Request failed (HTTP ${res.status})`);
}

for await (const evt of parseSse(res.body)) {
  if (evt.event === "meta") {
    const meta = JSON.parse(evt.data) as { conversationId: number; messageId: number };
    setActive(meta.conversationId);
  } else if (evt.event === "token") {
    if (evt.id) lastEventIdRef.current = evt.id;
    patchMessage(assistantId, (m) => ({ content: m.content + evt.data }));
  } else if (evt.event === "done") {
    sawDone = true;
    patchMessage(assistantId, { partial: false });
  } else if (evt.event === "error") {
    setError(evt.data);
    patchMessage(assistantId, { partial: false });
  }
}

Each token frame appends its data to the assistant message’s content and remembers the frame’s id (that’s our resume cursor). The meta frame, which the backend sends first, tells us which conversation this stream belongs to, which we need when the thread was only just created server-side. The done frame clears the streaming flag. When the user sends a message we optimistically push their bubble and an empty assistant bubble to stream into, using descending negative ids as stable React keys so switching threads mid-stream can never write a stray token into the wrong bubble.

The UI in app/components/ChatApp.tsx is plain Tailwind: a sidebar of conversations, a scrolling message pane with a blinking caret on the streaming reply, and a composer that flips its Send button to a red Stop button while streaming. The empty state shows a few clickable suggested prompts (“Hype me up for Monday”, “Pitch me a sandwich”). It’s all in the repo; nothing in it is load-bearing for the streaming mechanics.

Making it survive production: the resilience spine

Everything so far gets you a working chat demo. This section is the part most tutorials skip. Four things go wrong in production that the happy path never mentions, and the ChatService is built around all four. Here’s the complete class, then I’ll go through each mechanism.

package com.tucanoo.streamingllmdemo.service;

import com.tucanoo.streamingllmdemo.config.SseProperties;
import com.tucanoo.streamingllmdemo.dto.ChatRequest;
import com.tucanoo.streamingllmdemo.entity.Conversation;
import com.tucanoo.streamingllmdemo.entity.Message;
import com.tucanoo.streamingllmdemo.entity.Role;
import com.tucanoo.streamingllmdemo.metrics.SseConnectionMetrics;
import com.tucanoo.streamingllmdemo.repository.ConversationRepository;
import com.tucanoo.streamingllmdemo.repository.MessageRepository;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.UserMessage;
import org.springframework.http.codec.ServerSentEvent;
import org.springframework.stereotype.Service;
import reactor.core.publisher.Flux;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;

import java.time.Instant;
import java.util.ArrayList;
import java.util.List;

@Service
@Slf4j
@RequiredArgsConstructor
public class ChatService {

    private final ChatClient chatClient;
    private final ConversationRepository conversationRepository;
    private final MessageRepository messageRepository;
    private final SseProperties sseProperties;
    private final SseConnectionMetrics metrics;

    public Flux<ServerSentEvent<String>> streamReply(ChatRequest request, String headerLastEventId) {
        String resumeCursor = request.lastEventId() != null ? request.lastEventId() : headerLastEventId;
        if (resumeCursor != null && !resumeCursor.isBlank()) {
            return withConnectionMetrics(replayFromCursor(resumeCursor));
        }
        return withConnectionMetrics(freshGeneration(request));
    }

    private Flux<ServerSentEvent<String>> freshGeneration(ChatRequest request) {
        Conversation conversation = resolveConversation(request);

        List<org.springframework.ai.chat.messages.Message> priorHistory =
                buildPromptHistory(conversation.getId());

        persistUserMessage(conversation, request.message());

        Message assistant = persistAssistantPlaceholder(conversation);
        Long assistantId = assistant.getId();

        StringBuilder accumulated = new StringBuilder();

        Flux<ServerSentEvent<String>> tokenEvents = chatClient.prompt()
                .messages(priorHistory)
                .user(request.message())
                .stream()
                .content()
                .map(chunk -> {
                    accumulated.append(chunk);
                    return tokenEvent(assistantId, accumulated.length(), chunk);
                })
                .doOnComplete(() -> persistAssistantContent(assistantId, accumulated.toString(), false))
                .doOnCancel(() -> {
                    log.info("Stream for message {} cancelled at {} chars", assistantId, accumulated.length());
                    persistAssistantContent(assistantId, accumulated.toString(), true);
                })
                .doOnError(error -> {
                    log.warn("Stream for message {} failed at {} chars: {}",
                            assistantId, accumulated.length(), error.toString());
                    persistAssistantContent(assistantId, accumulated.toString(), true);
                })
                .concatWith(Mono.fromSupplier(() -> doneEvent(assistantId, accumulated.length())))
                .onErrorResume(error -> Flux.just(errorEvent("Generation failed: " + error.getMessage())));

        ServerSentEvent<String> meta = ServerSentEvent.<String>builder()
                .event("meta")
                .data("{\"conversationId\":" + conversation.getId() + ",\"messageId\":" + assistantId + "}")
                .build();

        return Flux.concat(Flux.just(meta), withHeartbeats(tokenEvents));
    }

    private Flux<ServerSentEvent<String>> replayFromCursor(String cursor) {
        long messageId;
        int offset;
        try {
            int sep = cursor.indexOf(':');
            messageId = Long.parseLong(cursor.substring(0, sep));
            offset = Integer.parseInt(cursor.substring(sep + 1));
        } catch (RuntimeException ex) {
            return Flux.just(errorEvent("Malformed Last-Event-ID: " + cursor));
        }

        return messageRepository.findById(messageId)
                .map(message -> {
                    String content = message.getContent();
                    int safeOffset = Math.max(0, Math.min(offset, content.length()));
                    List<ServerSentEvent<String>> frames = new ArrayList<>();
                    if (safeOffset < content.length()) {
                        frames.add(tokenEvent(messageId, content.length(), content.substring(safeOffset)));
                    }
                    frames.add(doneEvent(messageId, content.length()));
                    return Flux.fromIterable(frames);
                })
                .orElseGet(() -> Flux.just(errorEvent("No message to resume for id " + messageId)));
    }

    private Flux<ServerSentEvent<String>> withHeartbeats(Flux<ServerSentEvent<String>> dataStream) {
        Flux<ServerSentEvent<String>> heartbeats = Flux.interval(sseProperties.getHeartbeatInterval())
                .map(tick -> ServerSentEvent.<String>builder().comment("keep-alive").build());

        return Flux.merge(dataStream, heartbeats)
                .takeUntil(event -> "done".equals(event.event()) || "error".equals(event.event()))
                .take(sseProperties.getTimeout());
    }

    private Flux<ServerSentEvent<String>> withConnectionMetrics(Flux<ServerSentEvent<String>> stream) {
        return stream
                .doOnSubscribe(subscription -> metrics.increment())
                .doFinally(signal -> metrics.decrement());
    }

    // resolveConversation, persistUserMessage, persistAssistantPlaceholder, persistAssistantContent,
    // saveAssistantContent, buildPromptHistory, tokenEvent, doneEvent and errorEvent omitted here —
    // they're in the repo. The event builders set id = "<messageId>:<charOffset>".
}

I’ve trimmed the small persistence and event-builder helpers from the listing above to keep it readable; the complete file is in the repo. Let’s take the four resilience mechanisms one at a time.

Heartbeats, to tell a dead peer from a slow one

withHeartbeats merges a periodic comment frame into the token stream: Flux.interval(...) emitting a : keep-alive comment every fifteen seconds. The reason this matters is subtle and specific to the servlet model: a servlet only discovers a client has vanished on the next write. If the model is mid-thought and nothing is being written, a connection that died ten seconds ago looks identical to a healthy idle one. The heartbeat forces a write on a fixed cadence, so a dead peer surfaces promptly and the server can reap the thread.

This is exactly where the gpt-5-mini reasoning pause from the opening becomes a real problem rather than a curiosity. That model can sit silent for several seconds before the first token while it reasons. Without heartbeats, an impatient proxy or load balancer might decide the idle connection is dead and cut it, before a single token has been generated. The heartbeat keeps the pipe visibly alive through the think time. The takeUntil and take(timeout) make sure the heartbeat stream stops when the real stream finishes and that nothing runs past the overall timeout.

Reconnection with Last-Event-ID: resume, not restart

I want to be clear about what “resume” actually means here, because it’s easy to oversell. Every token frame carries an id of <messageId>:<charOffset>: the persisted assistant message’s id, plus how many characters we’ve streamed. The browser remembers the last id automatically. If the connection drops mid-stream, the client re-issues the request with that id, and streamReply routes it to replayFromCursor instead of freshGeneration.

What replayFromCursor does is honest and worth understanding: it loads the persisted message, slices off everything after the cursor offset, and streams that tail back, then sends done. It does not re-attach to the original OpenAI generation; once that upstream call is cancelled or gone, the live stream is unrecoverable, and pretending otherwise would be a lie. So resume means “redeliver the text we already saved so the user sees no gap”, not “resurrect the model call”. If the generation had completed before the drop, replay finishes the message cleanly. If it was cut off mid-flight, the user sees everything captured so far and can send again to continue. That’s the realistic, shippable version, and it’s precisely why the lightweight persistence layer earns its place: there’d be nothing to replay from without it.

Backpressure, almost for free

Because the whole pipeline is a Reactor Flux, backpressure is mostly handled for you. If the client reads slowly, the demand signal propagates back up through the chain and Spring AI stops pulling from OpenAI faster than the client can consume. You don’t write explicit buffering logic; the reactive types do it. It’s the one resilience property you get nearly for nothing by building on Flux in the first place.

Defeating proxy buffering

The last failure mode is infrastructure rather than code. A reverse proxy or load balancer that buffers responses will happily collect your entire stream and hand the client one big lump at the end, which silently destroys the streaming UX while every line of your code looks correct. The fix is partly headers, which we already set: X-Accel-Buffering: no on the Spring response (Nginx honours this), Cache-Control: no-cache and no-transform on the proxy. If you front this with Nginx yourself, you also want proxy_buffering off; in the relevant location block. I kept the repo simple, with no bundled Docker or Nginx, but there’s an optional ten-line Nginx config in the README if you want to watch the before-and-after for yourself. It’s an afternoon-saver to know about before you deploy, not after.

Cancellation is money

I skipped past one part of the service deliberately, because it deserves its own section: cancellation. When a user clicks Stop, or navigates away, or closes the tab, you don’t just want the UI to stop updating. You want the OpenAI call itself to terminate, because you’re billed for every token the model generates, whether or not anyone reads it.

Here’s the full path. In the browser, the Stop button calls abortRef.current?.abort(), which aborts the fetch. That abort travels to the Next.js proxy, which forwarded request.signal to its own upstream fetch, so undici tears down the connection to Spring. Spring sees the client disconnect and cancels the Flux. That cancellation propagates up through Spring AI to the WebClient call that’s pulling from OpenAI, and the upstream HTTP request is terminated. Generation stops, and so does the meter.

The ordering matters here. Look at doOnCancel in the service: before the cancellation finishes propagating, we call persistAssistantContent(assistantId, accumulated.toString(), true) to save whatever has streamed so far, marked partial. We save first, then let the cancel kill the upstream call. The same path covers both jobs: the partial we save on a Stop is exactly what replayFromCursor replays if the user comes back.

One honest caveat. The persistence helper schedules its save on Reactor’s boundedElastic scheduler rather than blocking the cancellation path, because the save is a blocking JDBC call and the cancel fires on a Reactor thread. The client already has its tokens by then; the database just catches up a beat later. In a high-stakes system you’d want to confirm the write landed, but for capturing a partial reply it’s the right trade.

Persisting conversations on H2

We’ve already met the entities and the seeding, so this section is short. The point is just to see where the read and write path actually runs, because the resilience layer leans on it. The flow for a fresh generation is: resolveConversation either looks up the existing thread or creates a new one; buildPromptHistory reads the prior turns from H2 to give the model context (skipping any empty or still-partial assistant turns, so a half-finished reply doesn’t poison the next prompt); we save the user’s message; then we save an empty assistant placeholder up front so it has a database id before the first token streams. That id is the stable half of every SSE frame id, which is the whole reason we persist the placeholder early rather than at the end.

As tokens stream, accumulated grows in memory, and on completion (or cancellation, or error) we write the final text back to that same row, flipping partial to false only on a clean finish. The read API in ConversationController serves the sidebar list and a conversation’s history straight from the repositories, so when you reload the page the messages genuinely come from the database, not from React state.

It resets on every startup, by design: ddl-auto: create-drop rebuilds the schema and re-runs import.sql each time. That’s perfect for a demo and a one-command repo, and completely wrong for production, which is the cue for the next section.

Taking it to production

This is a focused streaming demo, not a finished product, and I want to be straight about what’s missing rather than tuck it into a footnote. Two things in particular you’d add before shipping.

First, authentication, which is entirely out of scope here: the seeded “demo users” are cosmetic strings, and anyone can read or write any conversation. The good news is that the proxy architecture makes auth straightforward to slot in. Because the browser talks to the Next.js route handler and never to Spring directly, session cookies are the simplest fit: the browser sends them automatically on the streaming fetch, and the route handler forwards them. If you’d rather go stateless with JWTs, you’d add Spring’s OAuth2 Resource Server on the backend. Either way, watch the SameSite and CORS implications between the Next.js origin and the Spring API.

Second, a durable database. H2 in-memory is ideal for a demo and hopeless for anything real, since it evaporates on restart. Swap it for PostgreSQL (add the Postgres driver), put Flyway in front of the schema instead of create-drop, and drop the import.sql seeding. The entities and repositories don’t change, which is the payoff of having used real JPA rather than a Map from the start.

I’m keeping this brief on purpose; building either out fully would double the length and bury the streaming story, which is the part worth your time.

The AI SDK shortcut

If your team is Node-native and would rather not maintain a hand-rolled reader, you can consume the exact same Spring SSE endpoint with Vercel’s AI SDK. You’d add ai and @ai-sdk/react, and use the useChat hook configured for a plain-text stream protocol, pointing it at the proxy route and reading our data: token frames. It’s genuinely less client code, and for a Node shop already standardised on the SDK that’s a fair trade.

I didn’t build the project this way, and I’d be honest about why. The SDK couples your client to its wire-format expectations, which is fine until our frame shape (the meta event, the <messageId>:<charOffset> ids, the resume protocol) doesn’t line up with what it assumes, and then you’re fighting the abstraction to get the resilience features back that we just built by hand. The fifteen-line reader is the better teacher and gives you full control over the parser, which, as the leading-space bug showed, you sometimes genuinely need. Reach for the SDK when brevity matters more than control; keep the hand-rolled reader when the resilience layer is the point.

Final thoughts

What we’ve built is a streaming chat app that holds up when things go wrong, but I’d downplay the scope honestly. There’s no auth, the database resets every time you restart it, the resume is a replay of saved text rather than a true resurrection of the model call, and a real product would need a great deal more around the edges. This isn’t a finished chat platform; it’s the streaming-and-resilience core done properly, with everything else stripped away so that core stays visible.

That said, the four mechanisms at the heart of it (heartbeats, Last-Event-ID resume, upstream cancellation, and defeating proxy buffering) are the exact things the demos skip and the exact things that bite you in production. Get those right and the rest is ordinary application work.

If you want to take it further, here are a few directions worth your time:

Add real auth. Wire session cookies through the proxy route, or Spring’s OAuth2 Resource Server for JWTs, and mind the cross-origin SameSite rules.

Make persistence durable. Swap H2 for PostgreSQL with Liquibase or Flyway migrations, and keep the conversations across restarts.

Stress the concurrency. Open dozens of streams at once, watch the sse.connections.active gauge, and see for yourself why virtual threads matter for this workload.

Run the Nginx config. Put the optional proxy config from the README in front and watch the stream go from one-big-lump to real-time when you flip proxy_buffering off;.

Try the AI SDK variant. Rebuild the client with @ai-sdk/react and feel where the abstraction helps and where it fights you.

Don’t forget all the code is on GitHub at github.com/tucanoo/streaming-llm-spring-next. Clone it, run the backend, run the frontend, and the whole thing is in front of you.

If you found this useful, the companion piece on virtual threads in Spring Boot 4 goes deep on why holding thousands of blocked SSE connections is suddenly cheap. I hope this has shown you that streaming an LLM properly is far less mysterious than it looks: most of it is just text on a wire, and the hard part is caring about what happens when the wire breaks.