Spring AI in Production: Building a RAG Application with Spring Boot 4 and PostgreSQL pgvector

What you’ll learn: How Retrieval-Augmented Generation (RAG) works in plain terms, how to set up Spring AI with the OpenAI and pgvector starters on Spring Boot 4, how to load your own documents into a vector store, how to ask grounded questions with the QuestionAnswerAdvisor, how to show the sources behind every answer, and the production details that matter (version pinning, cost, tuning, and security). The complete working example is on GitHub: tucanoo/springboot-ai-rag-tutorial.

Large language models are confident, fluent, and — when asked about your business — often wrong. A model trained on the public internet does not know your refund policy, your API rate limits, or the steps to reset a password in your product. Ask it anyway and it will happily invent an answer. That is called a hallucination, and it is the single biggest reason teams hesitate to put an LLM in front of customers.

Retrieval-Augmented Generation (RAG) is the fix. Instead of hoping the model already knows the answer, you retrieve the relevant pieces of your own documentation first, add them to the prompt as context, and let the model generate an answer based only on what you gave it. The result is an assistant that answers from your knowledge base — and admits when it does not know.

For a long time, building this meant Python. Not any more. With Spring AI now mature and PostgreSQL’s pgvector extension widely available, Java developers can build a complete, production-grade RAG application end to end — and reuse the database they are already running.

In this tutorial we will build a customer support assistant for a fictional product called Tucanoo Cloud. It answers questions from a folder of help articles, shows which article each answer came from, and politely refuses when the answer is not in its knowledge base. By the end you will have a working web app you fully understand.

What is RAG, in plain English?

Let us define the few terms you need. Keep them in mind and the rest of the article is easy.

• Embedding — a list of numbers (a vector) that captures the meaning of a piece of text. Two texts with similar meaning have similar vectors, even if they use different words. A model called an embedding model produces these numbers.
• Vector store — a database that stores these vectors and can quickly find the ones most similar to a given vector. We will use PostgreSQL with the pgvector extension.
• Chunk — a small piece of a document. We split long documents into chunks because searching and answering both work better on small, focused passages.
• Similarity search — given a question, we turn it into a vector and ask the store: “which chunks are closest in meaning to this?”

Here is the whole flow in three steps:

1. Ingest (done once): read your documents, split them into chunks, turn each chunk into an embedding, and store them in pgvector.
2. Retrieve (on every question): turn the user’s question into an embedding and find the most similar chunks.
3. Generate: put those chunks into the prompt as context and ask the language model to answer using only that context.

That is RAG. The clever part is that Spring AI does steps 1 and 2 for you with very little code.

The stack, and why each piece

We use PostgreSQL as the vector store on purpose. Most teams already run Postgres. Adding the pgvector extension turns it into a capable vector database, so you do not need to operate a new, separate system just for AI. That is a real production advantage.

Prerequisites

• Java 17 or newer. This sample is built and tested on Java 25.
• A running PostgreSQL with the pgvector extension. You need both. The easiest way is the official pgvector/pgvector Docker image, which already includes the extension. If you have an existing Postgres, install pgvector for it and run CREATE EXTENSION vector; in your database.
• An OpenAI API key with a little credit. The whole tutorial costs only a few cents. (Prefer a free, local setup? See the Ollama section near the end.)
• About 30–45 minutes.

Creating the project

We start from start.spring.io (the Spring Initializr), which is built into IntelliJ IDEA under File → New → Project → Spring Boot.

Fill in the project details: a Gradle (Groovy) project, language Java, Spring Boot 4.0.6, group com.tucanoo, artifact support-rag, and JDK 25.

On the next screen, add these dependencies:

• Spring Web, Thymeleaf, Spring Boot Actuator
• OpenAI and PGvector Vector Database (under AI)
• Markdown Document Reader (it reads our .md help files)
• PostgreSQL Driver

Click Create and open the project. Initializr also adds Spring Boot DevTools (handy live-reload while you develop) — keep it. If you happened to tick Lombok, you can remove it; this sample uses plain Java records and does not need it.

The build file

Spring Initializr generates a build.gradle very close to this. There are a few Spring Boot 4 details worth pointing out, because they trip people up:

plugins {
    id 'java'
    id 'org.springframework.boot' version '4.0.6'
    id 'io.spring.dependency-management' version '1.1.7'
}
group = 'com.tucanoo'
version = '0.0.1-SNAPSHOT'
java {
    toolchain { languageVersion = JavaLanguageVersion.of(25) }
}
repositories {
    mavenCentral()   // Spring AI 2.0.0-M8 is on Maven Central — no special repository needed
}
ext { set('springAiVersion', '2.0.0-M8') }
dependencies {
    // In Spring Boot 4 the web starter was renamed: 'spring-boot-starter-web' -> '-webmvc'
    implementation 'org.springframework.boot:spring-boot-starter-webmvc'
    implementation 'org.springframework.boot:spring-boot-starter-thymeleaf'
    implementation 'org.springframework.boot:spring-boot-starter-actuator'
    implementation 'org.springframework.ai:spring-ai-starter-model-openai'
    implementation 'org.springframework.ai:spring-ai-starter-vector-store-pgvector'
    implementation 'org.springframework.ai:spring-ai-advisors-vector-store'    // QuestionAnswerAdvisor
    implementation 'org.springframework.ai:spring-ai-markdown-document-reader'
    developmentOnly 'org.springframework.boot:spring-boot-devtools'
    runtimeOnly 'org.postgresql:postgresql'
}
dependencyManagement {
    imports { mavenBom "org.springframework.ai:spring-ai-bom:${springAiVersion}" }
}

Three things to notice:

1. The Spring AI BOM (dependencyManagement) sets the version of every Spring AI module for you. You pin it once, with springAiVersion.
2. No special repository. Some older tutorials add a Spring “milestone” repository. With 2.0.0-M8 that is no longer needed — it is on Maven Central.
3. spring-boot-starter-webmvc. In Spring Boot 4 the old spring-boot-starter-web was renamed. If you copy an older build.gradle, this is the change that breaks the build.

Setting up PostgreSQL and pgvector

You need a PostgreSQL database with the vector extension available. The fastest way to get one is Docker:

docker run --name pgvector -e POSTGRES_PASSWORD=postgres \  -e POSTGRES_DB=spring-ai-rag -p 5432:5432 -d pgvector/pgvector:pg17

That image already contains pgvector. If you instead point the app at your own Postgres, just make sure pgvector is installed for it and run CREATE EXTENSION vector; once in your database.

You do not need to create any tables by hand. Spring AI creates and manages the vector table for you (we enable that below).

Configuration

Put this in src/main/resources/application.yaml:

spring:
  application:
    name: support-rag
  datasource:
    url: jdbc:postgresql://localhost:5432/spring-ai-rag
    username: postgres
    password: postgres
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o-mini          # a small, cheap chat model
          temperature: 0.2            # low value = stay close to the context
      embedding:
        options:
          model: text-embedding-3-small   # 1536 dimensions
    vectorstore:
      pgvector:
        initialize-schema: true       # creates the vector_store table + the 'vector' extension
        index-type: HNSW
        distance-type: COSINE_DISTANCE
        dimensions: 1536              # MUST match the embedding model above
app:
  kb:
    location: classpath:/kb/*.md
  rag:
    top-k: 4                          # how many chunks to retrieve per question
    similarity-threshold: 0.45        # ignore matches weaker than this

Two rules to remember:

• Never hard-code your API key. We read it from an environment variable, OPENAI_API_KEY. Set it in your shell or in a local .env file that you keep out of version control.
• dimensions must match the embedding model. text-embedding-3-small produces vectors of length 1536. If you change the embedding model, change this number too, or pgvector will reject the data.

initialize-schema: true tells Spring AI to create the vector_store table and enable the vector extension on startup. That is why you do not write any SQL.

The knowledge base

Our assistant answers from a folder of Markdown help articles in src/main/resources/kb/. We wrote twelve short ones for Tucanoo Cloud — billing, password reset, API rate limits, refunds, regions, and so on. Here is a shortened example, 07-api-rate-limits.md:

# API Rate Limits
Every Tucanoo Cloud plan can call our API, but the number of requests per
minute depends on your plan.
## Limits per plan
- **Free:** 60 requests per minute
- **Pro:** 600 requests per minute
- **Business:** 3,000 requests per minute
- **Enterprise:** custom limits, set with your account manager
## When you go over the limit
If you send too many requests, the API returns HTTP status **429 Too Many
Requests**. The response includes a `Retry-After` header and an
`X-RateLimit-Remaining` header so your code knows when to try again.

Good source documents make a good assistant. Keep articles focused, factual, and consistent — if two articles disagree, your assistant will too.

Step 1 — Ingestion: getting your documents into the vector store

Now the first real piece of code. This component runs once when the app starts. It reads every Markdown file, splits each into chunks, and stores them. As it stores each chunk, Spring AI automatically calls OpenAI to create the embedding — you never call the embedding API yourself.

@Component
public class KnowledgeBaseLoader implements ApplicationRunner {
    private static final Logger log = LoggerFactory.getLogger(KnowledgeBaseLoader.class);
    private final VectorStore vectorStore;
    private final JdbcTemplate jdbcTemplate;
    private final String knowledgeBaseLocation;
    public KnowledgeBaseLoader(VectorStore vectorStore, JdbcTemplate jdbcTemplate,
                               @Value("${app.kb.location}") String knowledgeBaseLocation) {
        this.vectorStore = vectorStore;
        this.jdbcTemplate = jdbcTemplate;
        this.knowledgeBaseLocation = knowledgeBaseLocation;
    }
    @Override
    public void run(ApplicationArguments args) throws Exception {
        // Idempotency: if we already have data, do nothing. Safe to restart.
        Integer existing = jdbcTemplate.queryForObject("SELECT count(*) FROM vector_store", Integer.class);
        if (existing != null && existing > 0) {
            log.info("Knowledge base already loaded ({} chunks). Skipping.", existing);
            return;
        }
        Resource[] files = new PathMatchingResourcePatternResolver().getResources(knowledgeBaseLocation);
        // 1. Read each markdown file, remembering which file each chunk came from.
        List<Document> documents = new ArrayList<>();
        for (Resource file : files) {
            var config = MarkdownDocumentReaderConfig.builder()
                    .withAdditionalMetadata("source", file.getFilename())
                    .build();
            documents.addAll(new MarkdownDocumentReader(file, config).get());
        }
        // 2. Split documents into smaller chunks. Metadata (our "source") is copied to each chunk.
        List<Document> chunks = TokenTextSplitter.builder()
                .withChunkSize(400)
                .build()
                .apply(documents);
        // 3. Store them. Spring AI embeds each chunk via OpenAI and writes it to pgvector.
        log.info("Embedding and storing {} chunks...", chunks.size());
        vectorStore.add(chunks);
        log.info("Knowledge base ingestion complete.");
    }
}

A few points:

• MarkdownDocumentReader reads a Markdown file and turns it into Document objects. We attach the file name as source metadata so we can show it as a citation later.
• TokenTextSplitter breaks long text into chunks of about 400 tokens. It copies our metadata onto every chunk.
• vectorStore.add(chunks) is where the magic happens: each chunk is embedded and stored in one call.
• The idempotency check means you can restart the app as often as you like without creating duplicate data. (To force a reload, empty the table: TRUNCATE vector_store;.)

When you start the app, you will see it find 12 files and store the chunks. In our run that produced 66 chunks.

Step 2 — The RAG query: ChatClient and QuestionAnswerAdvisor

The retrieval and generation steps are handled by a ChatClient with a QuestionAnswerAdvisor attached. An advisor in Spring AI is a piece that sits between your call and the model. The QuestionAnswerAdvisor does the RAG work automatically: before each call it searches the vector store, and it adds the best matches to the prompt as context.

@Configuration
public class ChatClientConfig {
    private static final String SYSTEM_PROMPT = """
            You are a friendly customer support assistant for "Tucanoo Cloud", a cloud platform
            for hosting and managing web applications.
            Answer the user's question using ONLY the context that is given to you from the
            knowledge base. Follow these rules:
            - If the answer is not in the context, say clearly that you do not have that
              information, and suggest the user contact support. Never guess or invent details.
            - Keep answers short, clear and friendly. Use simple words.
            - When it helps, give the steps as a short numbered list.
            """;
    @Bean
    public ChatClient supportChatClient(ChatClient.Builder builder,
                                        VectorStore vectorStore,
                                        @Value("${app.rag.top-k}") int topK,
                                        @Value("${app.rag.similarity-threshold}") double similarityThreshold) {
        SearchRequest searchRequest = SearchRequest.builder()
                .topK(topK)                                  // how many chunks to retrieve
                .similarityThreshold(similarityThreshold)    // ignore weak matches
                .build();
        QuestionAnswerAdvisor ragAdvisor = QuestionAnswerAdvisor.builder(vectorStore)
                .searchRequest(searchRequest)
                .build();
        return builder
                .defaultSystem(SYSTEM_PROMPT)
                .defaultAdvisors(ragAdvisor)
                .build();
    }
}

The system prompt is doing important work. It tells the model to answer only from the provided context and to admit when the answer is not there. This is what makes the assistant trustworthy instead of inventive. RAG retrieves the right context; the system prompt makes sure the model stays inside it.

Step 3 — Showing your sources

A support answer you cannot trace is hard to trust. The good news: the QuestionAnswerAdvisor keeps the documents it retrieved, and exposes them in the response metadata under a known key. We read them back and turn them into a tidy list of sources.

@Service
public class RagService {
    private final ChatClient chatClient;
    public RagService(ChatClient supportChatClient) {
        this.chatClient = supportChatClient;
    }
    public ChatAnswer ask(String question) {
        ChatResponse response = chatClient.prompt()
                .user(question)
                .call()
                .chatResponse();
        String answer = (response != null && response.getResult() != null)
                ? response.getResult().getOutput().getText()
                : "Sorry, I could not generate an answer. Please try again.";
        return new ChatAnswer(answer, extractSources(response));
    }
    private List<Source> extractSources(ChatResponse response) {
        if (response == null) return List.of();
        // The advisor stores the retrieved chunks under this key.
        List<Document> documents = response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);
        if (documents == null || documents.isEmpty()) return List.of();
        // Several chunks can come from the same article — keep one entry per file.
        Map<String, Source> byFile = new LinkedHashMap<>();
        for (Document document : documents) {
            String file  = String.valueOf(document.getMetadata().getOrDefault("source", "knowledge-base"));
            String title = String.valueOf(document.getMetadata().getOrDefault("title", file));
            byFile.computeIfAbsent(file, k -> new Source(title, file, excerpt(document.getText())));
        }
        return new ArrayList<>(byFile.values());
    }
    // prettifyFileName(...) and excerpt(...) omitted for brevity — see the GitHub repo.
}

The key line is response.getMetadata().get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS). That constant ("qa_retrieved_documents") is how the advisor hands back the chunks it used. We de-duplicate by source file so the user sees one entry per article, not four chunks from the same one.

The small DTOs (ChatRequest, ChatAnswer, Source) are plain Java records — see the repo.

Step 4 — The web layer

A tiny controller serves the chat page and one JSON endpoint:

@Controller
public class ChatController {
    private final RagService ragService;
    public ChatController(RagService ragService) {
        this.ragService = ragService;
    }
    @GetMapping("/")
    public String index() {
        return "chat";   // the Thymeleaf template at templates/chat.html
    }
    @PostMapping("/api/chat")
    @ResponseBody
    public ChatAnswer chat(@RequestBody ChatRequest request) {
        return ragService.ask(request.question());
    }
}

The front end is a single Thymeleaf page with a small piece of plain JavaScript that posts the question and renders the answer. No build step, no framework. Here is the heart of the JavaScript:

const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ question: question })
});
const data = await res.json();
// data.answer  -> the text;  data.sources -> the list of articles used

he full chat.html, CSS and JavaScript are in the repository.

Running it

Set your key and start the app:

export OPENAI_API_KEY="sk-..."./gradlew bootRun

On the first start the app ingests the knowledge base (you will see it embed and store the chunks). Then open http://localhost:8080 and ask a question.

Ask “What are the API rate limits on the Pro plan?” and you get 600 requests per minute — straight from our 07-api-rate-limits.md article — with the source listed underneath. The answer is grounded in your documents, not in the model’s imagination.

Now ask something the knowledge base does not cover, such as “How do I set up SAML single sign-on?”:

nstead of inventing an SSO guide, the assistant says it does not have that information and suggests contacting support — and shows no sources. That honest “I don’t know” is exactly the behaviour you want in front of customers, and it is the combination of the similarity threshold (weak matches are dropped) and the strict system prompt that produces it.

Taking it to production

A working demo is not the same as a production service. Here is what to tighten before you ship.

• Pin your versions. We are on a milestone (2.0.0-M8). Pin the exact Spring AI and Spring Boot versions, and treat an upgrade to 2.0.0 GA as a deliberate, tested step — not an automatic one.
• Control cost. Every question costs one embedding call plus one chat call. Use a small chat model (gpt-4o-mini here) and a small embedding model. Cache or rate-limit if traffic is high.
• Tune retrieval. Three knobs change answer quality: chunk size (smaller = more precise, larger = more context), top-k (how many chunks to retrieve), and the similarity threshold (how strict the match must be). We use a threshold of 0.45; raise it and the assistant says “I don’t know” more often, lower it and it tries harder but risks weaker answers. We made these configuration properties so you can tune without recompiling.
• Mind the index. We use an HNSW index, which is fast and accurate for similarity search and the right default for most workloads.
• Watch prompt injection. A user (or a poisoned document) might try to make the model ignore your rules. Keep your system prompt strict, never let retrieved text override your instructions, and do not give the model powerful tools it does not need.
• Use Actuator. We included Spring Boot Actuator. Expose /actuator/health so your platform can check the app — and the database connection — is alive.
• Plan re-ingestion. Our loader runs once and then skips. When your documents change, you need a strategy: empty the table and reload, or upsert by document id. For a real product, ingestion usually becomes its own scheduled job, separate from the web app.

Bonus: run it for free with Ollama

Do not want to pay OpenAI, or need everything to stay on your own machine? Swap OpenAI for Ollama, which runs models locally.

Change the starter in build.gradle:

// replace spring-ai-starter-model-openai with:
implementation 'org.springframework.ai:spring-ai-starter-model-ollama'

And the configuration:

spring:
  ai:
    ollama:
      chat:
        options:
          model: llama3.1
      embedding:
        options:
          model: nomic-embed-text
    vectorstore:
      pgvector:
        dimensions: 768   # nomic-embed-text is 768 dimensions, NOT 1536 — update this!

The one trap: the embedding dimensions are different. nomic-embed-text produces 768-length vectors, so you must change dimensions to 768 and start with an empty table. Everything else — the ingestion, the advisor, the citations — stays exactly the same. That is the beauty of Spring AI’s abstractions: the provider is a configuration detail.

Troubleshooting

• A dimension error from pgvector. Your dimensions setting does not match your embedding model. Fix the value and empty the vector_store table.
• “OPENAI_API_KEY … must not be empty.” The environment variable is not set in the shell that runs the app.
• Cannot connect / extension "vector" does not exist. Your Postgres is not running, or pgvector is not installed on it. Use the pgvector/pgvector Docker image, or install the extension.
• Every answer is “I don’t know.” Your similarity threshold is too high, or ingestion did not run. Check the startup log for the “ingestion complete” line and the row count in vector_store.

Conclusion

You have built a complete RAG application in Java: it loads your documents into PostgreSQL with pgvector, retrieves the most relevant pieces for each question, answers using only that context, shows its sources, and honestly declines when it does not know. No Python, no separate vector database, and surprisingly little code — Spring AI does the heavy lifting.

The same pattern scales far beyond a support bot: internal documentation search, policy assistants, product help, research tools. Swap the knowledge base and the system prompt, and you have a new assistant.

The full, working source is on GitHub: tucanoo/spring-ai-rag-tutorial. Clone it, point it at your own documents, and see what your data can answer.

Building AI features into a Java application and want a hand getting it production-ready? Talk to us at Tucanoo — it is what we do.