How to Build a RAG Chatbot with Spring Boot and OpenAI — Step by Step (2026)

Learn how to build a RAG chatbot with Spring Boot and OpenAI — the pattern behind modern AI assistants that answer questions from your own documents.

TL;DR

• RAG combines document retrieval with an LLM so a chatbot can answer questions using your own data.

• Spring AI and OpenAI provide the building blocks for embeddings, chat models, and vector-based retrieval in Spring Boot.

• The tutorial covers the complete RAG pipeline, from ingesting and splitting documents to storing embeddings and retrieving relevant context.

• A Spring Boot REST endpoint retrieves relevant document chunks and passes them to the chat model to generate grounded answers with source references.

• For production, focus on security, chunking, rate limiting, observability, and a persistent vector store such as PostgreSQL with pgvector.

This spring boot rag tutorial walks through document ingestion, vector search, and a REST API that returns grounded answers with source citations.

Prerequisites: Basic Java and REST API knowledge. Familiarity with Spring Boot helps. If you are new to securing APIs, see JWT authentication in Spring Boot.

Estimated time: 90–120 minutes. Difficulty: Intermediate.

What Is RAG and Why Use Spring Boot?

Retrieval-Augmented Generation (RAG) combines a large language model (LLM) with a search step over your private data. Instead of fine-tuning the model, you:

1. Ingest documents (PDF, Markdown, HTML) into chunks.
2. Embed each chunk into a vector and store it in a vector database.
3. Retrieve the most relevant chunks when a user asks a question.
4. Generate an answer using the LLM with retrieved context in the prompt.

Spring Boot is a strong choice because Spring AI provides unified APIs for embeddings, vector stores, and chat models — keeping your RAG pipeline testable and production-ready.

Architecture Overview

What You Need

• Java 21 and Maven 3.9+
• Spring Boot 3.3+
• An OpenAI API key (platform.openai.com)
• Sample documents in src/main/resources/docs/ (Markdown or plain text)
• IDE: IntelliJ IDEA or VS Code with Java extensions

Step 1: Create the Spring Boot Project

Generate a project at start.spring.io with:

• Spring Boot 3.3.x
• Dependencies: Spring Web, Spring AI OpenAI
• Group: com.kindsonthegenius, Artifact: rag-chatbot

Or add these dependencies to your pom.xml:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>

Step 2: Configure Spring AI and OpenAI

Create src/main/resources/application.yml:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        options:
          model: text-embedding-3-small
      chat:
        options:
          model: gpt-4o-mini
          temperature: 0.3

server:
  port: 8080

Never commit your API key. Export it before running:

export OPENAI_API_KEY=sk-your-key-here
mvn spring-boot:run

Step 3: Ingest Documents into the Vector Store

Add sample docs under src/main/resources/docs/faq.md. Create a configuration class that loads, splits, embeds, and stores documents on startup:

@Configuration
public class RagConfig {

    @Bean
    CommandLineRunner ingestDocuments(EmbeddingModel embeddingModel,
                                      ResourcePatternResolver resolver) {
        return args -> {
            VectorStore store = new SimpleVectorStore(embeddingModel);
            Resource[] resources = resolver.getResources("classpath:docs/*.md");

            for (Resource resource : resources) {
                String text = resource.getContentAsString(StandardCharsets.UTF_8);
                List<Document> chunks = new TokenTextSplitter().split(new Document(text));
                store.add(chunks);
            }
            // expose store as bean in a @Configuration class
        };
    }
}

Tip: For production, replace SimpleVectorStore with PostgreSQL + pgvector or a managed vector DB. See the InventoryMS Spring Boot API series for database integration patterns.

Step 4: Build the RAG Chat REST Endpoint

Create RagController.java with a POST endpoint that retrieves context and calls the chat model:

@RestController
@RequestMapping("/api/chat")
public class RagController {

    private final VectorStore vectorStore;
    private final ChatModel chatModel;

    public RagController(VectorStore vectorStore, ChatModel chatModel) {
        this.vectorStore = vectorStore;
        this.chatModel = chatModel;
    }

    @PostMapping
    public ChatResponse ask(@RequestBody ChatRequest request) {
        List<Document> context = vectorStore.similaritySearch(
            SearchRequest.query(request.question()).withTopK(4));

        String prompt = """
            Answer using ONLY the context below. If unknown, say you don't know.
            Context:
            %s

            Question: %s
            """.formatted(formatContext(context), request.question());

        String answer = chatModel.call(prompt);
        return new ChatResponse(answer, context.stream().map(Document::getId).toList());
    }

    record ChatRequest(String question) {}
    record ChatResponse(String answer, List<String> sources) {}
}

Step 5: Test the Chatbot API

Start the app and send a request with curl or Postman:

curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"question":"How do I reset my password?"}'

Expected JSON response:

{
  "answer": "To reset your password, click Forgot Password on the login page...",
  "sources": ["doc-faq-003", "doc-faq-007"]
}

Production Tips

• Rate limiting: Protect /api/chat with Spring Security or an API gateway.
• Chunk size: Tune the text splitter (500–1000 tokens) for your document type.
• Observability: Log retrieved chunk IDs and latency for debugging hallucinations.
• Deploy: Containerize with Docker and deploy to Kubernetes — see Deploy Spring Boot to Kubernetes (2026).
• Python alternative: For ML-heavy pipelines, explore the Python tutorials subsite.

Frequently Asked Questions

What is RAG in Spring Boot?
RAG (Retrieval-Augmented Generation) in Spring Boot means loading your documents into a vector store, retrieving relevant chunks at query time, and passing them as context to an LLM via Spring AI — so answers are grounded in your data, not just the model’s training set.

Do I need OpenAI or can I use a local model?
Spring AI supports OpenAI, Azure OpenAI, Ollama, and other providers. Swap the starter dependency and configuration — the RAG pipeline code stays largely the same.

How much does a RAG chatbot cost to run?
Costs depend on embedding volume and chat volume. Using text-embedding-3-small and gpt-4o-mini keeps dev/testing costs low. Cache embeddings after first ingest to avoid re-embedding unchanged documents.

Why does my chatbot hallucinate?
Increase topK retrieval, tighten the system prompt (“answer ONLY from context”), or improve chunk quality. Empty vector stores also cause hallucinations — verify ingest logs on startup.

Can I use PDFs instead of Markdown?
Yes. Spring AI provides PDF document readers. Add the PDF reader dependency, load files from classpath:docs/*.pdf, and pipe through the same splitter and vector store.

How do I secure the RAG API?
Add Spring Security with JWT — follow the JWT in Spring Boot tutorial and protect /api/chat with @PreAuthorize or OAuth2 resource server config.

Final Thought

RAG provides a practical way to build AI applications that can work with your own documents without fine-tuning a language model. With Spring Boot and Spring AI, the process can be integrated into a familiar Java application architecture, from document ingestion and vector search to a REST API that returns grounded answers.

The most important part of a reliable RAG chatbot is not simply connecting an LLM. Retrieval quality matters just as much. Good document preparation, appropriate chunking, relevant search results, clear prompts, and source tracking all contribute to more useful and trustworthy responses.

Once the basic pipeline works, you can take it further with persistent vector storage, authentication, monitoring, better document loaders, conversation history, and production deployment. The tutorial provides the foundation; the next step is adapting that foundation to the data and workflows your application actually needs.

Next Steps

• Spring Boot 3 + Java 21 Virtual Threads — scale concurrent chat requests
• Deploy Spring Boot to Kubernetes — containerize and ship your RAG service
• Secure REST APIs with JWT in Spring Boot
• Python tutorials — LangChain and data science paths
• Alkademy courses — live Spring Boot and AI classes