How to Build a RAG Chatbot with Spring Boot and OpenAI — Step by Step (2026)

Learn how to build a RAG chatbot with Spring Boot and OpenAI — the pattern behind modern AI assistants that answer questions from your own documents. This spring boot rag tutorial walks through document ingestion, vector search, and a REST API that returns grounded answers with source citations.

Prerequisites: Basic Java and REST API knowledge. Familiarity with Spring Boot helps. If you are new to securing APIs, see JWT authentication in Spring Boot.

Estimated time: 90–120 minutes. Difficulty: Intermediate.

What Is RAG and Why Use Spring Boot?

Retrieval-Augmented Generation (RAG) combines a large language model (LLM) with a search step over your private data. Instead of fine-tuning the model, you:

1. Ingest documents (PDF, Markdown, HTML) into chunks.
2. Embed each chunk into a vector and store it in a vector database.
3. Retrieve the most relevant chunks when a user asks a question.
4. Generate an answer using the LLM with retrieved context in the prompt.

Spring Boot is a strong choice because Spring AI provides unified APIs for embeddings, vector stores, and chat models — keeping your RAG pipeline testable and production-ready.

Architecture Overview

What You Need

• Java 21 and Maven 3.9+
• Spring Boot 3.3+
• An OpenAI API key (platform.openai.com)
• Sample documents in src/main/resources/docs/ (Markdown or plain text)
• IDE: IntelliJ IDEA or VS Code with Java extensions

Step 1: Create the Spring Boot Project

Generate a project at start.spring.io with:

• Spring Boot 3.3.x
• Dependencies: Spring Web, Spring AI OpenAI
• Group: com.kindsonthegenius, Artifact: rag-chatbot

Or add these dependencies to your pom.xml:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
</dependency>

Step 2: Configure Spring AI and OpenAI

Create src/main/resources/application.yml:

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      embedding:
        options:
          model: text-embedding-3-small
      chat:
        options:
          model: gpt-4o-mini
          temperature: 0.3

server:
  port: 8080

Never commit your API key. Export it before running:

export OPENAI_API_KEY=sk-your-key-here
mvn spring-boot:run

Step 3: Ingest Documents into the Vector Store

Add sample docs under src/main/resources/docs/faq.md. Create a configuration class that loads, splits, embeds, and stores documents on startup:

@Configuration
public class RagConfig {

    @Bean
    CommandLineRunner ingestDocuments(EmbeddingModel embeddingModel,
                                      ResourcePatternResolver resolver) {
        return args -> {
            VectorStore store = new SimpleVectorStore(embeddingModel);
            Resource[] resources = resolver.getResources("classpath:docs/*.md");

            for (Resource resource : resources) {
                String text = resource.getContentAsString(StandardCharsets.UTF_8);
                List<Document> chunks = new TokenTextSplitter().split(new Document(text));
                store.add(chunks);
            }
            // expose store as bean in a @Configuration class
        };
    }
}

Tip: For production, replace SimpleVectorStore with PostgreSQL + pgvector or a managed vector DB. See the InventoryMS Spring Boot API series for database integration patterns.

Step 4: Build the RAG Chat REST Endpoint

Create RagController.java with a POST endpoint that retrieves context and calls the chat model:

@RestController
@RequestMapping("/api/chat")
public class RagController {

    private final VectorStore vectorStore;
    private final ChatModel chatModel;

    public RagController(VectorStore vectorStore, ChatModel chatModel) {
        this.vectorStore = vectorStore;
        this.chatModel = chatModel;
    }

    @PostMapping
    public ChatResponse ask(@RequestBody ChatRequest request) {
        List<Document> context = vectorStore.similaritySearch(
            SearchRequest.query(request.question()).withTopK(4));

        String prompt = """
            Answer using ONLY the context below. If unknown, say you don't know.
            Context:
            %s

            Question: %s
            """.formatted(formatContext(context), request.question());

        String answer = chatModel.call(prompt);
        return new ChatResponse(answer, context.stream().map(Document::getId).toList());
    }

    record ChatRequest(String question) {}
    record ChatResponse(String answer, List<String> sources) {}
}

Step 5: Test the Chatbot API

Start the app and send a request with curl or Postman:

curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"question":"How do I reset my password?"}'

Expected JSON response:

{
  "answer": "To reset your password, click Forgot Password on the login page...",
  "sources": ["doc-faq-003", "doc-faq-007"]
}

Production Tips

• Rate limiting: Protect /api/chat with Spring Security or an API gateway.
• Chunk size: Tune the text splitter (500–1000 tokens) for your document type.
• Observability: Log retrieved chunk IDs and latency for debugging hallucinations.
• Deploy: Containerize with Docker and deploy to Kubernetes — see Deploy Spring Boot to Kubernetes (2026).
• Python alternative: For ML-heavy pipelines, explore the Python tutorials subsite.

Frequently Asked Questions

What is RAG in Spring Boot?
RAG (Retrieval-Augmented Generation) in Spring Boot means loading your documents into a vector store, retrieving relevant chunks at query time, and passing them as context to an LLM via Spring AI — so answers are grounded in your data, not just the model’s training set.

Do I need OpenAI or can I use a local model?
Spring AI supports OpenAI, Azure OpenAI, Ollama, and other providers. Swap the starter dependency and configuration — the RAG pipeline code stays largely the same.

How much does a RAG chatbot cost to run?
Costs depend on embedding volume and chat volume. Using text-embedding-3-small and gpt-4o-mini keeps dev/testing costs low. Cache embeddings after first ingest to avoid re-embedding unchanged documents.

Why does my chatbot hallucinate?
Increase topK retrieval, tighten the system prompt (“answer ONLY from context”), or improve chunk quality. Empty vector stores also cause hallucinations — verify ingest logs on startup.

Can I use PDFs instead of Markdown?
Yes. Spring AI provides PDF document readers. Add the PDF reader dependency, load files from classpath:docs/*.pdf, and pipe through the same splitter and vector store.

How do I secure the RAG API?
Add Spring Security with JWT — follow the JWT in Spring Boot tutorial and protect /api/chat with @PreAuthorize or OAuth2 resource server config.

Next Steps

• Spring Boot 3 + Java 21 Virtual Threads — scale concurrent chat requests
• Deploy Spring Boot to Kubernetes — containerize and ship your RAG service
• Secure REST APIs with JWT in Spring Boot
• Python tutorials — LangChain and data science paths
• Alkademy courses — live Spring Boot and AI classes