knowledge-rag-assistant

Knowledge RAG Assistant

A production-ready RAG-powered knowledge assistant demonstrating modern LLM backend development practices: custom RAG pipeline, MCP server, PydanticAI tool-calling agent, LangGraph Corrective RAG, Langfuse observability, streaming, and semantic caching.

Example domain: Digimon knowledge base (DAPI)

Demo

Record your own: install asciinema and agg, then run ./scripts/record_demo.sh — it records the full multi-turn conversation and converts to docs/demo.gif automatically.

pip install asciinema          # or: brew install asciinema
cargo install --git https://github.com/asciinema/agg   # or: brew install agg
./scripts/record_demo.sh

# Multi-turn conversation via streaming
curl -N -X POST http://localhost:8000/api/v1/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"query": "Tell me about Agumon"}'

# Follow-up — pass the conversation_id from the previous response
curl -N -X POST http://localhost:8000/api/v1/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"query": "What are its digivolutions?", "conversation_id": "<id from above>"}'

Tech Stack

Layer	Technology
API	FastAPI (Python 3.11+)
LLM	Anthropic claude-haiku-4-5
Embeddings	Voyage AI voyage-3 (1024 dims)
Vector DB	Qdrant
Relational DB	PostgreSQL + SQLAlchemy
Caching	Redis
Observability	Langfuse
Agent Protocol	MCP (Model Context Protocol)
Typed Agent	PydanticAI (tool-calling agent)
Agentic RAG	LangGraph (Corrective RAG workflow)
Guardrails	Custom InputGuard (LLM-based, fail-open)
RAG Evaluation	RAGAS (faithfulness, relevancy, precision, recall)
Infrastructure	Docker + docker-compose

Architecture

  ┌─────────────────────────────────────────────────────┐
  │                    Client (HTTP)                    │
  └─────────────────────────┬───────────────────────────┘
                            │
             ┌──────────────┴─────────────┐
             │                            │
  ┌──────────▼──────────────┐  ┌──────────▼───────────────────┐
  │   POST /api/v1/chat     │  │   POST /api/v1/agent         │
  │      (Custom RAG)       │  │   POST /api/v1/agent/graph   │
  └──────────┬──────────────┘  └──────────┬───────────────────┘
             │                            │
             └──────────────┬─────────────┘
                            │
  ┌─────────────────────────▼───────────────────────────┐
  │                      RAG Core                       │
  │      Retriever → PromptBuilder → LLMClient          │
  └───────────────────┬─────────────────┬───────────────┘
                      │                 │
           ┌──────────▼──┐     ┌────────▼──────┐
           │   Qdrant    │     │     DAPI      │
           │  (vectors)  │     │  (live API)   │
           └─────────────┘     └───────────────┘

  LangGraph Corrective RAG flow:
  retrieve → grade → generate
                └──→ rewrite → retrieve (max 2 loops)

  Observability: Langfuse traces all LLM calls and retrievals
  Caching:       Redis caches /chat responses by query + filters

Features

• Custom RAG Pipeline: Retriever → Prompt Builder → LLM, built from scratch without LangChain
• PydanticAI Agent (/api/v1/agent): type-safe agent with 4 tools (RAG search, by name, by level, skills) and automatic tool selection
• LangGraph Corrective RAG (/api/v1/agent/graph): agentic retrieve → grade → generate workflow with query rewriting when docs aren't relevant
• MCP Server: 4 tools exposing Digimon data (by name, level, ID, skills)
• Conversation History: Multi-turn memory stored in Redis — pass conversation_id to maintain context across calls (last 5 turns, 1h TTL)
• Streaming: SSE endpoint (/chat/stream) returns tokens as they are generated — zero-buffering, real-time UX
• Observability: Full tracing with Langfuse — per-trace numeric scores (latency, tokens, retrieval quality) and async LLM-as-judge evaluation (faithfulness, relevance, context relevance)
• Semantic Caching: Redis-based response caching to reduce API costs
• Data Ingestion: Async pipeline from DAPI → Voyage AI embeddings → Qdrant + PostgreSQL

Setup

Prerequisites

• Docker and Docker Compose
• Python 3.11+
• Anthropic API key → console.anthropic.com
• Voyage AI API key → dash.voyageai.com

Installation

1. Clone the repository:

git clone https://github.com/JaimeRam/knowledge-rag-assistant.git
cd knowledge-rag-assistant

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install Python dependencies:

pip install -r requirements.txt

4. Copy and configure the environment file:

cp .env.example .env
# Edit .env — set ANTHROPIC_API_KEY and VOYAGE_API_KEY at minimum

5. Start infrastructure services (Qdrant, Redis, PostgreSQL, Langfuse):

make services

6. Run data ingestion:

make ingest

Voyage AI free tier note: without a payment method the limit is 3 RPM. The default INGEST_LIMIT=100 in .env keeps ingestion under ~5 minutes. Set INGEST_LIMIT=0 to ingest all ~1450 Digimon once you add a payment method (200M free tokens still apply).

7. Start the API:

make run

The API will be available at http://localhost:8000. Interactive docs at http://localhost:8000/docs.

Docker (full stack)

docker build -t digimon-rag-assistant .
docker run -p 8000:8000 --env-file .env digimon-rag-assistant

API Endpoints

Chat with RAG

POST /api/v1/chat
Content-Type: application/json

{
  "query": "What is Agumon?",
  "level_filter": "Rookie",
  "type_filter": "Reptile"
}

PydanticAI Agent (tool-calling)

POST /api/v1/agent
Content-Type: application/json

{"query": "What are the skills of Agumon?"}
# → agent selects get_digimon_by_name + get_digimon_skills automatically

{"query": "Which Rookie Digimon are best for beginners?"}
# → agent selects rag_search + get_digimon_by_level automatically

Response includes tool_calls (list of tools used) and token usage.

LangGraph Corrective RAG

POST /api/v1/agent/graph
Content-Type: application/json

{"query": "Tell me about fire-type Rookie Digimon"}

Response includes iterations (0 = direct, 1+ = query was rewritten) and documents_used.

Get Digimon by ID

GET /api/v1/digimon/{id}

List Digimons

GET /api/v1/digimon?limit=10&offset=0

Health Check

GET /api/v1/health

Data Ingestion

The ingestion pipeline fetches data from DAPI:

1. Fetches Digimon data (respects INGEST_LIMIT in .env)
2. Prepares text chunks per Digimon (basic info, description, skills, evolutions)
3. Generates vector embeddings with Voyage AI (voyage-3, 1024 dims) in rate-limited batches
4. Stores embeddings in Qdrant and metadata in PostgreSQL

Key .env variables for ingestion:

Variable	Default	Description
INGEST_LIMIT	100	Max Digimon to ingest (0 = all ~1450)
VOYAGE_EMBED_BATCH_SIZE	20	Chunks per embedding API call
VOYAGE_RPM	3	Voyage AI requests per minute (free tier = 3)

To re-run ingestion:

make ingest

Project Structure

knowledge-rag-assistant/
├── app/
│   ├── api/              # FastAPI routes and app setup
│   │   ├── routes/
│   │   │   ├── chat.py       # /chat — Custom RAG + Redis caching
│   │   │   ├── agent.py      # /agent — PydanticAI + /agent/graph — LangGraph
│   │   │   ├── digimon.py    # Digimon metadata endpoints
│   │   │   └── health.py     # Health check
│   │   └── main.py
│   ├── core/             # Config (pydantic-settings) and logging
│   ├── ingestion/        # DAPI client, embedder, ingest pipeline
│   ├── rag/              # Retriever, prompt builder, LLM client
│   ├── guardrails/       # InputGuard — LLM-based topic classifier (fail-open)
│   ├── mcp/              # MCP server with 4 Digimon tools
│   ├── agents/
│   │   ├── pydantic_agent.py # PydanticAI agent with 4 typed tools
│   │   └── graph/            # LangGraph Corrective RAG
│   │       ├── state.py      # GraphState TypedDict
│   │       ├── nodes.py      # retrieve / grade / generate / rewrite nodes
│   │       └── workflow.py   # Compiled StateGraph
│   ├── observability/    # Langfuse tracing integration
│   └── db/               # Qdrant, PostgreSQL, Redis managers
├── docker/
│   └── docker-compose.yml
├── tests/
├── requirements.txt
├── Dockerfile
├── CONTRIBUTING.md
└── README.md

Technical Highlights

This project demonstrates key skills for LLM Backend Developer roles:

1. Custom RAG without LangChain: Built the full pipeline (retrieval → prompt → generation) from first principles — shows deep understanding of how RAG works, not just framework usage.

2. Three agent patterns side by side: Custom RAG (/chat), PydanticAI typed agent (/agent), and LangGraph Corrective RAG (/agent/graph) — each demonstrates a different architectural trade-off.

3. PydanticAI typed tools: Type-safe agent with @agent.tool_plain decorators, AnthropicModel, and automatic tool selection. Zero boilerplate.

4. LangGraph Corrective RAG: Stateful retrieve → grade → generate loop with query rewriting when retrieved docs aren't relevant — demonstrates iterative reasoning beyond single-shot generation.

5. MCP (Model Context Protocol): Implements the emerging standard for tool integration between LLMs and external systems.

6. Production Observability: Full tracing with Langfuse — numeric scores per trace (latency, tokens, retrieval quality) plus async LLM-as-judge evaluation (faithfulness, relevance, context relevance) as a background task so it never adds latency to the API response.

7. Cost Optimization: Redis caching avoids redundant LLM calls for identical queries, with proper cache-key design to prevent collisions across different filter combinations.

8. Input Guardrails: LLM-based classifier blocks off-topic queries before they reach the RAG pipeline. Fail-open design ensures a guardrail outage never takes down the main service.

9. RAGAS Evaluation: Industry-standard RAG evaluation with 4 metrics (faithfulness, answer relevancy, context precision, context recall) against a 10-question golden dataset — measures actual pipeline quality, not just test pass/fail.

Roadmap

• Custom RAG pipeline (Retriever → PromptBuilder → LLMClient)
• MCP server with 4 tools
• Anthropic Claude Haiku + Voyage AI voyage-3 integration
• Langfuse observability
• Redis caching with proper cache-key design
• Docker infrastructure
• PydanticAI typed agent with automatic tool selection
• LangGraph Corrective RAG with query rewriting
• Comprehensive test suite with pytest (mocked, no external services required)
• RAG evaluation pipeline with RAGAS (faithfulness, relevancy, precision, recall)
• Input guardrails to block off-topic queries (fail-open design)
• LLM-as-judge scoring in Langfuse (faithfulness, relevance, context relevance — async background task)
• Streaming SSE endpoint with real-time token delivery
• Conversation history with Redis (multi-turn, last 5 turns, 1h TTL)
• Rate limiting and authentication

Running Tests

Tests are fully mocked — no external services required:

# Activate venv
source venv/bin/activate

# Run all unit tests (test_ingestion.py excluded by default — needs live services)
make test
# or: pytest tests/ -v --ignore=tests/test_ingestion.py

# With coverage report
pytest tests/ -v --ignore=tests/test_ingestion.py --cov=app --cov-report=term-missing

# Integration tests (requires make services + valid API keys)
pytest -m integration

Test suite covers: PromptBuilder, LLMClient, Retriever, /chat endpoint (including cache hit), and InputGuard guardrails.

tests/test_ingestion.py is marked @pytest.mark.integration and excluded from the default run.

RAG Evaluation

Evaluates the pipeline against a 10-question golden dataset using RAGAS metrics:

Metric	Measures
faithfulness	Is the answer grounded in the retrieved context?
answer_relevancy	Is the answer relevant to the question?
context_precision	Are the retrieved chunks precise (not noisy)?
context_recall	Do the chunks cover the ground-truth answer?

# Requires: make services + make ingest (Qdrant must have data)
source venv/bin/activate
python scripts/evaluate_rag.py            # build dataset + evaluate (~60 min total)
python scripts/evaluate_rag.py --eval-only  # reuse cached dataset, skip rebuild (~55 min)

Output: per-question scores table + overall averages. Faithfulness < 0.7 indicates a prompt issue; context recall < 0.6 indicates an ingestion or embedding issue.

Baseline scores on default 100-Digimon ingest:

Metric	Score	Threshold
faithfulness	1.000	≥ 0.70 ✓
answer_relevancy	0.848	≥ 0.70 ✓
context_precision	0.883	≥ 0.70 ✓
context_recall	1.000	≥ 0.60 ✓

All four metrics pass threshold. answer_relevancy originally scored 0.558 — two compounding root causes were found and fixed:

1. Prompt scope discipline: the system prompt didn't constrain answers to what was actually asked, so narrow questions ("What level is X?") got full Digimon profiles back. RAGAS's ResponseRelevancy reconstructs questions from the answer and compares them to the original — a bloated answer reconstructs into a generic question that doesn't match a narrow original. Fixed by instructing the model to answer only the requested aspect unless the question is an open request ("tell me about X").
2. Evaluation harness bug: RAGAS's ResponseRelevancy embeds the original question with embed_query() and the reconstructed questions with embed_documents(). Voyage-3 is an asymmetric retrieval model — input_type="query" and input_type="document" land in different subspaces — so comparing a question against other questions this way artificially compressed the cosine similarity (verified directly: two near-identical paraphrases scored only 0.59 under the asymmetric call vs. 0.96 with symmetric embeddings). Fixed in scripts/evaluate_rag.py by embedding both sides with input_type=None (Voyage's general-purpose symmetric mode) for this metric only — production retrieval is untouched and still correctly uses the asymmetric query/document embeddings it was designed for.

context_precision (0.883) has two individually low-scoring questions (~0.33–0.5) despite retrieving the correct chunk every time — RAGAS's precision metric is rank-sensitive, and for a couple of narrow single-fact questions the info-bearing chunk is outranked by that same Digimon's skills or evolution chunk. context_recall stays perfect because it only checks presence, not rank.

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT — see LICENSE for details.