llm-dev-kit

LLM Dev Kit

A private AI workspace for your team. Chat with your documents on your own hardware.

Ask questions in plain language and get answers grounded in your own PDFs, from models that run on your machine. Bring in your team with accounts, roles and single sign-on. Connect your scripts, notebooks and agents through an OpenAI-compatible API and MCP.

Quick start · Tour · Features · Use it from code · Architecture · Docs

---

Why LLM Dev Kit

• Private by default. Chat and embeddings run on Ollama on your own machine. Nothing leaves it unless you choose a cloud model.
• Answers you can check. Upload PDFs and every answer is grounded in the passages it retrieves from them. When the documents do not cover a question, the assistant says so rather than inventing an answer.
• Ready for a team. Accounts, roles (admin, user, viewer), personal API keys, optional Keycloak single sign-on, and an admin page for users and service health.
• Open at every seam. An OpenAI-compatible /v1 API works with any OpenAI SDK and with LangChain. An MCP server plugs into Claude Desktop and other agents. Model providers, the vector store, loaders and chunkers are plugins you can swap.

Quick start

You need Docker with Compose v2 and Ollama running on the host.

1. Pull the models (once, about 5 GB):

ollama pull llama3.1
ollama pull nomic-embed-text

2. Configure and start the stack:

git clone https://github.com/sameeralam3127/llm-dev-kit.git
cd llm-dev-kit

# Inside Docker, Ollama is reached through host.docker.internal, so drop the
# localhost setting meant for running the services outside Docker.
grep -v '^OLLAMA_HOST=' .env.example > .env
echo "AUTH_SECRET=$(openssl rand -base64 32)" >> .env

docker compose up --build

3. Open http://localhost:8080 and create an account. The first account becomes the admin.

4. Add a document and ask about it. Create an API key from the account menu (bottom left) → API keys, then:

KEY=sk-ldk-...   # the key you just created
curl -H "Authorization: Bearer $KEY" -F "file=@handbook.pdf" \
  http://localhost:8080/api/rag/ingest/pdf
# {"chunks":12}

Ask a question about the document in the chat. The answer comes from the handbook, generated by llama3.1 on your machine.

[!TIP] Everything listens on 127.0.0.1:8080 only, so nothing is exposed until you decide. docs/usage.md covers inviting a team on your network, and docs/setup.md walks through every step in more detail.

A quick tour

Features

Chat

• Streaming answers with a stop button. A partial answer is saved even if you stop it or the model host fails mid-sentence.
• Edit and regenerate. Editing a message rewinds the conversation to that point. Earlier answers stay one click away through ‹ 2/3 ›.
• Organised history. Chats grouped by date, folders, pinning, and a sidebar that collapses to an icon rail.
• Keyboard first. ⌘K palette, ⇧⌘O new chat, ⌘B sidebar.
• Readable output. GitHub-flavoured Markdown, syntax highlighting loaded on demand, per-block copy.
• Sharing. Publish a chat as a read-only link and revoke it at any time.
• Accessible. Light, dark and system themes with WCAG AA contrast, full keyboard support, and motion that stops when your system asks for reduced motion.

Knowledge

• Retrieval-augmented answers over your PDFs: text is chunked, embedded locally with nomic-embed-text and stored in ChromaDB.
• Any model. Local Ollama models by default. Add a key in .env and openai/…, gemini/… and anthropic/… models appear in the model picker, through LiteLLM.
• Cached answers. Repeated questions to /api/rag/chat come back from Redis instantly.

Identity and access

• Every API call is authenticated. The browser uses its session; scripts and agents use personal API keys (sk-ldk-…, stored hashed, shown once).
• Roles: viewer chats, user also adds documents, admin also manages people and clears the index.
• Single sign-on: docker compose --profile sso up adds Keycloak with a ready-made realm, served at /sso.
• No account takeover by email. An external sign-in whose email already has an account is refused, not silently linked.
• Short-lived credentials. The services verify 5-minute signed tokens themselves, so the gateway is never trusted blindly.

Operations

• One command, one port. docker compose up starts everything behind an Nginx gateway on 127.0.0.1:8080; no other port is published.
• Scales out. rag-service runs two replicas by default; add more with --scale and the gateway picks them up without a restart.
• Observable. Structured JSON logs, one request id that follows a call across every service, and one error format everywhere.
• Gated releases. CI lints, type-checks, tests and starts every image before anything is published.

Use it from code

The /v1 API speaks the OpenAI protocol, so existing clients work by changing two settings. Every answer is grounded in your documents.

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-ldk-...")
reply = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "When does BLUEFIN-42 ship?"}],
)
print(reply.choices[0].message.content)

LangChain

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(base_url="http://localhost:8080/v1", api_key="sk-ldk-...", model="llama3.1")
print(llm.invoke("Summarise the onboarding handbook").content)

curl, with streaming

curl -N http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"llama3.1","stream":true,"messages":[{"role":"user","content":"Hello"}]}'

MCP (Claude Desktop and other agents). Add mcp.json to your client with LDK_API_KEY set to an API key. It provides ask_llm_dev_kit and list_models.

All paths go through the gateway at http://localhost:8080. Everything except the health checks needs Authorization: Bearer <api key>.

Method and path	Role	What it does
GET /api/rag/health, GET /api/llm/health	none	Liveness; llm-service also reports whether Ollama is ready
GET /v1/models	viewer	Models, in OpenAI format
POST /v1/chat/completions	viewer	Grounded chat, streaming or not, in OpenAI format
POST /api/rag/chat	viewer	Grounded chat with its sources; "api_key" in the body uses your own cloud key for that call
POST /api/rag/chat/stream	viewer	The same, streamed as NDJSON (meta, delta, done)
GET /api/llm/models	viewer	Local and configured cloud models
POST /api/rag/ingest/pdf	user	Add a PDF (multipart field file, up to 64 MB)
GET /api/rag/documents/stats, POST /api/rag/documents/clear	admin	Count or clear the index
GET /api/rag/cache/stats, POST /api/rag/cache/clear	admin	Inspect or clear cached answers

Errors share one shape, {"error": {"code": "...", "message": "..."}}, with the status implied by the code: invalid_request 400, unauthorized 401, forbidden 403, not_found 404, conflict 409, rate_limited 429, internal 500, upstream_error 502, upstream_unavailable 503, timeout 504. Every response carries an X-Request-ID; send your own to find a call in the logs.

How it works

flowchart LR
  B[Browser] --> GW
  S[Scripts and SDKs] -->|API key| GW
  A[MCP agents] --> MCP[mcp-service]
  GW[Nginx gateway<br/>127.0.0.1:8080] --> WEB[web<br/>Next.js app and identity]
  GW -->|/v1, /api/rag| RAG[rag-service ×2<br/>retrieval and chat]
  GW -->|/api/llm| LLM[llm-service<br/>model routing]
  WEB -->|user's token| RAG
  MCP -->|API key| RAG
  RAG -->|caller's token| LLM
  RAG -.->|verify tokens| WEB
  LLM -.->|verify tokens| WEB
  RAG --> CH[(ChromaDB)]
  RAG --> RD[(Redis)]
  LLM --> OL[Ollama on the host]
  LLM -.->|optional| CL[Cloud models]

Service	What it does
nginx	The gateway and the only published port. Routes to the app and APIs and balances across rag-service replicas.
web	The Next.js chat app. It is also the identity service: it signs short-lived tokens and publishes their keys at /.well-known/jwks.json.
rag-service	Retrieval, grounded chat, the /v1 API, PDF ingestion and the answer cache. Two replicas by default.
llm-service	One interface to every model: Ollama locally, cloud providers through LiteLLM.
mcp	An MCP server over stdio for agents (profile mcp).
redis, chroma	Cache and vector store, reachable only inside the Compose network.
keycloak	Optional single sign-on (profile sso).

The Python services never trust the gateway: each verifies the caller's token itself. ARCHITECTURE.md has the request flows and the Docker build strategy.

Configuration

Settings live in .env; .env.example documents every one. The ones you are most likely to change:

Setting	Default	What it is for
AUTH_SECRET	none, required	Signs sessions and encrypts the token signing keys. openssl rand -base64 32
WEB_PUBLIC_URL	http://localhost:8080	The address people type into their browser
GATEWAY_BIND	127.0.0.1	Set 0.0.0.0 to let other machines on your network in
ALLOW_REGISTRATION	true	Set false once your team has signed up
DEFAULT_CHAT_MODEL	llama3.1	Any model you have pulled in Ollama
OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY	unset	Enable those providers' models for everyone
KEYCLOAK_ADMIN_PASSWORD, AUTH_KEYCLOAK_*	unset	Single sign-on; see the SSO section of .env.example
ALLOW_LEGACY_ANONYMOUS	false	Temporary escape hatch for clients without keys; see MIGRATION.md

Want Ollama in Docker instead of on the host? Run docker compose --profile ollama up and set OLLAMA_HOST=http://ollama:11434. It runs on CPU only on macOS.

Extend it with plugins

Model providers, embedders, the vector store, document loaders, chunkers, authentication and MCP tools are plugins behind typed interfaces in ldk_core.ports. A plugin is a small Python package with a manifest and an entry point:

[project.entry-points."llm_dev_kit.plugins"]
"llm.my-host" = "my_plugin:plugin"     # <kind>.<name>

Each interface ships a contract test suite. Your plugin passes the same tests as the built-in ones in services/plugins/:

from ldk_core.testing.contracts import LLMProviderContract

class TestMyHost(LLMProviderContract):
    @pytest.fixture
    def provider(self):
        return MyHostProvider(...)

    @pytest.fixture
    def model(self):
        return "my-model"

Plugins run in-process with the service's permissions. Their manifests declare what they do, but that is disclosure, not a sandbox.

Development

Python services

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements-dev.txt       # ldk_core and every plugin, editable
ruff check services tests && ruff format --check services tests
mypy
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest

Web app (Node 22.12+)

cd web && npm install
cp .env.example .env                      # then set AUTH_SECRET
npm run typecheck && npm run lint && npm test && npm run build
npm run dev                               # http://localhost:3000

Hot reload in Docker: docker compose -f docker-compose.yml -f compose.dev.yml up --build.

CI runs all of the above on every push and pull request, then builds and starts each image. Pull request titles follow Conventional Commits (feat:, fix:, docs: …).

web/                     Next.js app: chat, accounts, admin, identity service
  src/server/identity/     roles, signing keys, access tokens, API keys
  src/server/services/     every database write and ownership check
services/
  core/                    ldk_core: interfaces, plugin registry, auth, errors, logging
  plugins/                 ollama, litellm, chroma, loader_pdf, chunker_fixed, tools, auth
  llm_service/             model routing
  rag_service/             retrieval, grounded chat, /v1, ingestion, cache
  mcp_service/             MCP server
nginx/nginx.conf         the gateway
deploy/keycloak/         the realm the sso profile imports
docs/                    setup, usage, the v2 plan and migration notes
tests/                   Python unit, contract and service tests

Project status

LLM Dev Kit is becoming a sovereign, local-first AI workspace. v2 is in active development and not yet released. The plan of record is docs/v2/PLAN.md, tracked in issue #20.

	Phase
✅	Plan	Architecture, interfaces and phases
✅	Core refactor	Interfaces, plugin registry, contract tests, CI gate
✅	Authentication	Tokens, API keys, roles, SSO. Breaking for /v1: see MIGRATION.md
🚧	Admin	Users, roles and health are done; usage, audit and plugin settings follow
⏳	Sovereign mode	No outbound traffic by default, backup and wipe
⏳	Per-user documents	Private collections on pgvector, upload in the app, citations
⏳	Limits and observability	Rate limits, tracing, dashboards, audit log
⏳	Plugin marketplace	Signed plugins installed without rebuilding
⏳	Distribution	Multi-arch images, one-file install

Good to know today: documents are uploaded through the API and shared by every user until per-user collections arrive. Models whose names end in -cloud run on ollama.com, not on your machine.

Troubleshooting

Symptom	Fix
required variable AUTH_SECRET is missing	Add one: echo "AUTH_SECRET=$(openssl rand -base64 32)" >> .env
"No local models" in the model picker	Check ollama list, and that .env has no OLLAMA_HOST=http://localhost:11434 line
401 from the API	Send Authorization: Bearer sk-ldk-…; a revoked key stops working within a minute
403 from the API	Your role cannot do that: only user and admin add documents
Nothing at localhost:8080	docker compose ps; if services show Exited, run docker compose up -d

More in docs/setup.md.

Documentation

Guide	Read it to
docs/setup.md	Install step by step and fix common problems
docs/usage.md	Add documents, chat, and bring in your team
docs/v2/MIGRATION.md	Upgrade clients to API keys
ARCHITECTURE.md	Understand request flows and the Docker build
docs/v2/PLAN.md	See where the project is going
web/README.md	Work on the web app

License

MIT © 2025 Sameer Alam