{"id":"71fb2e70-c7b5-4733-ab10-0009cbc9e178","slug":"minics","kind":"skill","name":"minics","summary":"Your small model dataset platform for building datasets, mixed preview of ChatML Studio and Tyness","owner":"jasonjimnz","repoUrl":"https://github.com/jasonjimnz/minics","license":"MIT","licenseTier":"permissive","trust":"community","stars":2,"match":null,"fewStars":true,"format":"skill","createdAt":"2026-09-23T18:27:19.000Z","addedAt":"2026-10-22T01:54:56.455Z","logoUrl":"https://avatars.githubusercontent.com/u/6759345?u=d4b4ef7daae5b1d74e5e842ae6bf22a1da2afdbf&v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["writing"],"tags":[],"runtime":"python","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-22T01:55:27.514Z","sample":false,"url":"https://conduits.dev/tools/minics","language":"HTML","transports":["stdio"],"install":{"pypi":"minichat-studio"},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# MiniCS Lite\n\n**Mini ChatML Studio Lite** — a local-first studio for building high quality\nLLM datasets. MiniCS combines a ChatML-aware dataset library, a hybrid RAG +\nknowledge-graph grounding engine, an LLM authoring toolbox and a markdown\neditor into a single desktop application (Flask + PyWebView). Everything lives\ninside your own `~/.minics` folder — no cloud, no accounts, no lock-in.\n\n> **MiniCS is a mixed project.** It combines two sibling codebases: **ChatML\n> Studio**, a platform for handling datasets and entries, and **Tyness**, a\n> *tiny documental harness* for enhancing LLM capabilities with embed\n> databases. MiniCS Lite distills both into one small, local-first tool — the\n> root projects will keep growing with more capabilities.\n>\n> **MiniCS is OpenAI-protocol first.** Any OpenAI-compatible endpoint works:\n> OpenAI, Azure-style gateways, vLLM, llama.cpp (`llama-server`), LM Studio,\n> Ollama's OpenAI shim, or anything else speaking `/v1/chat/completions` and\n> `/v1/embeddings`.\n\n---\n\n## Highlights\n\n| Area | What you get |\n| --- | --- |\n| **Dataset library** | Datasets, ChatML entries with system/user/assistant roles, draft → review → approved workflow, automatic versioning on every message edit, structural ChatML validation. |\n| **Grounding engine** | Import **PDF, DOCX, TXT, Markdown, LaTeX** — every file is converted to cleaned markdown (optionally repaired by the LLM), then chunked and indexed. |\n| **Hybrid retrieval** | ChromaDB vector search **plus** Ladybug graph topology, fused with Reciprocal Rank Fusion (RRF). Each side degrades gracefully when the other is unavailable. |\n| **Agent skillset** | A complete, precise skillset ships **inside the package** — one parent skill plus ten sub-skills that teach agents (Pi, OpenCode, Codex…) how to drive MiniCS. `minics skills install` drops them into `.agents/skills` locally or `~/.agents/skills` globally. |\n| **LLM everywhere** | Enhance fields, suggest tags, evaluate entries against 6 quality dimensions, generate grounded new entries, clean markdown, extract graph entities. Structured output with 3 fallback strategies. |\n| **Background-first** | Long operations (import, indexing, evaluation) run on a worker thread pool with live progress streamed to the UI over **Server-Sent Events**, backed by a concurrency-safe SQLite writer thread. |\n| **Chat with switches** | Per-conversation toggles for RAG, graph and grounding — retrieval and injection are separate, so you can *preview* what would be used without forcing it into the prompt. |\n| **Export anywhere** | ChatML, JSONL, Alpaca, ShareGPT and Markdown — byte-identical from UI, CLI or Python. |\n| **Themable UI** | 15 hand-tuned themes — System, Light, Dark, Ocean, Forest, Sunset, Nord, Rose, Midnight, Paper, Mono, Grape, Dracula, Catppuccin Mocha and Solarized — with native dark form controls, tinted shadows and themed scrollbars. Themes are pure CSS-variable swaps. |\n\n---\n\n## A walkthrough of the web app\n\nThe screenshots below (from [`docs/screenshots/`](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/)) follow a\nreal end-to-end session against two local model servers: a **llama.cpp\n`llama-server`** instance serving the chat model and an **Ollama** instance\nserving the embedding model — exactly the split-endpoint setup MiniCS is\ndesigned for.\n\n### 1. Home / Dashboard\n\n![Dashboard](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/01_home.png)\n\nThe landing screen. Live counters for datasets, entries, approved entries,\ndocuments, indexed chunks and graph entities, a *Getting started* checklist\nthat ticks itself off as you complete each step, and a recent-activity feed.\nEverything in the sidebar — Datasets, Entries, Collections, Documents, Chat,\nGraph, Activity, Settings — is one click away.\n\n### 2. Settings — wiring up the endpoints\n\n![Settings](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/02_settings.png)\n\nHere the app is pointed at the two local servers: the **LLM endpoint** is a\n`llama-server` instance (`qwen-3-8` on `:8000/v1`) and the **embeddings\nendpoint** is Ollama (`bge-m3:latest` on `:11434/v1`, 1024 dimensions — hit\n*Detect* to sniff it automatically, since it drives the Chroma vector store).\nThe Retrieval panel tunes the hybrid search defaults (Top K, RRF k, graph\nhops), and the Application panel picks one of the 15 themes (Catppuccin Mocha\nhere) and the background-worker pool size. *Test connection* validates both\nendpoints before *Finish setup* locks them in.\n\n### 3. Creating a dataset\n\n![New dataset](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/03_new_dataset.png)\n\nDatasets are the top-level containers for training examples. Give one a name,\na short description and optional comma-separated tags — that's all it takes.\n\n### 4. Creating an entry\n\n![New entry](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/04_new_entry.png)\n\nEach entry is a ChatML conversation: **system**, **user** and **assistant**\nmessages. Fill them by hand, or type a topic into *Generate with LLM\n(grounded)* and let the model draft the whole entry using your imported\ndocuments as context.\n\n### 5. The entry editor\n\n![Entry editor](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/05_created_entry.png)\n\nThe generated entry lands in the editor. Messages can be reordered, deleted\nand switched between roles; a *Valid ChatML* banner confirms the structure in\nreal time. The LLM helpers are one click away — *Enhance assistant*, *Suggest\ntags* (in action here) and *Evaluate* against the six quality dimensions —\nand every message edit bumps the entry version automatically.\n\n### 6. Importing a markdown document\n\n![Document viewer](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/06_adding_document.png)\n\nDropping a markdown file (or PDF/DOCX/TXT/LaTeX — it gets converted) opens the\ndocument viewer: a split-pane markdown editor with live preview and formatting\ntoolbar. This one has been chunked into **127 chunks**, ready to index. From\nhere you can *Clean with LLM*, *Enrich graph* and *Reindex* without leaving\nthe dialog.\n\n### 7. Querying the knowledge graph\n\n![Graph](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/07_searching_graph.png)\n\nThe Graph screen shows the entity topology extracted from your documents —\n923 entities, 1492 mentions and 4779 co-occurrences from a single import.\nType a query (`Sorting` here) and hit *Explore* to render the matching\nsub-graph; *Rebuild* regenerates the whole topology from the index.\n\n### 8. Testing the LLM\n\n![Chat](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/08_chat_with_model.png)\n\nChat is where the model gets smoke-tested. Each conversation carries its own\n**RAG**, **Graph** and **Grounding** toggles — you can preview what retrieval\n*would* inject without forcing it into the prompt. This call is a plain\nsanity-check prompt with all toggles off.\n\n### 9. Creating a collection\n\n![New collection](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/09_new_collection.png)\n\nCollections are curated sets of entries destined for export — a name and a\ndescription and you're in.\n\n### 10. Curating and exporting\n\n![Collection](https://raw.githubusercontent.com/jasonjimnz/minics/main/docs/screenshots/10_adding_entries.png)\n\nSearch your entries and *Add* them to the collection, then download the whole\nset in one click — **ChatML, JSONL, Alpaca, ShareGPT or Markdown** —\nbyte-identical whether exported from the UI, the CLI or Python.\n\nThat's the full loop: **configure → create → author → import → ground → chat\n→ curate → export.**\n\n---\n\n## Quick start\n\nThe package is **live on PyPI**, so the fastest way in is simply:\n\n```bash\npip install minichat-studio      # from PyPI: https://pypi.org/project/minichat-studio/\n\nminics init                      # create ~/.minics\nminics setup                     # interactive LLM + embedding wizard\nminics scan ./my-docs            # bulk-import every compatible document in a tree\nminics skills install            # install the agent skillset into ./.agents/skills\nminics                           # launch the desktop app (default command)\n```\n\nOr run it **with Docker** — a prebuilt server image is published on GHCR, no\nlocal build required:\n\n```bash\ndocker pull ghcr.io/jasonjimnz/minics:latest\ndocker run -d -p 8765:8765 -v minics-data:/data ghcr.io/jasonjimnz/minics:latest\n```\n\nFor development, install from a clone instead:\n\n```bash\ngit clone https://github.com/jasonjimnz/minics.git\ncd minics\npip install -e \".[dev]\"\n```\n\nPoint the wizard at any OpenAI-compatible chat endpoint and any embedding\nendpoint (they may be different servers). The embedding dimension is detected\nautomatically and required — Chroma collections are created with it.\n\n### Example notebook: MiniCS + vLLM on Google Colab\n\nPrefer to see the library in action before installing anything? The\n[`examples/colab-vllm`](https://github.com/jasonjimnz/minics/tree/main/examples/colab-vllm)\nnotebook runs the whole MiniCS library programmatically on a **free Colab T4\nGPU** — vLLM serves a small chat LLM and an embedding model in the\nbackground, and the notebook builds a knowledge base end to end (documents →\nhybrid retrieval → grounded chat → dataset authoring → JSONL export). See\n[docs/colab-example.md](docs/colab-example.md).\n\nOther entry points:\n\n```bash\nminics serve --port 8800        # Flask only, opened in a browser\nminics models                   # list models from the endpoint\nminics scan ./docs --pdf --txt  # bulk-import a directory (one-by-one, live progress)\nminics scan ./docs --dry-run    # preview what a scan would import\nminics search \"what is RRF?\"    # hybrid retrieval from the terminal\nminics export <collection> -o out.jsonl --format jsonl\nminics documents --import paper.pdf --index\nminics skills install --global  # agent skills into ~/.agents/skills\nminics reindex                  # rebuild the vector index + graph\npython -m minics --help         # everything else\n```\n\n### Run as a server (Docker)\n\nA prebuilt server image is published on **GHCR** for every release — pull it\ndirectly instead of building:\n\n```bash\ndocker pull ghcr.io/jasonjimnz/minics:0.4.0   # or :latest\n```\n\nThe container keeps the whole store in a volume and\nauto-creates it on first start — no setup steps on the host:\n\n```bash\ndocker run -d -p 8765:8765 -v minics-data:/data ghcr.io/jasonjimnz/minics:latest\ndocker compose up -d --build   # or build locally: docker build -t minics-lite .\n```\n\nOpen `http://localhost:8765`, finish setup in the UI. Details in\n[docs/docker.md](docs/docker.md).\n\n### Use it as a library\n\nMiniCS is a library first; the Flask API and the desktop shell are thin layers\non top of a single application context (`minics.services.context.AppContext`).\n\n```python\nfrom minics.services.context import get_context\n\nctx = get_context()\ndataset = ctx.datasets.create(\"Support QA\", tags=[\"support\"])\nentry = ctx.entries.create(dataset.id, user=\"What is RRF?\", assistant=\"A fusion method.\")\nctx.indexer.index_document(ctx.documents.import_file(\"handbook.pdf\").id)\nresult = ctx.retriever.retrieve(\"how does fusion work?\")\nprint(result.context_text())\n```\n\n---\n\n## What's inside\n\n```\n~/.minics/\n├── config.json        # LLM / embedding / retrieval / app settings\n├── minics.sqlite3     # datasets, entries, collections, documents, chats (WAL)\n├── vectordb/          # ChromaDB persistent store (cosine space)\n├── graphdb/           # Ladybug graph database (minics.graph)\n├── documents/\n│   ├── originals/     # untouched uploads, kept for provenance/re-extraction\n│   └── markdown/      # cleaned markdown used for chunking + grounding\n├── logs/\n└── cache/\n```\n\nOverride the location with the `MINICS_HOME` environment variable (used by the\ntest suite to run against throwaway homes).\n\n---\n\n## Architecture\n\n```\nCLI / PyWebView ──► Flask API ──► services (library API) ──┬─► SQLite (single writer thread)\n                                                           ├─► ChromaDB (vectors)\n                                                           ├─► Ladybug (graph)\n                                                           └─► OpenAI-compatible LLM + embeddings\n```\n\n- **`minics.core`** — paths, config, SQLite (WAL + serialised writer thread),\n  in-process event bus, background job manager, generic pydantic repository.\n- **`minics.domain`** — pydantic entities (Dataset, Entry, Collection, Document,\n  Conversation, …). Storage-agnostic: serialised to rows *and* to JSON.\n- **`minics.llm`** — OpenAI/LangChain client factory, embeddings, structured\n  output with fallbacks, all authoring helpers and prompts.\n- **`minics.rag`** — markdown-aware chunking, heuristic entity extraction,\n  Chroma store, Ladybug graph, RRF fusion, hybrid retriever.\n- **`minics.documents`** — PDF/DOCX/TXT/Markdown/LaTeX → clean markdown.\n- **`minics.services`** — the library API: datasets, entries, collections,\n  documents, authoring, chat, export.\n- **`minics.server`** — Flask app factory, REST API blueprints, SSE event\n  stream, and the no-build static SPA.\n- **`minics.desktop` / `minics.cli`** — the PyWebView desktop shell and the CLI.\n\nEvery layer is documented in depth in [`docs/`](docs/index.md):\n\n| Document | Contents |\n| --- | --- |\n| [Documentation index](docs/index.md) | Reading order and full map of the docs. |\n| [Architecture](docs/architecture.md) | Layers, request lifecycle, threading model, design decisions. |\n| [Libraries](docs/libraries.md) | Every dependency, why it's there and where it's used. |\n| [Configuration](docs/configuration.md) | Every config key, the setup wizard, `MINICS_HOME`, secrets masking. |\n| [CLI reference](docs/cli.md) | Every `minics` subcommand — setup, `minics scan`, documents, search, export, skills. |\n| [Agent skills](docs/skills.md) | The bundled MiniCS skillset for Pi, OpenCode and Codex: installation and the full catalogue. |\n| [Core & backend](docs/backend.md) | SQLite writer, event bus, job manager, repositories, services, CLI, desktop shell. |\n| [LLM layer](docs/llm.md) | OpenAI protocol client, embeddings, structured output + truncation handling, prompts, authoring helpers. |\n| [RAG & graph](docs/rag.md) | Conversion, chunking, entity extraction, Ladybug schema, Chroma, indexing pipeline, RRF hybrid retrieval. |\n| [HTTP API](docs/api.md) | Full REST reference, SSE event stream, job workflow, error mapping. |\n| [Docker](docs/docker.md) | Server container: build, run, Compose, volumes, first-run setup, troubleshooting. |\n| [Frontend](docs/frontend.md) | The no-build SPA: modules, hash routing, SSE, themes, markdown, graph view, every screen. |\n| [Development](docs/development.md) | Environment, tests, fixtures, style, migrations, how to contribute. |\n| [Colab + vLLM example](docs/colab-example.md) | Run the whole MiniCS library programmatically on a free Google Colab T4 GPU with vLLM serving the models — the `examples/colab-vllm` notebook. |\n\n---\n\n## The 60-second tour\n\n1. **Settings** → point MiniCS at your chat + embedding endpoints, hit\n   *Test connection*, *Finish setup*.\n2. **Documents** → drop a PDF/DOCX/TXT/MD/TeX file. It is converted to markdown,\n   cleaned (heuristically, plus the LLM if enabled), chunked along headings,\n   embedded into Chroma, and wired into the entity graph.\n3. **Datasets / Entries** → author ChatML entries by hand or generate them from\n   a topic grounded in your documents. Enhance, tag and evaluate with the LLM;\n   every message edit bumps the entry version.\n4. **Collections** → curate approved entries and export to ChatML / JSONL /\n   Alpaca / ShareGPT / Markdown.\n5. **Chat** → talk to your model. Toggle RAG / graph / grounding per\n   conversation; answers cite the chunks they used as `[n]` references.\n6. **Graph** → explore the entity topology extracted from your documents.\n7. **Activity** → every long operation is a background job with live progress,\n   cancellable at any time.\n\n---\n\n## Development\n\n```bash\npip install -e \".[dev]\"\npytest -q          # tests (no model server required — embeddings are faked)\nruff check src tests\n```\n\nThe test suite runs against an isolated `MINICS_HOME` with deterministic fake\nembeddings, so it needs no network access. See\n[docs/development.md](docs/development.md).\n\n## Contributing\n\nContributions are welcome! MiniCS is a small project maintained in tiny,\nversioned increments, so keep pull requests focused. Contributors must follow\nthe project's core principles:\n\n- **Runs everywhere.** The application must work on any OS — Linux, macOS and\n  Windows — using **Python 3.12 or newer**. Avoid OS-specific code paths\n  unless they are guarded and tested.\n- **Local-first.** No cloud, no accounts, no telemetry; everything stays in\n  `~/.minics`.\n- **OpenAI-protocol only.** Talk to models exclusively through the\n  OpenAI-compatible HTTP protocol.\n- **Lite by design.** MiniCS Lite is the compact edition: the root projects\n  (ChatML Studio and Tyness) are the ones that will grow bigger and gain more\n  capabilities over time.\n\nSee [docs/development.md](docs/development.md) for environment setup, tests\nand style conventions.\n\n## Status\n\n**Version 0.4.0 is ready.** The project is still a **beta** and intentionally\na **small project** — a lite blend of ChatML Studio and Tyness — built in\nsmall, versioned increments (see `git log`).\n\n| Artifact | Where |\n| --- | --- |\n| Python package | [PyPI — `minichat-studio`](https://pypi.org/project/minichat-studio/) |\n| Server image | [GHCR — `ghcr.io/jasonjimnz/minics`](https://github.com/jasonjimnz/minics/pkgs/container/minics) |\n\n## License\n\nMIT — see [pyproject.toml](pyproject.toml).\n"}