lmswitch

██╗     ███╗   ███╗███████╗██╗    ██╗██╗████████╗ ██████╗██╗  ██╗
██║     ████╗ ████║██╔════╝██║    ██║██║╚══██╔══╝██╔════╝██║  ██║
██║     ██╔████╔██║███████╗██║ █╗ ██║██║   ██║   ██║     ███████║
██║     ██║╚██╔╝██║╚════██║██║███╗██║██║   ██║   ██║     ██╔══██║
███████╗██║ ╚═╝ ██║███████║╚███╔███╔╝██║   ██║   ╚██████╗██║  ██║
╚══════╝╚═╝     ╚═╝╚══════╝ ╚══╝╚══╝ ╚═╝   ╚═╝    ╚═════╝╚═╝  ╚═╝

lmswitch

local LLM switcher · GGUF + vLLM

List and toggle local LLMs from per-model YAML configs.

Adding a model? See SKILL.md — a self-contained guide (for humans or AI agents) to authoring ai-models/*.yaml recipes: every key, the extra_args passthrough, and the GB10 gotchas. Point an agent at this folder and it can add a model from that doc alone.

lmswitch shows a table of every configured model (grouped by family) with its size, download state, port, and whether it's currently serving — and lets you start/stop them interactively. GGUF models run under llama.cpp (llama-server as a background process); vLLM models run in Docker. It waits for each model to actually become ready, refuses loads that would exceed free RAM, and keeps your coding agents' configs in sync — opencode, hermes, grok, and omp — with whatever is serving.

Running lmswitch (the same wordmark above greets you):

  RAM    │ 122Gi total   33Gi used   89Gi available
  Models │ ~20.8G weights   1 / 3 loaded
  Disk   │ 113.1G   3 / 3 downloaded
  ● loaded   ○ stopped      ✓ downloaded   ✗ missing

   #  S  TYPE   NAME            SIZE  DL   PORT  DISPLAY
   Qwen ──────────────────────────────────────────────────────────────────────
   1  ○   gguf   qwen3-4b        2.3G  ✓   8085  Qwen3-4B
   2  ●   gguf   qwen3.6-35b    20.8G  ✓   8089  Qwen3.6-35B-A3B
   Nex ───────────────────────────────────────────────────────────────────────
   3  ○   gguf   nex-n2-pro     90.0G  ✓   8104  Nex-N2-Pro 397B-A17B (IQ1_M)

  Toggle # (space/comma separated, enter or q to quit):

Features

• One table for everything — loaded (●) vs stopped (○), downloaded (✓) vs missing (✗), per-model size/port, and RAM / disk / loaded-count totals.
• Six runtimes — GGUF via llama-server, safetensors/quantized via vLLM in Docker, SGLang in Docker (sglang), and three ways to serve one model across two DGX Sparks over a CX7 link: vllm-dual and sglang-dual (both tensor-parallel) and llama-dual (GGUF split over llama.cpp RPC). Pick per model with runtime:.
• Cluster view (optional) — with CLUSTER_HOSTS set, the table merges the other node's models with a HOST column (spark / gigabyte / dual) and toggling a peer's model delegates over SSH. Without it, nothing changes: single-box output and behavior are identical to previous versions.
• Readiness-aware — after launch it polls the model's /v1/models endpoint and only reports Ready once it's actually serving (with a …loading progress heartbeat and crash detection), so the synced configs reflect reality, not guesses.
• Pre-load RAM guard — refuses a start that would blow past available memory (overridable per-model with force: true), so a too-big model can't OOM-lock the machine.
• Config sync — on every toggle / on / off / sync, the currently-serving models are written into your coding agents' configs: opencode (opencode.json), hermes (config.yaml), grok (config.toml), and omp (models.yml). Pick which targets are active during lmswitch init. See Config sync.
• Optional systemd auto-restart per model via restart: on-failure.

Requirements

lmswitch targets Linux (it uses /proc/meminfo, ss, Docker --gpus, and systemd user units).

For	You need
lmswitch itself	Python 3.10+, curl, ss (iproute2). pyyaml is installed automatically as a dependency (a minimal built-in parser is used as a fallback if it's ever missing).
GGUF models	A built llama.cpp with llama-server (a CUDA build for GPU offload). Default binary path: <lmswitch>/../llama.cpp/build/bin/llama-server — override per-model with llama_bin:.
vLLM models	Docker + the NVIDIA Container Toolkit (--gpus all). Pulls the vllm/vllm-openai image.
restart: on-failure	A running systemd user instance (systemctl --user).
config sync	Any of opencode, hermes, grok, omp (all optional — only configs that exist are synced).

Submodule

This repo includes a git submodule (ai-models/) with community-contributed YAML recipes for GGUF and vLLM models. The submodule lives at jvr0x/ai-models.

# Clone with recipes baked in:
git clone --recurse-submodules https://github.com/jvr0x/lmswitch.git

# If you already cloned (without --recurse):
cd lmswitch
git submodule update --init

If you only need the recipes (no lmswitch code), clone the submodule repo directly: https://github.com/jvr0x/ai-models

Install

lmswitch is a Python package exposing a lmswitch console-script entry point (lmswitch = "lmswitch.cli:main"). The cleanest install is as an isolated CLI tool with uv (or pipx) — it keeps lmswitch's deps out of your system Python and sidesteps the externally-managed-environment (PEP 668) error on Debian/Ubuntu:

# from the repo dir (e.g. ~/utils/lmswitch)
uv tool install -e .        # editable; puts `lmswitch` on your PATH (~/.local/bin)
uv tool update-shell        # one-time: ensure uv's bin dir is on $PATH
lmswitch init

pipx install -e .                                  # same idea, via pipx
pip install --user -e . --break-system-packages    # plain pip --user (overrides PEP 668)
pip install -e .                                    # inside an activated virtualenv

init asks where your models live (writes ai-models/.lmswitch), creates the ai-models/ config dir, and asks which sync targets to enable (opencode / hermes / grok / omp — only the ones whose configs it finds). It does not reinstall the command: if a lmswitch console script is already on your PATH (from the step above) it leaves it alone; only if none is found does it drop a small launcher in ~/.local/bin pinned to the current interpreter. Ensure ~/.local/bin is on your $PATH.

Upgrade after pulling changes with uv tool upgrade lmswitch (an editable install picks up code edits automatically); remove with uv tool uninstall lmswitch.

Getting started

1. Install the requirements above.
2. lmswitch init — set your models directory (default ~/models).
3. Download a model into that directory (see Where to get models).
4. Create a config — lmswitch add <name>, or copy a template from examples/ into ai-models/<name>.yaml.
5. lmswitch → type the model's number to start it. It loads, waits until the endpoint answers, prints Ready on port <port>, and syncs your enabled configs (opencode / hermes / grok / omp).
6. Hit it: curl localhost:<port>/v1/models.

Usage

lmswitch                  # interactive: show the table, then type model #s to toggle
lmswitch list             # just print the table (read-only)
lmswitch on  <name|#>     # start a model
lmswitch off <name|#>     # stop a model
lmswitch sync             # regenerate enabled configs from currently-serving models
lmswitch add  <name>      # create a model config interactively
lmswitch serve <name>     # run a model in the foreground (used by systemd)
lmswitch stats            # usage: starts/stops, uptime, tokens in/out per model
lmswitch stats-clear      # wipe the recorded usage history
lmswitch init             # bootstrap ai-models/, .lmswitch, and sync targets
lmswitch -h, --help       # show help
lmswitch -v, --version    # print the version

In the interactive prompt you can toggle several at once, space/comma separated: 8 9 24. A toggle blocks until the model is ready (or its ready_timeout elapses); Ctrl-C aborts cleanly — the model keeps loading detached, so re-run lmswitch or lmswitch sync to pick it up once it's up.

Model configs (ai-models/<name>.yaml)

The filename (minus .yaml) is the model's id, its served-model-name, and — for vLLM — its container name (vllm-<id>). model: is a path relative to your models directory. Fully-commented templates live in examples/llama-gguf.yaml and examples/vllm.yaml.

Common keys

Key	Default	Meaning
runtime	llama	llama (GGUF), vllm (Docker), sglang (Docker), or the two-node vllm-dual / sglang-dual / llama-dual
model	—	path to the .gguf file (llama) or model dir (vLLM), relative to the models dir
port	8081	OpenAI-compatible server port
ctx	65536	context length
display_name	<name>	label in the table / synced configs
ready_timeout	600 (vLLM) / 300 (llama)	seconds to wait for readiness
force	false	bypass the pre-load RAM guard
restart	—	on-failure → run under a systemd user unit

llama (GGUF) keys: gpu_layers (99), threads (12), batch (1024), ubatch (512), alias, mmproj, llama_bin, and fit — default off, which skips llama.cpp's auto memory-fit step (it aborts in cudaMemGetInfo on some CUDA builds, e.g. GB10/Blackwell); set fit: none to omit the flag entirely on older llama.cpp builds that don't support -fit.

vLLM keys: gpu_memory_utilization (0.15), image, tool_call_parser, reasoning_parser, trust_remote_code, max_num_seqs, extra_args, and more — see examples/vllm.yaml for the full list.

SGLang keys: mem_fraction_static (0.95 — a hard up-front reservation, so the RAM guard sizes the model from it, not from the weights), image, attention_backend, chunked_prefill_size, disable_prefill_cuda_graph (true), kv_cache_dtype, tp_size, max_running_requests, tool_call_parser, reasoning_parser, sampling_defaults, cpuset (5-9,15-19 — GB10's performance cores; "" to disable pinning), shm_size (32g), json_model_override_args, plus the shared extra_mounts / env / extra_args. Containers are named sglang-<id>. A ctx above the model's native 262144 also needs a YaRN rope override in json_model_override_args, or SGLang silently clamps back down. See ai-models/qwen3.8-27b-nvfp4-sglang.yaml.

sglang-dual keys: every SGLang key above, plus the two-node set shared with vllm-dual — model_path (this node) / worker_model_path (the peer's path to the same weights; both are bound at /model inside the container so --model-path is one string), worker_host, master_addr, worker_ip, master_port (25000), tp_size (2), worker_env, head_extra_mounts / worker_extra_mounts, ready_timeout (2400). It launches sglang.launch_server --nnodes 2 --node-rank N --dist-init-addr HOST:PORT on both ranks; only rank 0 binds the API. Use it when a checkpoint simply does not fit one node — SGLang cannot offload, so a 125.9 GiB checkpoint on a 121 GiB box only serves sharded. See ai-models/qwen3.8-flash-next-nvfp4-dual.yaml.

Cluster mode: two Sparks (runtime: vllm-dual)

Serves one model tensor-parallel (TP=2) across two DGX Sparks over the CX7 link, using vLLM's native multi-node launcher (--nnodes 2 + --headless worker — no Ray). The head container runs on this node and exposes the API; the worker container is started on the peer over SSH and computes rank 1 of every forward pass. lmswitch on/off manages both nodes; a failed head tears the worker back down.

Weights can live once on either node: give each node its own path to the same directory (model_path / worker_model_path) — typically the owning node's ~/models/... and the peer's NFS mount of it. Both are mounted at a canonical /model inside the containers.

vllm-dual keys (on top of the vLLM keys above):

Key	Default	Meaning
worker_host	—	SSH alias of the peer node (passwordless)
master_addr	—	this node's CX7-link IP
worker_ip	—	peer's CX7-link IP (its VLLM_HOST_IP)
master_port	25000	torch distributed rendezvous port
nccl.ifname	—	CX7 interface (also pins GLOO_SOCKET_IFNAME — Gloo otherwise grabs arbitrary NICs)
nccl.hca	—	RDMA device for NCCL_IB_HCA
nccl.gid_index	—	NCCL_IB_GID_INDEX
model_path	—	weights dir on this node (may be an NFS mount)
worker_model_path	model_path	weights dir on the peer
worker_env	—	env overrides applied only to the worker container
gpu_memory_utilization	0.80	per node — each holds its TP shard
ready_timeout	1800	TP=2 loads are slow on first boot

Set enforce_eager: false in dual YAMLs when the profile relies on cudagraph capture — the single-node default is eager. And stop local models first: a loaded model on either node shrinks vLLM's memory budget there. See examples/vllm-dual.yaml.

Cluster mode: GGUF across two Sparks (runtime: llama-dual)

Serves one GGUF split across both nodes with llama.cpp's RPC backend: this node runs llama-server, the peer runs rpc-server (started over SSH), and --tensor-split decides how many layers land on each. No containers, no NCCL. RDMA over the CX7 link is auto-negotiated when both llama.cpp builds have -DGGML_RPC=ON -DGGML_RPC_RDMA=ON.

Weights live once, on this node: the head reads the GGUF and pushes the peer's share over the link, so nothing has to be copied or NFS-mounted. The peer caches what it receives (rpc_cache), so restarts skip the re-push.

llama-dual keys (on top of the llama keys above):

Key	Default	Meaning
worker_host	—	SSH alias of the peer node (passwordless)
worker_ip	—	peer's CX7-link IP — the RPC endpoint
rpc_port	50052	port rpc-server listens on
tensor_split	even	share per device, e.g. "0.45,0.55"
rpc_cache	true	peer caches received tensors under ~/.cache/llama.cpp/rpc
rpc_threads	—	peer's CPU threads (rpc-server -t)
worker_rpc_bin	~/utils/llama.cpp/build/bin/rpc-server	binary on the peer

tensor_split follows llama-server's device order — local CUDA devices first, then one RPC<n> per --rpc endpoint. Confirm with llama-server --list-devices --rpc <worker_ip>:<rpc_port> after a llama.cpp upgrade; reversing it silently loads the wrong share onto the wrong box. Both nodes must run the same llama.cpp commit.

Readiness waits on /health, not just the port: llama-server binds the port before loading and answers 503 meanwhile, which for a cross-node load is minutes of "ready" that isn't. lmswitch off kills the peer's rpc-server even when the head is already gone — a stranded one keeps its whole share resident.

Cluster visibility: add CLUSTER_HOSTS="<ssh-alias>" (comma-separated) to ai-models/.lmswitch on each node. lmswitch list then merges the peers' tables (via ssh <peer> lmswitch list --json) under a HOST column, and toggling a peer's model — by name or table number — delegates over SSH. Unreachable peers are skipped silently; the local table never breaks.

Where to get models

Models come from Hugging Face into your models directory; each config's model: path is relative to it.

pip install -U "huggingface_hub[cli]"

# GGUF (llama.cpp) — e.g. Unsloth / bartowski quants; grab the .gguf file(s)
hf download unsloth/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf \
  --local-dir ~/models/unsloth/Qwen3-4B-GGUF
#   → model: "unsloth/Qwen3-4B-GGUF/Qwen3-4B-Q4_K_M.gguf"

# vLLM (safetensors, incl. FP8 / NVFP4 quants) — grab the whole repo into a dir
hf download nvidia/Qwen3.5-MoE-...-NVFP4 --local-dir ~/models/nvidia/qwen3-...-nvfp4
#   → model: "nvidia/qwen3-...-nvfp4"

Good sources: unsloth/, bartowski/, ggml-org/ for GGUF; the upstream model repos and nvidia/ (FP8 / NVFP4) for vLLM. Multi-shard GGUF (*-00001-of-0000N.gguf) is detected automatically — point model: at the first shard.

How it works

• GGUF → llama-server is launched as a detached background process; its PID and full output go to ai-models/running/<name> and …/<name>.log. -fit off is passed by default (see fit: above).
• vLLM → docker run -d --name vllm-<name> --gpus all --network host …; any stale/exited container of the same name is docker rm -f'd first to avoid a name conflict.
• Readiness → after launch it polls http://localhost:<port>/v1/models until it answers (Ready on port <port>), the process/container dies (✗ … exited during startup + a pointer to the log / docker logs), or ready_timeout elapses (WARNING).
• Token accounting → on every off, lmswitch scrapes the server's Prometheus counters (llamacpp: / vllm: / sglang: …_tokens_total on /metrics) and its real process uptime before the kill, and appends them to ai-models/lmswitch-usage.json for lmswitch stats. --metrics (llama.cpp) and --enable-metrics (SGLang) are added to the launch command automatically — vLLM serves them by default. Counters are per server process, so they reset whenever it restarts.
• RAM guard → before launching, free RAM (MemAvailable from /proc/meminfo) is compared to an estimate: gpu_memory_utilization × total for vLLM, on-disk weight size × 1.3 for GGUF. If short, the start is refused unless force: true.
• Config sync → each enabled target gets the currently-serving models, all pointing at http://<SPARK_HOST>:<port>/v1: opencode one provider per model, hermes the active model + a custom_providers entry per model (so they show in /model), grok one [model.<id>] table per model, omp one lmswitch provider holding every model. SPARK_HOST is a constant in lmswitch.system.io (spark-8912.local) — change it if your host differs. See Config sync.

Config sync

lmswitch keeps your coding agents' configs honest: on every on / off / toggle / sync it rewrites the currently-serving models into each enabled target, every endpoint pointing at http://<SPARK_HOST>:<port>/v1. Your agent always sees the models that are actually up — right ports, right names — with no hand-editing and no calls to a model that isn't loaded. SPARK_HOST is a constant defined in the package (lmswitch.system.io.SPARK_HOST); set it to your serving host.

Targets are chosen during lmswitch init and stored as SYNC_OPENCODE / SYNC_HERMES / SYNC_GROK / SYNC_OMP in ai-models/.lmswitch (only configs that exist on disk are touched; a target with no config is skipped). Each shapes its own file:

• opencode → ~/.config/opencode/opencode.json gets one provider per serving model. If ~/.local/share/opencode-export/ exists, a copy is written there too, so a remote client (a laptop/Mac over Tailscale, LAN, or a Samba mount) can pick up the same config and point straight at the serving host.
• hermes → ~/.hermes/config.yaml. Hermes runs one active model, so the model: block is set to the serving model and kept sticky (only switched when the current one stops); a vision model (id containing vl) is wired into auxiliary.vision. Every serving model is also registered under custom_providers: (with discover_models: false, so the picker doesn't live-probe and hang) — that's what makes them all selectable from hermes' /model. Custom providers you added by hand (pointing at other hosts) are preserved. Because discovery is off, /model reflects the last sync — re-run lmswitch sync (or just toggle) after starting a model to refresh the list.
• grok → ~/.grok/config.toml gets one [model.<id>] table per serving model; all your other grok settings ([cli], [ui], marketplace, the [models] default, …) are left untouched.
• omp → ~/.omp/agent/models.yml gets a single lmswitch provider whose models: list carries one entry per serving model, each with its own baseUrl, so selectors read lmswitch/<model>. Models are written out explicitly instead of using omp's discovery: block: discovery re-probes /v1/models on every catalog refresh, which costs a full HTTP timeout per dead lane (the same hang discover_models: false avoids for hermes). Other providers in the file (ollama, a hosted proxy) are preserved, and the lmswitch provider is dropped entirely when nothing is serving. SYNC_OMP is opt-in — an existing .lmswitch predates omp support, so set SYNC_OMP=true (or re-run lmswitch init) to enable it.

Run lmswitch sync to regenerate on demand — handy after a detached load finishes.

Architecture

lmswitch is a small Python package with deliberate module boundaries. Runtimes are pluggable: adding a new backend (sglang, TGI, …) means writing one file that subclasses BaseRuntime and registering it — no changes to the CLI, sync, or loader code.

lmswitch/
├── __main__.py          # `python -m lmswitch`
├── cli.py               # arg parsing, table rendering, interactive TUI, commands
├── sync.py              # opencode / hermes / grok / omp config sync
├── models/
│   └── loader.py        # discover & parse ai-models/*.yaml → model dicts
├── runtimes/            # how a model is started / stopped / probed
│   ├── base.py          #   BaseRuntime ABC + RuntimeRegistry
│   ├── llama.py         #   GGUF via llama-server (detached background process)
│   ├── vllm.py          #   vLLM via Docker
│   ├── sglang.py        #   SGLang via Docker
│   ├── sglang_dual.py   #   SGLang TP=2 across two nodes (head + ssh worker)
│   ├── systemd.py       #   restart: on-failure → systemd user unit
│   └── wait.py          #   readiness polling
└── system/
    ├── io.py            # paths, constants (SPARK_HOST), YAML, family rules
    ├── checks.py        # port / docker / process-state detection
    └── memory.py        # /proc/meminfo + pre-load RAM guard

A toggle flows: cli resolves the name → runtimes.start_model runs the RAM guard (system.memory) and dispatches to the matching BaseRuntime → runtimes.wait polls until the endpoint answers → sync rewrites the enabled agent configs. All filesystem state lives under ai-models/ (configs, .lmswitch, and running/ PID files); the LMSWITCH_DATA_DIR env var overrides that root (used by the tests).

Development & tests

cd ~/utils/lmswitch
uv venv                            # create .venv from pyproject (Python >=3.10)
uv pip install -e .                # the package (pulls in pyyaml)
uv pip install pytest              # test runner
uv run pytest -q                   # run the whole suite (57 tests)

Run a single file / test:

uv run pytest tests/test_sync.py -q
uv run pytest tests/test_sync.py::test_regen_hermes_keeps_running_default_sticky -q

File	Covers
tests/test_cli.py	name/index resolution, rendering, command dispatch, init
tests/test_llama_cmd.py	llama-server command construction
tests/test_vllm_and_abort.py	vLLM start, readiness, RAM guard, Ctrl-C, opencode sync
tests/test_sync.py	config sync to opencode / hermes / grok / omp (selection, idempotency, round-trip)
tests/test_process_lifecycle.py	start → detect-running → stop lifecycle

All tests are pure unit tests — subprocess / Docker / curl / ports are stubbed and configs are written to temp dirs (via LMSWITCH_DATA_DIR), so they run anywhere (no GPU, no models, no Docker) and never touch your real configs.

License

Apache-2.0 © 2026 jvr0x.