{"id":"1cd34891-4a2c-438c-a466-389ff18eb950","slug":"gemellistudio","kind":"mcp","name":"GemelliStudio","summary":"Run Isaac Sim USD scenes in pure C#/.NET — PhysX physics, RTX sensors, robot control.","owner":"CristianMori","repoUrl":"https://github.com/CristianMori/GemelliStudio","license":"Apache-2.0","licenseTier":"permissive","trust":"community","stars":2,"match":null,"fewStars":true,"format":"mcp","createdAt":"2026-06-29T04:48:07.000Z","addedAt":"2026-10-21T21:26:05.938Z","logoUrl":"https://avatars.githubusercontent.com/u/412262?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["utilities"],"tags":[],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-22T22:32:02.623Z","sample":false,"url":"https://conduits.dev/tools/gemellistudio","language":"C#","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# Gemelli Studio\n\n*Gemelli* — Italian for \"twins\" (as in *digital twin*).\n\nA **C#/.NET digital-twin runtime and editor**. Load an Isaac Sim–exported USD scene and run rigid-body +\narticulation physics alongside physically-accurate sensor rendering, entirely in managed code — no live\nIsaac Sim in the loop. Everything is .NET 8/10, C#, x64.\n\n![Gemelli Studio — a Franka Panda driven live by a C# script in the fast rasterized viewport (431 fps), with the outliner, sensor (depth) panel, and per-frame script editor](docs/gemelli-studio.png)\n\n*The Studio UI: a Franka Panda driven live by the per-frame C# script (bottom), rendered in the fast\nrasterized viewport at 431 fps, with the prim outliner (left), the depth-sensor panel (right), and the\nstatus bar showing physics 1 ms / render 17 ms.*\n\nIt is built on two native engines, each wrapped in a thin C# binding:\n\n- [`Nvidia.OvPhysx`](https://github.com/CristianMori/PhysX) — physics (PhysX articulations via *ovphysx*)\n- [`Nvidia.Ovrtx`](https://github.com/CristianMori/ovrtx) — RTX path-traced rendering + sensors (*ovrtx*)\n\nThe single hand-off artifact between an authoring tool (Isaac Sim) and Gemelli is **USD** — exactly what\nIsaac Sim exports. Gemelli treats that USD as an editable, save-back-able document, not an opaque blob.\n\n> **Platform: Windows x64 only (for now).** The native RTX/Omniverse engine binaries ship as Windows\n> libraries, and a few input/crash/lib-resolution touchpoints use Win32 APIs. The architecture itself is\n> portable — the .NET orchestrator, Avalonia UI, GL rasterizer, IPC, and scripting are all cross-platform —\n> so a **Linux build is a possible future direction**, pending Linux engine binaries and abstracting those\n> Windows-specific bits.\n\n---\n\n## Highlights\n\n- **Two interchangeable viewports.** A custom **GL rasterizer** for a fast (~hundreds of fps) interactive\n  view, and the **ovrtx** path tracer for ground-truth RTX rendering and the sensor cameras.\n- **Real robot control.** Drive articulations by joint-drive targets, Cartesian end-effector targets\n  (differential IK), a per-frame C# script, or live keyboard / Xbox-controller teleop.\n- **Sensors & synthetic data.** Color, depth, and instance segmentation as typed frames; record a labeled\n  dataset to disk (color + raw/preview depth + segmentation + manifest).\n- **Live editing + USD save-back.** Edit transforms in the inspector; bake the current body poses *and*\n  robot joint state back to a standalone `.usda`.\n- **FMI co-simulation.** Industrial behavior models (FMI 2.0 FMUs and SSP 1.0 archives) embedded in the\n  scene via the [ovfmi](https://github.com/NVIDIA-Omniverse/omniverse-labs/tree/main/projects/ovfmi)\n  USD-FMI schema run in-process each frame — sensors in, drive commands out (see the conveyor demo).\n- **Signal Mapper.** A live node graph of the signal wiring: sensors and operator inputs on the left,\n  behavior blocks in the middle, actuators on the right, with per-wire values every frame. Cut and\n  reconnect wires, add constants, gamepad/keyboard sources, multipliers, or ONNX policies from the\n  palette — all while the twin runs.\n- **ONNX policies as blocks.** An exported RL actor drops into the graph like any other block. The\n  bundled demo walks the **Open Duck Mini v2** biped with its MuJoCo-trained policy, steered live\n  from the mapper.\n- **ROS 1 in the graph.** Topic blocks speak the actual ROS wire protocols (XML-RPC + TCPROS, via\n  the [Mori.RosSharp](https://www.nuget.org/packages/Mori.RosSharp) package — no bridge, no native\n  deps): subscribe `/cmd_vel` to steer, publish joint states, odometry, and `/clock` back out.\n- **Adjustable time.** A settings pane exposes the physics timestep and a sim-time **time-scale** (slow-mo\n  through ~10× acceleration, bounded by physics throughput).\n- **One assembled-app folder.** A `dotnet build` drops the whole app into `dist\\<Config>\\`.\n\n---\n\n## Quick start\n\nWith the native libraries placed under `native\\` (see [Setup](#setup)), no environment setup is needed —\nthe apps auto-discover them. From the repo root (PowerShell):\n\n```powershell\n.\\run-studio.ps1            # incremental build, then launch the Studio UI (-NoBuild to skip)\n\n.\\run-headless.ps1 --usd scenes\\franka_studio.usda `\n    --products /Render/OmniverseKit/HydraTextures/camera_sensor_162912244368 `\n    --steps 60 --device gpu --record out\\dataset\n```\n\nIn Studio: pick a scene from the dropdown (or **Browse**), choose a device, press **Start**. Drag to\norbit · Shift-drag to pan · wheel to zoom. Select a body in the outliner to edit its transform in the\ninspector; switch the **Sensor** panel between Color / Depth / Segmentation; **⚙ Settings** opens the\npacing pane; **● Rec** records a dataset; **Save USD** bakes the current state (including robot joint pose)\nto a new `.usda`.\n\n---\n\n## Architecture: two processes, one document\n\n**ovphysx and ovrtx cannot share a process.** Both bundle their own (incompatible) versions of the\nOmniverse/Carbonite plugin stack, and Carbonite allows exactly one plugin registry per process, so\nco-loading them crashes. Gemelli therefore runs each engine in its own worker process and bridges them\nover named pipes, with a wrapper-free orchestrator (`Gemelli.Core`) in the middle:\n\n```\n                         Gemelli.Core  (orchestrator, no native engine)\n                              TwinSession\n        ┌───────────────────────┴────────────────────────┐\n   named pipe                                         named pipe\n        │                                                │\n Gemelli.PhysicsHost   ── rigid-body poses (N×7) ──►  Gemelli.RenderHost\n   (ovphysx)              orchestrator converts          (ovrtx)\n   step + read poses      to USD 4×4 matrices            WriteAttribute(omni:xform) → Step → frames\n```\n\nEach frame: physics steps and returns rigid-body + articulation-link poses → the orchestrator converts\nthem to USD matrices → the render worker writes `omni:xform` and renders the configured products → typed\nsensor frames come back. Poses cross the pipe; rendered images stay in the render process (transported via\nshared memory). The USD document is the single source of truth that both workers load.\n\n### The fast viewport\n\nThe interactive navigation viewport doesn't need physically-accurate light transport — only the *sensor*\ncameras do. So the main viewport is drawn by a small in-process **GL rasterizer** (`Gemelli.Viewport`):\nit reads the scene geometry once with USD.NET (meshes, USD shapes, instanced robot links, per-face\nmaterial colors from `GeomSubset` bindings), bakes each mesh into its controlling rigid body's frame, and\neach frame draws the latest poses with smooth (Gouraud) shading over a ground grid — offscreen, then reads\nthe pixels back into the viewport. It runs on its own thread reading the latest pose cache, so it stays\nsmooth independent of the sim/sensor rate. **RTX** mode swaps the viewport back to the ovrtx path tracer.\n\n### One assembled-app folder\n\nA `dotnet build -c Release` assembles the whole app under `dist\\Release\\`:\n\n```\ndist\\Release\\\n  Gemelli.Studio.exe          orchestrator + UI (carries the USD.NET / Silk.NET native libs)\n  Gemelli.Headless.exe        orchestrator, no UI\n  Gemelli.Mcp.exe             orchestrator, MCP server\n  physics\\Gemelli.PhysicsHost.exe    ovphysx worker (isolated subfolder)\n  render\\Gemelli.RenderHost.exe      ovrtx worker (isolated subfolder)\n```\n\nThe workers live in their own subfolders on purpose: the Studio process carries USD.NET's native USD/TBB\nlibraries (for the rasterizer), and ovrtx ships its *own* Omniverse USD + a different TBB. If they shared\none directory those libraries would shadow ovrtx's runtime and crash its loader — so each worker gets a\nclean directory, and the orchestrator finds it by a fixed relative path (no scanning).\n\n---\n\n## Project layout\n\n| Project | Role |\n|---|---|\n| `src/Gemelli.Core` | Orchestrator (no native engine): named-pipe IPC, `TwinSession`/`TwinService`, transform conversion, sensor frames, control layer (`ISimApi`, IK), PNG encoder |\n| `src/Gemelli.Viewport` | Fast GL rasterizer: USD geometry load, offscreen render, pose-driven viewport |\n| `src/Gemelli.Physics` → `physics\\Gemelli.PhysicsHost.exe` | ovphysx worker |\n| `src/Gemelli.Render` → `render\\Gemelli.RenderHost.exe` | ovrtx worker (`SensorHub` maps outputs to typed frames) |\n| `src/Gemelli.Studio` → `Gemelli.Studio.exe` | Avalonia UI: viewport, outliner, inspector, sensor panel, script panel, settings |\n| `src/Gemelli.Headless` → `Gemelli.Headless.exe` | Console host: load USD → run loop → save PNGs / record datasets |\n| `src/Gemelli.Scripting` | Roslyn `.csx` controller host (hot-reload) + script globals |\n| `src/Gemelli.Fmi` | Signal graph: FMI 2.0 co-simulation host (P/Invoke, no Python), SSP 1.0 runner, ovfmi USD-FMI schema reader, ONNX policy blocks, ROS topic blocks, device/constant/multiply blocks, `SignalGraphController` |\n| `src/Gemelli.Mcp` → `Gemelli.Mcp.exe` | Model Context Protocol server (stdio) — tool-drivable, incl. `render_frame` vision |\n| `scenes/` | Sample USD scenes (franka_studio, conveyor_fmi, duck_walk, …) |\n| `tools/` | USD authoring (`usd-addcam`, `usd-addseg`, `usd-snapshot`), `import_duck_urdf.py`, and de-risk probes (`raster-probe`, `ovrtx-smoke`, `artic-probe`) |\n| `tests/Gemelli.Tests` | xUnit; tier-1 (pure) + gated tier-2 live-twin tests |\n| `external/` | **Vendored** wrapper sources (gitignored — re-create below) |\n| `native/` | **Vendored** native libraries (gitignored — acquire below) |\n| `run-studio.ps1` / `run-headless.ps1` | One-command launchers (incremental build, auto-discover native libs) |\n\n---\n\n## Setup\n\nRe-create the vendored wrapper sources (shallow sparse clones):\n\n```bash\nmkdir -p external && cd external\ngit clone --depth 1 --filter=blob:none --sparse https://github.com/CristianMori/ovrtx.git ovrtx\n( cd ovrtx && git sparse-checkout set csharp include skills examples )\ngit clone --depth 1 --filter=blob:none --sparse https://github.com/CristianMori/PhysX.git physx\n( cd physx && git sparse-checkout set ovphysx/csharp )\n```\n\nNative libraries (gitignored, version-matched). The apps auto-discover these conventional paths, so no\nenvironment variables are required when they're in place:\n\n- **ovphysx** from the PyPI wheel:\n  `pip download ovphysx==0.4.13 --no-deps --only-binary=:all: -d native/ovphysx_wheel`, extract →\n  `native/ovphysx/ovphysx/lib/ovphysx.dll`. (Override with `OVPHYSX_LIB`.)\n- **ovrtx** from the ovrtx GitHub release `v0.2.0` (windows-x86_64 zip) → `native/ovrtx/bin/`.\n  (Override with `GEMELLI_OVRTX_DIR`.)\n\nBuild & test (no native libraries required for the tier-1 tests):\n\n```bash\ndotnet build Gemelli.slnx -c Release\ndotnet test tests/Gemelli.Tests --filter TransformConversion\n```\n\n---\n\n## Studio UI\n\n`Gemelli.Studio` is a code-only Avalonia app (no XAML) over the shared `TwinService`. Launch with\n`.\\run-studio.ps1`. Layout:\n\n- **Header** — brand · transport (Play / Pause / Step / Stop) · scene dropdown + Browse · device ·\n  viewport mode (**Fast** / **RTX**) · render scale · **⚙ Settings** · Script toggle · **Signals**\n  (the signal mapper) · **Save USD** · Start.\n- **Outliner** (left) — rigid-body tree; select a body to inspect/edit it.\n- **Viewport** (centre) — live render; drag = orbit, Shift-drag = pan, wheel = zoom.\n- **Sensor** panel (right top) — the scene's sensor camera (wrist-mounted on the Franka hand in the\n  franka_studio scenes) with Color / Depth / Segmentation toggle + **● Rec**.\n- **Inspector** (right) — editable transform (X/Y/Z + Apply, written live) and a live pose readout.\n- **Script** panel (toggle) — per-frame C# (`.csx`) compiled by Roslyn; keyboard + Xbox controller\n  readable in-script.\n\n### Settings — timestep & time-scale\n\nThe **⚙ Settings** pane exposes the two pacing knobs, applied live to a running twin (and seeded into the\nnext run):\n\n- **Time scale** — sim-time : wall-clock factor. `1×` is real speed; `10×` runs ten seconds of simulation\n  per real second; `<1×` is slow-mo. The achievable speed-up is bounded by physics throughput: sim time\n  advanced per wall-second ≈ `timestep ÷ physics-ms-per-step`. The pane shows the measured physics step\n  time and the resulting ceiling, and tells you when a request exceeds it (it then runs flat-out).\n- **Physics step** — the fixed timestep. Smaller is more accurate/stable; larger is cheaper but can\n  destabilize articulations.\n\n### Driving the robot\n\nThe default script (Script panel) is keyboard / gamepad teleop of the Franka's end-effector:\n\n```csharp\n// .csx body runs each frame with sim, frame, time, ReadKeyboard, ReadController, MoveTcp, LinkPosition in scope.\nvar kb  = ReadKeyboard();\nvar mov = ReadController();\nfloat fwd    = (kb.W ? 1f : 0f) - (kb.S ? 1f : 0f) + mov.LeftY;\nfloat strafe = (kb.D ? 1f : 0f) - (kb.A ? 1f : 0f) + mov.LeftX;\nfloat lift   = (kb.E ? 1f : 0f) - (kb.Q ? 1f : 0f) + (mov.RightTrigger - mov.LeftTrigger);\n\nvar tcp = LinkPosition(\"/World/robot\", 8);\nif (tcp.HasValue)\n    MoveTcp(\"/World/robot\", 8, tcp.Value.X + fwd*0.02f, tcp.Value.Y + strafe*0.02f, tcp.Value.Z + lift*0.02f);\n```\n\n`MoveTcp` runs one damped-least-squares differential-IK step toward a world-space target and commands the\njoint drives. It integrates onto the last *commanded* target (not the gravity-drooped measurement), so the\narm holds position when you release the keys instead of sagging.\n\n---\n\n## Sensors & recording\n\nEach render product exposes its render vars as typed frames. **Depth** (`DistanceToImagePlaneSD`, float32\ndistance, background = +inf) and **color** (`LdrColor`) work out of the box on products that carry them\n(e.g. franka_studio's `camera_sensor`). **Segmentation** (`InstanceSegmentationSD`) needs the AOV authored\n— run `usd-addseg` to produce a `*_seg.usda`. In Studio the **Sensor** panel toggles Color / Depth /\nSegmentation; headless `--record <dir>` writes a synthetic-data set: `color_NNNNN.png`, raw\n`depth_NNNNN.f32` + preview `depth_NNNNN.png`, `seg_NNNNN.*` when present, and a `manifest.jsonl`.\n\nThe render product needs a camera that exists in the USD. The `tools/` USD-authoring helpers:\n\n- `usd-addcam` — add a perspective camera + dome light if the scene lacks them.\n- `usd-addseg` — add an `InstanceSegmentationSD` AOV + semantic labels so segmentation renders.\n- `usd-snapshot` — bake current body poses **and robot joint state** (radians → USD degrees) into a\n  standalone `.usda`, so a posed scene reloads exactly as left.\n\n---\n\n## Headless\n\n```powershell\n.\\run-headless.ps1 --usd scenes\\franka_studio.usda `\n  --products /Render/OmniverseKit/HydraTextures/camera_sensor_162912244368 `\n  --steps 120 --out out --device gpu --record out\\dataset\n```\n\nFlags: `--usd --products --steps --out --record <dir> --dt --rigid <glob> --device cpu|gpu|auto\n--script <path.csx> --dump-dof <robotPath>` (`--ovrtx-lib` / `--ovphysx-lib` optional, auto-discovered).\nThe first render compiles RTX shaders (slow once, then cached). Color PNGs (plus normalized depth PNGs\nwhen the product carries depth) land in `out\\`; `--record` writes the labeled dataset described above.\n\n---\n\n## Control layer\n\nControllers drive the twin through one interface, `ISimApi` (read sensors + physics tensor channels, write\nDOF position/velocity targets and poses), via a `TwinRunner` that calls each `IController.OnPreStep` before\nstepping:\n\n- `PlaybackController` — passive (physics and any USD time-keyed animation play out on their own).\n- `ScriptController` (`Gemelli.Scripting`) — runs a user **C# `.csx`** once per frame, compiled by Roslyn\n  and **hot-reloaded** on save; a bad script reports once and never crashes the twin.\n- `IkDragController` — drives an articulation link toward a continuously-updated world target via\n  differential IK.\n\n```csharp\n// Read live state and command actuation through ISimApi (over IPC):\nfloat[] poses = sim.Read(RigidBodyPose, \"/World/Cube*\");\nsim.SetDofPositionTargets(\"/World/arm\", new[]{ 0f, -0.4f, 0f });\n```\n\nHow a robot moves, end to end: an articulation's joints each carry a PhysX PD drive. You write a\n**drive target** (`set_dof_position_targets` ≡ `ISimApi.SetDofPositionTargets`); the solver applies torques\nevery step to move the joints toward it, respecting mass, inertia, and limits. Position targets say \"go to\nthis pose,\" velocity targets say \"spin at this rate.\"\n\n---\n\n## FMI co-simulation (the conveyor demo)\n\nGemelli hosts **FMI 2.0 co-simulation** models (`.fmu`) and **SSP 1.0** archives (`.ssp`) embedded in a\nscene with the ovfmi **USD-FMI schema**: `FmuInstance`/`SspInstance` prims declare the archive, and\n`FmuConnection` → `FmuMapping` children bind model variables to scene state. Everything runs in the\norchestrator process — the FMU's native win64 binary is called directly (no Python, no FMPy).\n\n![The conveyor demo in Gemelli Studio: an SSP-packaged line controller driving the roller zones](docs/conveyor-fmi.png)\n\n*The conveyor demo in the fast viewport: a presence sensor, a five-zone line controller, and five\nmotor drives (one SSP, three FMUs inside) drive the roller zones; the parcel is an ordinary rigid\nbody carried by roller contact.*\n\nSupported routings per mapping (`fmi:usdAttribute`):\n\n| Routing | Direction | Backed by |\n|---|---|---|\n| any USD attribute (e.g. `xformOp:translate`) | input | value captured from the stage (an \"operator panel\" prim) |\n| `physx:position` / `physx:velocity` | input | rigid-body pose/velocity tensor reads |\n| `physx:overlap` | input | a physics sphere-overlap query at the target prim (presence sensor) |\n| `physx:force` | output | rigid-body force tensor write |\n| `drive:angular:physics:targetVelocity` | output | articulation DOF velocity target (joint matched by name) |\n| `fmi:dofPositions` / `fmi:dofVelocities` | input | whole articulation DOF vector (measured), one thick wire |\n| `fmi:dofPositionTargets` / `fmi:dofVelocityTargets` | output | whole articulation drive-target vector |\n| `fmi:bodyPose` / `fmi:bodyVelocity` | input | rigid-body pose `[px py pz qx qy qz qw]` / velocity `[v ω]` |\n\nThe `fmi:usdMapping` `(offset, count)` selector doubles as the wire width: a count above 1 authors a\nvector wire that carries the whole slice.\n\nFMI prims are auto-detected at Start in both Studio and headless; instances step once per frame in USD\nauthoring order, SSPs atomically. The **conveyor demo** wires a presence sensor, a five-zone line\ncontroller, and five motor drives (one SSP, three FMUs) to a physical roller conveyor:\n\n```powershell\n# one-time: vendor the ovfmi assets and build the demo FMUs (needs MSVC x64)\ngit clone --depth 1 --filter=blob:none --sparse https://github.com/NVIDIA-Omniverse/omniverse-labs.git external/omniverse-labs\ngit -C external/omniverse-labs sparse-checkout set projects/ovfmi\n.\\tools\\build-fmus.ps1\n\n# run it (or pick conveyor_fmi in Studio's scene dropdown)\n.\\run-headless.ps1 --usd scenes\\conveyor_fmi.usda `\n    --products /Render/OmniverseKit/HydraTextures/omni_kit_widget_viewport_ViewportTexture_0 `\n    --steps 400 --device cpu\n```\n\nThe operator panel is the `/World/FMI/ConveyorControls` prim: `translate = (operatorSpeed,\nrejectSpeed, eStop)`, `scale.x = enable` — set `eStop` to 1 and the whole line winds down through the\ncontroller and every motor drive. Current limits: FMI 2.0 co-simulation only (no 3.0, no model\nexchange), Real variables only, and FMU outputs mapped to visual-only USD attributes update the input\ncache but are not yet written back to the renderer.\n\n### The Signal Mapper\n\n![The Signal Mapper: sensors and operator inputs wired through the SSP to the roller drive joints, with live values on every wire](docs/signal-mapper.png)\n\n*The conveyor's wiring mid-run: the presence sensor and operator panel feed the SSP (blue wires), its\nzone outputs drive the five roller joints (orange wires, carrying −16 — a package reject reversal in\nprogress), and unconnected output pins still show their live values.*\n\n**Signals** (header button, while a twin runs) opens the mapper — a node graph of the live wiring\nbacked directly by the running controller:\n\n- **Nodes**: signal sources (overlap sensors, operator-panel values) on the left, behavior blocks in\n  the middle, actuators (drive joints, force targets) on the right. Drag them by their title bar.\n- **Wires** are splines that follow the nodes and are labelled with the value that crossed them on\n  the latest frame; unconnected output pins show their live value beside the dot. Vector pins are\n  marked `[N]` and connect with **thick wires** carrying the whole vector; dropping a thick wire on\n  a scalar pin opens an element picker so you choose which component the wire carries.\n- **Right-click a wire to cut it; drag from one port dot to another to connect** — any source-side\n  port to any sink-side port, including block output → block input (chains of blocks). Changes\n  apply to the running twin on the next frame.\n- **The palette** (toolbar) adds blocks at runtime:\n  - **＋ Constant** — a live-editable value; wire it to a block input or straight to an actuator\n    (e.g. pin one roller zone to a fixed speed, bypassing the models entirely).\n  - **＋ Gamepad** — the Xbox controller as a source block: sticks, triggers, and buttons as pins.\n  - **＋ Keyboard** — keys as 0/1 pins, with an add-key box to grow the pin list on the fly.\n  - **＋ Multiply** — `out = a × b`, the everyday scaler (an unwired input acts as 1).\n  - **＋ ONNX…** — load an exported ONNX network; its named tensors become vector pins.\n  - Right-click a block's title bar to remove it.\n\n---\n\n## RL policies as blocks (the duck demo)\n\nThe same graph hosts neural controllers: an `OnnxInstance` prim (or the **＋ ONNX…** palette button)\nwraps an exported network in a block whose named tensors are vector pins. The bundled demo is the\n**Open Duck Mini v2** — a 3D-printed biped whose walking policy was trained with RL in MuJoCo\n(Open Duck Playground) and runs here, unmodified, against PhysX.\n\n![The Open Duck Mini v2 standing in the Fast viewport, driven by its ONNX walking policy](docs/duck-walk.png)\n\n*The duck in the Fast viewport (424 fps), standing in the policy's home pose. The robot is the\nURDF-converted model; the two thin lines are its antennae.*\n\n![The duck's signal graph: joint, IMU, and foot-pose vectors feed the policy block; its 16-wide target vector drives the articulation](docs/duck-signal-blocks.png)\n\n*The whole controller is five wires: joint positions/velocities `[16]`, body pose `[7]` and velocity\n`[6]` (the IMU), and the two foot poses feed the policy block; one thick `[16]` wire carries its\njoint targets back to the articulation drives. The command pins (`cmd_vx`, `cmd_vy`, `cmd_wz`) are\nleft unwired — wire a constant, a multiplier, or a gamepad stick to steer it live.*\n\nSince the network alone isn't a controller, the scene's `fmi:preset = \"open_duck_mini_v2\"` selects a\nhost block that replicates the training-side conventions around the raw actor: observation assembly\nin training order (body-frame gyro/accelerometer synthesized from world-frame physics), the\nMuJoCo-actuator ↔ PhysX-DOF order remap, 50 Hz self-pacing against sim time, action scaling around\nthe home pose, and the servo's rate limit.\n\nThe demo runs from a fresh clone: the converted robot USD and the Apache-2.0-licensed policy network\nfrom the [Open Duck Mini](https://github.com/apirrone/Open_Duck_Mini) project are vendored under\n`scenes\\duck\\` (see the NOTICE there). Pick `duck_walk.usda` in Studio and Start — the duck holds\nits crouch, steps in place, and walks under keyboard, constant, or gamepad commands wired in the\nmapper. To regenerate the robot USD from the source URDF, clone the Open Duck Mini repo and run\n`tools\\import_duck_urdf.py` with Isaac Sim's bundled Python.\n\n---\n\n## ROS 1 integration\n\nThe mapper's **＋ ROS…** button adds topic blocks backed by\n[Mori.RosSharp](https://www.nuget.org/packages/Mori.RosSharp) — a pure managed ROS 1 node\n(XML-RPC graph APIs + TCPROS transport on raw sockets). The twin joins the ROS graph as a regular\nnode named `/gemelli`: no rosbridge, no ROS installation on the Windows side, no native libraries.\n\n| Block | Direction | Message |\n|---|---|---|\n| Twist subscriber | ROS → twin | `geometry_msgs/Twist` as `linear [3]` / `angular [3]` pins (e.g. `/cmd_vel`) |\n| JointState publisher | twin → ROS | wired position/velocity vectors at a fixed rate; joint names prefill from labelled DOF pins |\n| Odometry publisher | twin → ROS | a body's `fmi:bodyPose`/`fmi:bodyVelocity` as `nav_msgs/Odometry` |\n| Clock publisher | twin → ROS | sim time on `/clock` for `use_sim_time` consumers (follows the time-scale) |\n\nThe duck demo closes the loop end-to-end: `teleop_twist_keyboard` on the ROS side publishes\n`/cmd_vel`, the Twist block feeds the walking policy's command pins, and the policy drives PhysX —\nwhile joint states stream back out to `rostopic echo`:\n\n```bash\n# ROS side (e.g. Ubuntu 20.04 in WSL2 with Noetic):\nroscore &\nrosrun teleop_twist_keyboard teleop_twist_keyboard.py _speed:=0.12 _turn:=1.0\n```\n\nIn Studio: start `duck_walk.usda` → **Signals** → **＋ ROS…** (master URI, Subscribe Twist) → wire\n`linear` → `cmd_vx` (element `x`) and `angular` → `cmd_wz` (element `z`).\n\nNetworking note for WSL2 (NAT): ROS carries each node's callback URIs inside the protocol, so both\nsides must be mutually reachable — set `ROS_IP` on the WSL side to its `eth0` address, and allow\ninbound TCP from the WSL subnet on the Windows side (one elevated rule:\n`New-NetFirewallRule -DisplayName \"ROS inbound (WSL)\" -Direction Inbound -Action Allow -Protocol TCP -RemoteAddress 172.16.0.0/12`).\nROS 1 only for now; ROS 2 (DDS) is a different protocol family.\n\n---\n\n## Tool-drivable (MCP server)\n\n`Gemelli.Mcp` exposes the twin over the **Model Context Protocol** (stdio) so any MCP-compatible client can\ndrive it via tools: `start_twin`, `step`, `get_info`, `list_rigid_bodies`, `read_poses`, `read_dof`,\n`set_dof_targets`, `stop_twin`, and **`render_frame`** — which returns the camera image as MCP image\ncontent. It drives the shared `TwinService` (one twin per server, calls serialized onto the sim thread).\n\nPoint the client at the built exe; native libraries are auto-discovered under `native\\` (add an `env`\nblock only if they live elsewhere):\n\n```json\n{\n  \"mcpServers\": {\n    \"gemelli\": {\n      \"command\": \"C:\\\\DataDrive\\\\ovGemelli\\\\dist\\\\Release\\\\Gemelli.Mcp.exe\"\n    }\n  }\n}\n```\n\nNote: MCP handles tool calls concurrently — call `start_twin` and wait for its reply before stepping (it\nblocks during worker launch + warmup); a normal sequential client does this naturally.\n\n---\n\n## Performance notes\n\nMeasured on the development machine (falling-boxes scene):\n\n| Lever | Effect |\n|---|---|\n| Fast (rasterized) viewport vs RTX | the rasterizer runs at hundreds of fps; RTX is dominated by the path-tracing floor below |\n| RTX render mode: RealTimePathTracing vs PathTracing | ~22 ms vs ~27 ms @720p (minor for simple scenes) |\n| RTX resolution: 720p → 540p → 360p | 22 → 15 → 14 ms (diminishing returns; ~13 ms floor) |\n| Shared-memory frame transport vs pipe | −~4 ms @720p (scales with resolution / sensor count) |\n\nThere is a **~13 ms per-frame RTX floor** (denoise + light transport) independent of geometry, resolution,\nor transport — inherent to path tracing. That floor is why the navigation viewport is rasterized: physics\nand the bridge run at hundreds of fps, so a cheap rasterized view keeps the UI fluid while ovrtx is\nreserved for the sensor cameras and ground-truth RTX mode.\n\n---\n\n## Status\n\n- ✅ Two-process twin verified end-to-end (physics simulates + renders, motion across frames; CPU **and** GPU physics)\n- ✅ ovphysx (PyPI 0.4.13) and ovrtx (0.2.0) installed and verified live on an RTX GPU\n- ✅ Real Isaac Sim import verified — a genuine export loads into both engines and renders, with no live Isaac Sim\n- ✅ Bridged articulation link poses — robots render in motion\n- ✅ **Fast rasterized viewport** (`Gemelli.Viewport`): USD geometry incl. instanced robot links, per-face materials, smooth shading, pose-driven, hundreds of fps\n- ✅ **Sensors**: depth + segmentation; Studio Sensor panel (Color / Depth / Seg)\n- ✅ **Recording / synthetic data-gen**: color + raw/preview depth + segmentation + `manifest.jsonl`\n- ✅ **Live editing + USD save-back**: inspector transform edit; `usd-snapshot` bakes body poses **and robot joint state**\n- ✅ **Control**: `ISimApi` + playback / Roslyn C# scripting / differential IK; keyboard + Xbox-controller teleop\n- ✅ **Adjustable pacing**: live time-scale (slow-mo → ~10×) and physics timestep via the Settings pane\n- ✅ **FMI co-simulation** (`Gemelli.Fmi`) — FMI 2.0 + SSP 1.0 host via the ovfmi USD-FMI schema; conveyor demo verified end-to-end (sensor → controller → roller drives)\n- ✅ **Signal Mapper** — live node graph of the signal wiring: per-wire values, cut/reconnect at runtime, vector pins with thick wires + element picker, palette of constant / gamepad / keyboard / multiply / ONNX blocks\n- ✅ **ONNX policy blocks** — exported RL actors as graph blocks; the Open Duck Mini v2 walks in PhysX with its MuJoCo-trained policy (`duck_walk.usda`)\n- ✅ **ROS 1 integration** — topic blocks over Mori.RosSharp (pure managed XML-RPC + TCPROS); teleop-steered duck verified against a live roscore, joint states/odometry/clock published back\n- ✅ **MCP server** (`Gemelli.Mcp`) — tool-drivable incl. `render_frame` vision; verified end-to-end\n- ✅ **Avalonia Studio** — viewport + transport + outliner + inspector + sensor panel + settings over the shared `TwinService`\n- ✅ Packaging: single assembled-app folder (`dist\\<Config>\\`), native-lib auto-discovery, one-command launchers\n- ✅ Hardening: worker-crash surfacing, run-loop fault tolerance, clean shutdown, quiet console\n- ✅ Tier-1 tests (transform + control/scripting + `TwinService` threading); named-pipe IPC; PNG encoder\n\n---\n\n## License\n\nApache License 2.0 © 2026 Cristian Mori. See [LICENSE](LICENSE).\n\nThe two native engines (*ovphysx*, *ovrtx*) and their C# wrappers are acquired separately under their own\nterms; they are not redistributed here (see [Setup](#setup)).\n"}