{"id":"e0329b29-6c45-461a-b7f1-e86469d4ea69","slug":"ci-optimization-skill","kind":"skill","name":"ci-optimization-skill","summary":"","owner":"illideristudios","repoUrl":"https://github.com/illideristudios/ci-optimization-skill","license":"MIT","licenseTier":"permissive","trust":"community","stars":1,"match":null,"fewStars":true,"format":"skill","createdAt":"2026-10-13T00:59:53.000Z","addedAt":"2026-10-22T02:02:56.840Z","logoUrl":"https://avatars.githubusercontent.com/u/343826815?v=4","linkOnly":false,"collection":null,"skillPath":null,"categories":["writing"],"tags":[],"runtime":"binary","local":true,"cloudCapable":false,"cloudCapability":"unknown","cloudReadyOnPlatform":false,"needsApiKey":false,"reviewed":false,"syncedAt":"2026-10-26T09:56:21.998Z","sample":false,"url":"https://conduits.dev/tools/ci-optimization-skill","language":"Python","transports":["stdio"],"install":{},"requirements":{"env":[],"upstreamMayCharge":false},"tools":null,"readme":"# GitLab CI/CD Optimization Skill (`ci-optimization-skill`)\n\nAn agent skill, two small Python tools, and four pipeline templates for designing, auditing, and optimizing GitLab CI/CD pipelines (`.gitlab-ci.yml`).\n\nDerived from the **[Gauntlet GitLab Primer](https://gitlab-primer.netlify.app/)** and modern CI/CD engineering best practices.\n\n---\n\n## ⚡ The Problem: Why CI Pipelines Break & Waste Budget\n\n1. **Exit Code 137 (OOM Kills)**: Runner containers (typically 1GB–4GB RAM, 2 vCPUs) report host hardware (e.g. 16–32 CPUs). Tools that size their worker pool from the CPU count (Jest, Vitest, `pytest -n auto`) spawn dozens of worker processes, causing catastrophic CPU thrashing and immediate kernel OOM termination (`128 + 9 = SIGKILL`).\n2. **The 3 Setup Taxes**: Every job pays taxes pulling images, installing OS packages (`apt-get`, `curl`, `jq`), and reinstalling language dependencies. On small runners, overhead dominates real testing/building time.\n3. **Duplicate Pipelines**: Pushing to a branch with an open Merge Request often triggers **two identical pipelines** (branch pipeline + MR pipeline), doubling CI runner billing.\n4. **Queue Waves**: Sharding tests into 8 parallel jobs on a cluster with only 2 concurrent runner slots causes jobs to queue in waves, multiplying wall-clock wait times.\n\n---\n\n## 🛡️ The Core Optimization Doctrine\n\n### 1. Zero Exit 137 OOM Kills\n* **Cap the Workers**: Pin test processes to the runner's vCPUs (2 on Gauntlet runners):\n  * **Node.js**: `npx jest --maxWorkers=2 --workerIdleMemoryLimit=512MB`\n  * **Python**: `uv run pytest -n 2` (needs `pytest-xdist`; without `-n` pytest is already serial)\n  * **Go**: `go test -p 2` with `GOMAXPROCS=\"2\"`\n* **Cap Runtime Heap**: The ceilings add up to 75% of container RAM, and each applies per process, so divide by the processes the job runs. On a 2048 MB runner:\n  * **Node.js**: `NODE_OPTIONS=\"--max-old-space-size=1536\"` for one process, `512` under Jest with 2 workers\n  * **Go**: `GOMEMLIMIT=\"768MiB\"` with `go test -p 2`\n  * **JVM**: `JAVA_TOOL_OPTIONS=\"-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0\"`\n  * **Python**: CPython has no heap cap; `MALLOC_ARENA_MAX=\"2\"` limits fragmentation and the worker cap bounds memory\n\n### 2. Pay the Setup Tax Once\n* **Prebuilt CI Image (Layer 1)**: Keep tools in `Dockerfile.ci` built in `.pre` stage only when the Dockerfile changes on the default branch, pushing to `$CI_REGISTRY_IMAGE/ci:latest`.\n* **Install Once, Sync Everywhere (Layer 2)**: A single `install` job with frozen lockfiles (`npm ci`, `uv sync --frozen`, `go mod download`) fills a cache keyed on the lockfile (`policy: pull-push`), and the jobs that need it pull that cache (`policy: pull`). Vendor folders (`node_modules/`, `.venv/`, `.nuget/`, `obj/`) are never artifacts.\n* **Artifacts only where they must exist**: Most jobs upload nothing. An artifact is allowed only when a later job or a person needs that exact output and cannot rebuild it from the cache (the bundle a deploy ships, a readable report), and it sets `expire_in: 1 hour`.\n* **DAG Execution with `needs:` (Layer 3)**: Downstream jobs (`lint`, `test`, `build`) declare `needs: [install]` to start immediately with dependencies already mounted on disk.\n\n```\nA(j) := job j uploads artifacts to GitLab\nM(j) := a later job or a person needs j's exact output and cannot rebuild it from the cache\nV(j) := j's output is a dependency/vendor folder (node_modules/, .venv/, .nuget/, obj/)\nE(j) := j's artifacts set expire_in: 1 hour\n\n1.  A(j) → M(j)       [Zach, 2026-10-19: \"artifacts THAT MUST BE THERE expire after 1 hour\"]\n2.  V(j) → ¬M(j)      [the install job's lockfile-keyed cache carries dependencies to every consumer]\n3.  A(j) → E(j)       [Zach, 2026-10-19]\n4.  V(j) → ¬A(j)      [2, then the contrapositive of 1: ¬M(j) → ¬A(j)]\n∴   install jobs upload nothing, and any artifact that does exist expires in 1 hour\n```\n\n### 3. Pipeline Hygiene & Concurrency Tuning\n* **One Pipeline per Push**: Enforce top-level `workflow: rules:` to prioritize MR pipelines and eliminate duplicate branch runs.\n* **Auto-Cancel Stale Runs**: `default: interruptible: true` cancels outdated pipelines when new commits are pushed. Deploy, publish and image-build jobs set `interruptible: false`.\n* **Tight Timeouts**: `default: timeout: 15m` stops a hung job from draining the runner quota.\n* **Shallow Clone**: `variables: GIT_DEPTH: \"20\"` skips downloading full git commit histories.\n* **Shards <= Runner Slots**: Ensure `parallel: N` test shards never exceed available concurrent runner slots.\n* **Offload Heavy Scans**: Move end-to-end browser suites (Playwright/Cypress) and full SAST/DAST security scans to schedules (`$CI_PIPELINE_SOURCE == \"schedule\"`) or manual triggers.\n\n---\n\n## 📦 Install\n\n**Claude Code** installs both skills as a plugin, from GitHub or from a local checkout:\n\n```bash\n/plugin install gitlab-ci-optimizer --marketplace locriani/ci-optimization-skill\n# or: claude plugin marketplace add ~/path/to/ci-optimization-skill && claude plugin install gitlab-ci-optimizer@gitlab-ci-optimizer\n```\n\n**Other agents**: the scripts need Python 3 and nothing else. Every agent file refers to the skill at `.agents/skills/gitlab-ci-optimizer/`, so copy `skills/` there in the project whose pipeline you are working on:\n\n```bash\ngit clone https://github.com/locriani/ci-optimization-skill\ncd your-project\nmkdir -p .agents && cp -R ../ci-optimization-skill/skills .agents/skills\n```\n\nThen copy the adapter for each agent you use:\n\n| Agent | Copy into your project | What it does |\n| :--- | :--- | :--- |\n| **Google Antigravity** | nothing more | Loads both skills from `.agents/skills/`. |\n| **Claude Code** | `.claude/commands/simulate-budget.md` | Adds the `/simulate-budget` command. |\n| **Cursor** | `.cursorrules`, `.cursor/rules/*.mdc` | Attaches the optimization rules when a `.gitlab-ci*.yml` or `Dockerfile.ci` is open; `/simulate-budget` rule. |\n| **OpenAI Codex / Copilot** | `CODEX.md`, `AGENTS.md`, `.codex/instructions.md` | Multi-agent instructions and CLI memory budgeting guides. |\n\nIn this repository `.agents/skills` is a symlink to `skills/`, so the same paths work here.\n\n---\n\n## 🧰 Included Tools & Scripts\n\n### 1. Runner Memory & Concurrency Simulator (`/simulate-budget`)\nEstimates peak RAM per test job, flags OOM risk, and estimates wall-clock time on a given number of runner slots:\n\n```bash\npython3 skills/gitlab-ci-optimizer/scripts/simulate_budget.py \\\n  --stack node \\\n  --ram 2048 \\\n  --vcpus 2 \\\n  --slots 2 \\\n  --workers 2 \\\n  --shards 2\n```\n\n`--no-prebuilt` and `--no-install-once` model a pipeline without those two fixes. The exit code is 1 when the job would be OOM killed. The simulator does not read your pipeline: suite duration and per-worker memory are illustrative constants, so it compares configurations and does not predict a real job.\n\n### 2. 11-Point Lean Scorecard Pipeline Auditor\nAudits any `.gitlab-ci.yml` against the scorecard below and prints a Lean Score from 0 to 100, the share of applicable checks that pass:\n\n```bash\npython3 skills/gitlab-ci-optimizer/scripts/audit_pipeline.py path/to/.gitlab-ci.yml\n```\n\nPass `--ram` and `--max-workers` when the runner is not 2048 MB / 2 vCPU. The exit code is 0 when every applicable check passes, 1 when one fails, and 2 when the file cannot be audited. A check that does not apply to the pipeline (no artifacts, no test command, a parent pipeline that only triggers children) is reported as `N/A`.\n\nThe checks:\n1. One pipeline per push (`workflow: rules`)\n2. Stale pipelines cancel (`default: interruptible: true`)\n3. Default job timeout <= 15m\n4. Every `artifacts:` block expires within 1 hour\n5. Shallow git clone (`GIT_DEPTH` 1-20)\n6. Test workers capped to runner vCPUs (`--maxWorkers`, `-n`, `-p`, `--workers`)\n7. Runtime heap ceiling, summed over a job's processes, <= 75% of runner RAM (`NODE_OPTIONS`, `GOMEMLIMIT`)\n8. Dependencies installed once and shared by cache: one `install` job fills a lockfile-keyed cache (`pull-push`), the jobs that `needs:` it pull it (`pull`), and no install job uploads artifacts\n9. No tool installs at job time (prebuilt CI image)\n10. Every test or trigger job on the push path skips untouched code (`rules: changes:`)\n11. E2E and security scans off the push path (`schedule` or `manual`)\n\nThe auditor matches text in one block-style YAML file with comments removed. It does not follow `include:`, `extends:` or anchors, so read the YAML before acting on a result.\n\n### Tests\n\n```bash\npython3 -m unittest discover -s tests\n```\n\n---\n\n## 📁 Repository Layout\n\n```text\n.\n├── SKILL.md                          # Master skill entrypoint\n├── README.md                         # Project documentation\n├── AGENTS.md                         # Universal multi-agent specification\n├── CODEX.md                          # OpenAI Codex configuration & field manual\n├── .cursorrules                      # Project-level Cursor AI instructions\n├── .agents/skills -> ../skills       # The path every agent file uses\n├── .claude/commands/\n│   └── simulate-budget.md            # Claude Code /simulate-budget slash command\n├── .cursor/rules/\n│   ├── gitlab-ci.mdc                 # Auto-trigger rule for .gitlab-ci*.yml files\n│   └── simulate-budget.mdc           # Cursor slash command / simulation rule\n├── .codex/\n│   └── instructions.md               # Codex workspace prompt\n├── scripts/ references/ examples/    # Root copy of the skill's directories (a test keeps them identical)\n├── skills/\n│   ├── gitlab-ci-optimizer/          # Packaged skill bundle\n│   │   ├── SKILL.md                  # Skill entrypoint: the rules and the workflow\n│   │   ├── scripts/\n│   │   │   ├── audit_pipeline.py     # 11-point lean checklist auditor CLI\n│   │   │   └── simulate_budget.py    # Runner memory & concurrency simulator CLI\n│   │   ├── references/\n│   │   │   ├── memory-and-oom.md             # Exit 137, cgroups, host-core trap & heap caps\n│   │   │   ├── setup-tax-and-caching.md      # The 3 taxes, Dockerfile.ci & install-once DAG\n│   │   │   └── dag-and-sharding.md           # needs:, parallel: N sharding, workflow:rules\n│   │   └── examples/\n│   │       ├── node.gitlab-ci.yml        # 100/100 lean template for Node.js / TypeScript\n│   │       ├── python.gitlab-ci.yml      # 100/100 lean template for Python (uv + Pytest)\n│   │       ├── go.gitlab-ci.yml          # 100/100 lean template for Go (go mod + go test)\n│   │       └── monorepo.gitlab-ci.yml    # Parent pipeline that triggers per-package children\n│   └── simulate-budget/              # /simulate-budget skill (uses the simulator above)\n└── tests/\n    └── test_scripts.py               # Tests for both scripts; the templates must pass the auditor\n```\n\n---\n\n## 📜 License\n\nMIT License. See [LICENSE](LICENSE) for details."}