tao-run-on-virtualenv

Run a Python training/eval script directly in an existing local virtualenv — no docker, no container. Implements the four-verb consumer contract (submit/status/logs/cancel) over a vendored process-lifecycle runner with durable on-disk state, PID-reuse-safe identity, and process-group cleanup. Use for docker-free local execution, plain-Python model scripts, fast HPO/AutoML trial smokes, or hosts where containers are unavailable. Trigger phrases include "run in my venv", "no docker", "virtualenv execution", "local python training", "run this training script directly".

Virtualenv — docker-free local Python execution

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

The virtualenv platform runs a Python script natively in an existing venv — as an argv vector whose first element is <venv>/bin/python, never through a shell, never activating anything. The vendored runner (references/virtualenv_runner.py) is this platform's "native CLI" — the role docker/kubectl/sbatch play elsewhere — and owns only the process lifecycle. Job records stay with tao_job_record.py; specs are authored by the agent, exactly like every other platform.

When to use

• The workload is a plain Python script (its dependencies pip-installed in a venv), not a TAO container action.
• No docker on the host, or container startup cost isn't worth it (fast smokes, AutoML trial loops over lightweight models).
• Single node only. For TAO container actions use tao-run-on-docker; for clusters use -slurm / -kubernetes.

Preflight

# 1. The venv is real and has an executable interpreter.
[ -f "$VENV/pyvenv.cfg" ] && [ -x "$VENV/bin/python" ] || echo "MISSING: $VENV is not a venv"
# 2. The script's top-level imports resolve inside it (catches wrong-venv early);
#    substitute the real modules your script imports.
"$VENV/bin/python" -c "import torch" || echo "MISSING: script dependency not in $VENV"
# 3. GPU visibility only if the script needs CUDA.
nvidia-smi >/dev/null 2>&1 || echo "note: no GPU visible (fine for CPU scripts)"

No credentials are required by the platform itself; model-specific env vars (e.g. HF_TOKEN) pass through by NAME with -e (values never land on argv).

Storage

Tier A by definition — everything is local paths. Datasets must already be on local disk (stage with tao-data-io first if they live in S3). Outputs land in the job record's results_dir, which IS the runner's --job-dir.

Execution — the four verbs

$BANK = ${TAO_SKILL_BANK_PATH}; $RUNNER = $BANK/skills/platform/tao-run-on-virtualenv/references/virtualenv_runner.py.

submit

1. Author the spec (if the script takes one) at a local path — nested dicts, never flat dotted keys — and lint the assembled command with redact_secrets.py lint.
2. Open the record — mints the id, binds results_dir BEFORE launch:

   JOB_ID=$("$BANK/scripts/tao_job_record.py" open --platform virtualenv \
     --image "$VENV/bin/python" --network-arch "$ARCH" --action "$ACTION" \
     --storage-tier A --results-root "$RESULTS_ROOT")
   RESULTS_DIR="$RESULTS_ROOT/$JOB_ID"
   

3. Launch detached (the runner writes a durable wrapper that gates start, records identity, and cleans up the process group on exit):

   set -a; source /path/to/.env; set +a   # omit if already exported
   python3 "$RUNNER" submit --job-dir "$RESULTS_DIR" --venv "$VENV" \
     --script train.py --job-id "$JOB_ID" --config-path "$SPEC" \
     --arg train --arg=--config={config_path} --arg=--out={results_dir} \
     --gpu-ids 0 -e HF_TOKEN
   

   Placeholders {config_path} {results_dir} {job_id} render inside --arg tokens. A token starting with - must use the --arg=TOKEN form (argparse). --gpu-ids sets CUDA_VISIBLE_DEVICES; --gpus 0 hides GPUs; neither reserves anything.
4. Record RUNNING with the pid the runner printed:

   "$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state RUNNING --backend-ref "pid:<pid>"
   

One submit per job dir — a retry gets a NEW record (--retry-of), never a re-submit into the same dir.

status

python3 "$RUNNER" status --job-dir "$RESULTS_DIR"   # {"status": "...", ...}

Prints the fixed vocabulary directly: PENDING RUNNING COMPLETE ERROR CANCELED UNKNOWN — no mapping table needed. Status is derived from durable files (exit_status.json, launcher identity) and is safe to poll from any process, any time, including after reboots of the polling agent. On a terminal status, mark the record.

logs

python3 "$RUNNER" logs --job-dir "$RESULTS_DIR" --tail 200

cancel

python3 "$RUNNER" cancel --job-dir "$RESULTS_DIR"
"$BANK/scripts/tao_job_record.py" mark "$JOB_ID" --state CANCELED --source agent

Cancel marks first (a not-yet-started wrapper self-cancels at its start gate), verifies process identity (never kills a reused PID), then SIGTERM→SIGKILLs the whole process group. already_terminal in the reply means the job finished before the cancel — mark the record with the status it reports instead.

Platform caveats

• Linux first-class. Identity and group cleanup use /proc; on macOS the runner falls back to ps/pgrep — fine for local smokes, but GPU training targets are Linux hosts.
• No multi-node, no image resolution — there is no container. The "image" recorded is the venv's interpreter path.
• The runner never downloads anything. Remote inputs are the agent's job to stage first (tao-data-io).