> ## Documentation Index
> Fetch the complete documentation index at: https://docs.vers.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# Remote Subagents

> Spawn and drive coding agents on Vers VMs using the FIFO RPC protocol

Run coding agents (pi, Claude Code) on Vers VMs and drive them programmatically from a coordinator. This is the foundation for agent swarms, parallel task execution, and remote agent workflows.

## Architecture

A remote agent runs inside a Vers VM with a FIFO-based RPC channel. The coordinator (your local machine, a CI runner, or another agent) sends JSON messages over SSH and reads responses back.

```
Coordinator                         Agent VM
  │                                    │
  │  echo JSON > /tmp/pi-rpc/in       │
  ├───────────────────────────────────►│  pi --mode rpc
  │                                    │  reads from FIFO, executes task
  │  tail -f /tmp/pi-rpc/out          │
  │◄───────────────────────────────────┤  writes JSON responses
  │                                    │
```

Two tmux sessions run inside the VM:

* **pi-keeper**: Runs `sleep infinity > /tmp/pi-rpc/in` to hold the FIFO write-end open (prevents pi from getting EOF)
* **pi-rpc**: Runs `pi --mode rpc < /tmp/pi-rpc/in >> /tmp/pi-rpc/out 2>> /tmp/pi-rpc/err`

## Prerequisites

* Vers CLI [installed and authenticated](/agent-bootstrap)
* API keys configured via `vers env set` (e.g., `ANTHROPIC_API_KEY`)
* A golden image with pi pre-installed, or willingness to install it on a fresh VM

## Step 1: Boot an Agent VM

### From a Golden Image (Fast)

If you have a golden image with pi and the RPC scaffold pre-configured:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers run-commit <golden-commit-id> -N worker-1
```

The agent is immediately ready for tasks.

### From a Fresh VM (Manual Setup)

Start a fresh VM and install the agent manually:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers run -N worker-1
vers connect worker-1
```

Inside the VM:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
echo 'nameserver 8.8.8.8' > /etc/resolv.conf
apt-get update -qq && apt-get install -y tmux curl nodejs npm
npm install -g @mariozechner/pi-coding-agent
```

Then set up the RPC scaffold:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
# Create FIFO infrastructure
mkdir -p /tmp/pi-rpc
mkfifo /tmp/pi-rpc/in
touch /tmp/pi-rpc/out /tmp/pi-rpc/err

# Start keeper (holds FIFO open)
tmux new-session -d -s pi-keeper "sleep infinity > /tmp/pi-rpc/in"

# Start pi in RPC mode
tmux new-session -d -s pi-rpc "pi --mode rpc < /tmp/pi-rpc/in >> /tmp/pi-rpc/out 2>> /tmp/pi-rpc/err"

# Verify
tmux list-sessions
```

Exit the VM (`exit`).

## Step 2: Send a Task

From your coordinator, send a JSON message to the agent:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
VM="worker-1"

echo '{"type": "prompt", "message": "Create a REST API with Express that has GET /health and POST /echo endpoints"}' \
  | vers exec -i $VM "cat > /tmp/pi-rpc/in"
```

<Info>
  The `-i` flag is required when piping stdin into `vers exec`. Without it, stdin is not forwarded to the remote command. `vers execute` is an alias for `vers exec` and works identically.
</Info>

## Step 3: Read Output

### One-shot (last N lines)

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "tail -n 50 /tmp/pi-rpc/out"
```

### Streaming (live)

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "tail -f /tmp/pi-rpc/out"
# Ctrl+C to stop
```

### Check for errors

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "cat /tmp/pi-rpc/err"
```

## Message Protocol

### Standard Task

Queue a task. If the agent is idle, it starts immediately. If busy, it queues.

```json theme={"theme":{"light":"min-light","dark":"min-dark"}}
{"type": "prompt", "message": "your task description"}
```

### Steer (Interrupt + Redirect)

Interrupt current work and redirect the agent:

```json theme={"theme":{"light":"min-light","dark":"min-dark"}}
{"type": "steer", "message": "use FastAPI instead of Express"}
```

### Follow-up

Queue a task to run after the current one completes:

```json theme={"theme":{"light":"min-light","dark":"min-dark"}}
{"type": "follow_up", "message": "add rate limiting to the API"}
```

### Other Commands

```json theme={"theme":{"light":"min-light","dark":"min-dark"}}
{"type": "abort"}
{"type": "get_state"}
{"type": "new_session"}
```

### Response Format

The agent writes JSON lines to `/tmp/pi-rpc/out`. A completed task produces a line with:

```json theme={"theme":{"light":"min-light","dark":"min-dark"}}
{"type": "response", "success": true, ...}
```

Watch for `"type": "response"` to know when work is done.

## Polling Pattern for Long Tasks

For tasks that take minutes, poll for new output:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
VM="worker-1"
LAST_SIZE=0

while true; do
  CURRENT_SIZE=$(vers exec $VM "wc -c < /tmp/pi-rpc/out" 2>/dev/null || echo "0")
  if [[ "$CURRENT_SIZE" -gt "$LAST_SIZE" ]]; then
    echo "--- New output ---"
    vers exec $VM "tail -c +$((LAST_SIZE + 1)) /tmp/pi-rpc/out"
    LAST_SIZE=$CURRENT_SIZE
  fi
  sleep 10
done
```

## Multiple Agents (Swarm Pattern)

Spawn several agents and fan out work:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
# Boot 3 workers
for i in 1 2 3; do
  vers run-commit <golden-commit-id> -N "worker-$i" &
done
wait

# Send different tasks to each
echo '{"type":"prompt","message":"implement user authentication"}' \
  | vers exec -i worker-1 "cat > /tmp/pi-rpc/in"

echo '{"type":"prompt","message":"implement database models"}' \
  | vers exec -i worker-2 "cat > /tmp/pi-rpc/in"

echo '{"type":"prompt","message":"implement API endpoints"}' \
  | vers exec -i worker-3 "cat > /tmp/pi-rpc/in"

# Monitor all
for i in 1 2 3; do
  echo "=== worker-$i ==="
  vers exec "worker-$i" "tail -n 10 /tmp/pi-rpc/out"
done
```

## Snapshotting Agent State

Save an agent's working state as a commit for later reuse:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
# Commit the VM (captures files, memory, running processes)
vers commit worker-1
# Output: Commit ID: c1a2b3c4...

# Later, restore to exact same state
vers run-commit c1a2b3c4... -N worker-1-restored
```

This is useful for:

* Checkpointing before risky operations
* Creating reusable "warmed up" agent states
* Sharing agent environments across team members

## Building Golden Images for Agents

Manually setting up each agent VM is slow. For repeatable agent workflows, build a golden image once and boot from it instantly. See [Custom Golden Images](/agents/custom-golden-images) for the full process.

The key steps:

1. Start a fresh VM, install your agent and tools
2. Set up the tmux/FIFO scaffold and systemd auto-start
3. Clean secrets and regenerate SSH keys
4. Commit as a tagged snapshot

## Troubleshooting

### No output from agent

Check that tmux sessions are running:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "tmux list-sessions"
```

You should see both `pi-keeper` and `pi-rpc`. If missing, re-create the scaffold (Step 1 manual setup).

### Agent exits immediately

The `pi-keeper` session isn't running. Without it, the FIFO gets EOF and pi exits. Restart:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "tmux kill-session -t pi-rpc; tmux kill-session -t pi-keeper"
vers exec worker-1 "tmux new-session -d -s pi-keeper 'sleep infinity > /tmp/pi-rpc/in'"
vers exec worker-1 "tmux new-session -d -s pi-rpc 'pi --mode rpc < /tmp/pi-rpc/in >> /tmp/pi-rpc/out 2>> /tmp/pi-rpc/err'"
```

### Task sent but nothing happens

Check if pi is actually running:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers exec worker-1 "ps aux | grep pi"
```

Send a state check:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
echo '{"type": "get_state"}' | vers exec -i worker-1 "cat > /tmp/pi-rpc/in"
sleep 2
vers exec worker-1 "tail -n 5 /tmp/pi-rpc/out"
```

### SSH / connection issues

Verify the VM is running:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers status
```

If paused, resume it:

```bash theme={"theme":{"light":"min-light","dark":"min-dark"}}
vers resume worker-1
```

## See Also

* [Agent Bootstrap](/agent-bootstrap) for initial Vers setup (including authentication)
* [Custom Golden Images](/agents/custom-golden-images) for baking reusable agent VMs
* [Coding Agent (SSH)](/examples/coding-agent-ssh) for interactive agent use
* [Agent Swarms](/tutorials/agent-swarms) for multi-agent orchestration patterns