headscale/cmd/hi/README.md

# hi — Headscale Integration test runner

`hi` wraps Docker container orchestration around the tests in
[`../../integration`](../../integration) and extracts debugging artefacts
(logs, database snapshots, MapResponse protocol captures) for post-mortem
analysis.

**Read this file in full before running any `hi` command.** The test
runner has sharp edges — wrong flags produce stale containers, lost
artefacts, or hung CI.

For test-authoring patterns (scenario setup, `EventuallyWithT`,
`IntegrationSkip`, helper variants), read
[`../../integration/README.md`](../../integration/README.md).

## Quick Start

```bash
# Verify system requirements (Docker, Go, disk space, images)
go run ./cmd/hi doctor

# Run a single test (the default flags are tuned for development)
go run ./cmd/hi run "TestPingAllByIP"

# Run a database-heavy test against PostgreSQL
go run ./cmd/hi run "TestExpireNode" --postgres

# Pattern matching
go run ./cmd/hi run "TestSubnet*"
```

Run `doctor` before the first `run` in any new environment. Tests
generate ~100 MB of logs per run in `control_logs/`; `doctor` verifies
there is enough space and that the required Docker images are available.

## Commands

| Command            | Purpose                                              |
| ------------------ | ---------------------------------------------------- |
| `run [pattern]`    | Execute the test(s) matching `pattern`               |
| `doctor`           | Verify system requirements                           |
| `clean networks`   | Prune unused Docker networks                         |
| `clean images`     | Clean old test images                                |
| `clean containers` | Kill **all** test containers (dangerous — see below) |
| `clean cache`      | Clean Go module cache volume                         |
| `clean all`        | Run all cleanup operations                           |

## Flags

Defaults are tuned for single-test development runs. Review before
changing.

| Flag                | Default        | Purpose                                                                     |
| ------------------- | -------------- | --------------------------------------------------------------------------- |
| `--timeout`         | `120m`         | Total test timeout. Use the built-in flag — never wrap with bash `timeout`. |
| `--postgres`        | `false`        | Use PostgreSQL instead of SQLite                                            |
| `--failfast`        | `true`         | Stop on first test failure                                                  |
| `--go-version`      | auto           | Detected from `go.mod` (currently 1.27.0)                                   |
| `--test-binary`     | unset          | Execute a precompiled integration test binary instead of running `go test`  |
| `--clean-before`    | `true`         | Clean stale (stopped/exited) containers before starting                     |
| `--clean-after`     | `true`         | Clean this run's containers after completion                                |
| `--keep-on-failure` | `false`        | Preserve containers for manual inspection on failure                        |
| `--logs-dir`        | `control_logs` | Where to save run artefacts                                                 |
| `--verbose`         | `false`        | Verbose output                                                              |
| `--stats`           | `false`        | Collect container resource-usage stats                                      |
| `--hs-memory-limit` | `0`            | Fail if any headscale container exceeds N MB (0 = disabled)                 |
| `--ts-memory-limit` | `0`            | Fail if any tailscale container exceeds N MB                                |

### Precompiled tests in CI

CI builds the Headscale server, `hi`, and the integration test binary once
for the worker architecture. Each test job loads the prebuilt images and
runs its own fresh test container:

```bash
CGO_ENABLED=0 GOOS=linux go test -c ./integration -o integration.test
go run ./cmd/hi run --test-binary=./integration.test "^TestHeadscale$"
```

With `--test-binary`, the runner mounts the executable read-only and uses
`HEADSCALE_INTEGRATION_HEADSCALE_IMAGE` as its runtime image when set.
Otherwise it uses the usual Go image. Build the binary with `CGO_ENABLED=0`
for the runner architecture so it needs no compiler or shared Go libraries.
The checkout, test fixtures, Docker access, per-run logs, resource checks,
and cleanup work the same as in source-based runs. CI does not distribute
Go caches to these workers.

Automatic preflight checks Docker and the checkout. Images are prepared by
their consumers, so ordinary tests do not pull Kubernetes images. The
standalone `doctor` command retains the comprehensive development checks.

### Timeout guidance

The default `120m` is generous for a single test. If you must tune it,
these are realistic floors by category:

| Test type                 | Minimum     | Examples                              |
| ------------------------- | ----------- | ------------------------------------- |
| Basic functionality / CLI | 900s (15m)  | `TestPingAllByIP`, `TestCLI*`         |
| Route / ACL               | 1200s (20m) | `TestSubnet*`, `TestACL*`             |
| HA / failover             | 1800s (30m) | `TestHASubnetRouter*`                 |
| Long-running              | 2100s (35m) | `TestNodeOnlineStatus` (~12 min body) |
| Full suite                | 45m         | `go test ./integration -timeout 45m`  |

**Never** use the shell `timeout` command around `hi`. It kills the
process mid-cleanup and leaves stale containers:

```bash
timeout 300 go run ./cmd/hi run "TestName"   # WRONG — orphaned containers
go run ./cmd/hi run "TestName" --timeout=900s  # correct
```

## Concurrent Execution

Multiple `hi run` invocations can run simultaneously on the same Docker
daemon. Each invocation gets a unique **Run ID** (format
`YYYYMMDD-HHMMSS-6charhash`, e.g. `20260409-104215-mdjtzx`).

- **Container names** include the short run ID: `ts-mdjtzx-1-74-fgdyls`
- **Docker labels**: `hi.run-id={runID}` on every container
- **Port allocation**: dynamic — kernel assigns free ports, no conflicts
- **Cleanup isolation**: each run cleans only its own containers
- **Log directories**: `control_logs/{runID}/`

```bash
# Start three tests in parallel — each gets its own run ID
go run ./cmd/hi run "TestPingAllByIP" &
go run ./cmd/hi run "TestACLAllowUserDst" &
go run ./cmd/hi run "TestOIDCAuthenticationPingAll" &
```

### Safety rules for concurrent runs

- ✅ Your run cleans only containers labelled with its own `hi.run-id`
- ✅ `--clean-before` removes only stopped/exited containers
- ❌ **Never** run `docker rm -f $(docker ps -q --filter name=hs-)` —
  this destroys other agents' live test sessions
- ❌ **Never** run `docker system prune -f` while any tests are running
- ❌ **Never** run `hi clean containers` / `hi clean all` while other
  tests are running — both kill all test containers on the daemon

To identify your own containers:

```bash
docker ps --filter "label=hi.run-id=20260409-104215-mdjtzx"
```

The run ID appears at the top of the `hi run` output — copy it from
there rather than trying to reconstruct it.

## Artefacts

Every run saves debugging artefacts under `control_logs/{runID}/`:

```
control_logs/20260409-104215-mdjtzx/
├── hs-<test>-<hash>.stderr.log        # headscale server errors
├── hs-<test>-<hash>.stdout.log        # headscale server output
├── hs-<test>-<hash>.db                # database snapshot (SQLite)
├── hs-<test>-<hash>_metrics.txt       # Prometheus metrics dump
├── hs-<test>-<hash>-mapresponses/     # MapResponse protocol captures
├── ts-<client>-<hash>.stderr.log      # tailscale client errors
├── ts-<client>-<hash>.stdout.log      # tailscale client output
└── ts-<client>-<hash>_status.json     # client network-status dump
```

Artefacts persist after cleanup. Old runs accumulate fast — delete
unwanted directories to reclaim disk.

## Debugging workflow

When a test fails, read the artefacts **in this order**:

1. **`hs-*.stderr.log`** — headscale server errors, panics, policy
   evaluation failures. Most issues originate server-side.

   ```bash
   grep -E "ERROR|panic|FATAL" control_logs/*/hs-*.stderr.log
   ```

2. **`ts-*.stderr.log`** — authentication failures, connectivity issues,
   DNS resolution problems on the client side.

3. **MapResponse JSON** in `hs-*-mapresponses/` — protocol-level
   debugging for network map generation, peer visibility, route
   distribution, policy evaluation results.

   ```bash
   ls control_logs/*/hs-*-mapresponses/
   jq '.Peers[] | {Name, Tags, PrimaryRoutes}' \
       control_logs/*/hs-*-mapresponses/001.json
   ```

4. **`*_status.json`** — client peer-connectivity state.

5. **`hs-*.db`** — SQLite snapshot for post-mortem consistency checks.

   ```bash
   sqlite3 control_logs/<runID>/hs-*.db
   sqlite> .tables
   sqlite> .schema nodes
   sqlite> SELECT id, hostname, user_id, tags FROM nodes WHERE hostname LIKE '%problematic%';
   ```

6. **`*_metrics.txt`** — Prometheus dumps for latency, NodeStore
   operation timing, database query performance, memory usage.

## Heuristic: infrastructure vs code

**Before blaming Docker, disk, or network: read `hs-*.stderr.log` in
full.** In practice, well over 99% of failures are code bugs (policy
evaluation, NodeStore sync, route approval) rather than infrastructure.

Actual infrastructure failures have signature error messages:

| Signature                                                       | Cause                     | Fix                                                           |
| --------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------- |
| `failed to resolve "hs-...": no DNS fallback candidates remain` | Docker DNS                | Reset Docker networking                                       |
| `container creation timeout`, no progress >2 min                | Resource exhaustion       | `docker system prune -f` (when no other tests running), retry |
| OOM kills, slow Docker daemon                                   | Too many concurrent tests | Reduce concurrency, wait for completion                       |
| `no space left on device`                                       | Disk full                 | Delete old `control_logs/`                                    |

If you don't see a signature error, **assume it's a code regression** —
do not retry hoping the flake goes away.

## Common failure patterns (code bugs)

### Route advertisement timing

Test asserts route state before the client has finished propagating its
Hostinfo update. Symptom: `nodes[0].GetAvailableRoutes()` empty when
the test expects a route.

- **Wrong fix**: `time.Sleep(5 * time.Second)` — fragile and slow.
- **Right fix**: wrap the assertion in `EventuallyWithT`. See
  [`../../integration/README.md`](../../integration/README.md).

### NodeStore sync issues

Route changes not reflected in the NodeStore snapshot. Symptom: route
advertisements in logs but no tracking updates in subsequent reads.

The sync point is `State.UpdateNodeFromMapRequest()` in
`hscontrol/state/state.go`. If you added a new kind of client state
update, make sure it lands here.

### HA failover: routes disappearing on disconnect

`TestHASubnetRouterFailover` fails because approved routes vanish when
a subnet router goes offline. **This is a bug, not expected behaviour.**
Route approval must not be coupled to client connectivity — routes
stay approved; only the primary-route selection is affected by
connectivity.

### Policy evaluation race

Symptom: tests that change policy and immediately assert peer visibility
fail intermittently. Policy changes trigger async recomputation.

- See recent fixes in `git log -- hscontrol/state/` for examples (e.g.
  the `PolicyChange` trigger on every Connect/Disconnect).

### SQLite vs PostgreSQL timing differences

Some race conditions only surface on one backend. If a test is flaky,
try the other backend with `--postgres`:

```bash
go run ./cmd/hi run "TestName" --postgres --verbose
```

PostgreSQL generally has more consistent timing; SQLite can expose
races during rapid writes.

## Keeping containers for inspection

If you need to inspect a failed test's state manually:

```bash
go run ./cmd/hi run "TestName" --keep-on-failure
# containers survive — inspect them
docker exec -it ts-<runID>-<...> /bin/sh
docker logs hs-<runID>-<...>
# clean up manually when done
go run ./cmd/hi clean all   # only when no other tests are running
```