cli/policy: route check through gRPC; bypass goes direct to DB

policy check previously ran the full policy engine in-process inside
the CLI, building a sandbox PolicyManager from the file and the
database. That duplicated the engine's runtime dependencies onto the
CLI and forced --bypass-grpc-and-access-database-directly to pull in
full server config validation. nblock hit it on PR #3229: passing
--bypass with a real config produced a flood of 'Fatal config error:'
lines from validateServerConfig because the cobra init early-return
for 'policy check' skipped --config registration and OnInitialize.

Make 'policy check' a thin frontend for the new CheckPolicy gRPC
method. The server-side handler builds a fresh PolicyManager from the
request bytes and the state's live users/nodes, runs SetPolicy on the
sandbox so the tests block executes, and returns the result through
gRPC status. No persistence, no policy_mode coupling.

--bypass-grpc-and-access-database-directly keeps doing what its name
says — opens the DB directly for cases where the server is not
running — but is no longer the only way to evaluate a tests block.

Drop the 'policy check' early-return in cmd/headscale/cli/root.go
(added in PR #2580 when check was syntax-only). All paths now need
either gRPC or direct DB access, both of which want the config and
flags the rest of cobra init sets up.

integration/cli_policy_test.go covers the matrix nblock asked about:
policy_mode={file,database} x fixture={acl-only, acl+passing-tests,
acl+failing-tests} x bypass={false,true} = 12 rows. acl-only and
acl-plus-passing-tests must pass; acl-plus-failing-tests must surface
'test(s) failed'; the policy_mode axis proves check does not depend
on where the server stores its current policy.

Updates #1803

.claude/agents/headscale-integration-tester.md

@@ -1,763 +0,0 @@
----
-name: headscale-integration-tester
-description: Use this agent when you need to execute, analyze, or troubleshoot Headscale integration tests. This includes running specific test scenarios, investigating test failures, interpreting test artifacts, validating end-to-end functionality, or ensuring integration test quality before releases. Examples: <example>Context: User has made changes to the route management code and wants to validate the changes work correctly. user: 'I've updated the route advertisement logic in poll.go. Can you run the relevant integration tests to make sure everything still works?' assistant: 'I'll use the headscale-integration-tester agent to run the subnet routing integration tests and analyze the results.' <commentary>Since the user wants to validate route-related changes with integration tests, use the headscale-integration-tester agent to execute the appropriate tests and analyze results.</commentary></example> <example>Context: A CI pipeline integration test is failing and the user needs help understanding why. user: 'The TestSubnetRouterMultiNetwork test is failing in CI. The logs show some timing issues but I can't figure out what's wrong.' assistant: 'Let me use the headscale-integration-tester agent to analyze the test failure and examine the artifacts.' <commentary>Since this involves analyzing integration test failures and interpreting test artifacts, use the headscale-integration-tester agent to investigate the issue.</commentary></example>
-color: green
----
-
-You are a specialist Quality Assurance Engineer with deep expertise in Headscale's integration testing system. You understand the Docker-based test infrastructure, real Tailscale client interactions, and the complex timing considerations involved in end-to-end network testing.
-
-## Integration Test System Overview
-
-The Headscale integration test system uses Docker containers running real Tailscale clients against a Headscale server. Tests validate end-to-end functionality including routing, ACLs, node lifecycle, and network coordination. The system is built around the `hi` (Headscale Integration) test runner in `cmd/hi/`.
-
-## Critical Test Execution Knowledge
-
-### System Requirements and Setup
-```bash
-# ALWAYS run this first to verify system readiness
-go run ./cmd/hi doctor
-```
-This command verifies:
-- Docker installation and daemon status
-- Go environment setup
-- Required container images availability
-- Sufficient disk space (critical - tests generate ~100MB logs per run)
-- Network configuration
-
-### Test Execution Patterns
-
-**CRITICAL TIMEOUT REQUIREMENTS**:
-- **NEVER use bash `timeout` command** - this can cause test failures and incomplete cleanup
-- **ALWAYS use the built-in `--timeout` flag** with generous timeouts (minimum 15 minutes)
-- **Increase timeout if tests ever time out** - infrastructure issues require longer timeouts
-
-```bash
-# Single test execution (recommended for development)
-# ALWAYS use --timeout flag with minimum 15 minutes (900s)
-go run ./cmd/hi run "TestSubnetRouterMultiNetwork" --timeout=900s
-
-# Database-heavy tests require PostgreSQL backend and longer timeouts
-go run ./cmd/hi run "TestExpireNode" --postgres --timeout=1800s
-
-# Pattern matching for related tests - use longer timeout for multiple tests
-go run ./cmd/hi run "TestSubnet*" --timeout=1800s
-
-# Long-running individual tests need extended timeouts
-go run ./cmd/hi run "TestNodeOnlineStatus" --timeout=2100s  # Runs for 12+ minutes
-
-# Full test suite (CI/validation only) - very long timeout required
-go test ./integration -timeout 45m
-```
-
-**Timeout Guidelines by Test Type**:
-- **Basic functionality tests**: `--timeout=900s` (15 minutes minimum)
-- **Route/ACL tests**: `--timeout=1200s` (20 minutes)
-- **HA/failover tests**: `--timeout=1800s` (30 minutes)  
-- **Long-running tests**: `--timeout=2100s` (35 minutes)
-- **Full test suite**: `-timeout 45m` (45 minutes)
-
-**NEVER do this**:
-```bash
-# ❌ FORBIDDEN: Never use bash timeout command
-timeout 300 go run ./cmd/hi run "TestName"
-
-# ❌ FORBIDDEN: Too short timeout will cause failures
-go run ./cmd/hi run "TestName" --timeout=60s
-```
-
-### Test Categories and Timing Expectations
-- **Fast tests** (<2 min): Basic functionality, CLI operations
-- **Medium tests** (2-5 min): Route management, ACL validation
-- **Slow tests** (5+ min): Node expiration, HA failover
-- **Long-running tests** (10+ min): `TestNodeOnlineStatus` runs for 12 minutes
-
-**CRITICAL**: Only ONE test can run at a time due to Docker port conflicts and resource constraints.
-
-## Test Artifacts and Log Analysis
-
-### Artifact Structure
-All test runs save comprehensive artifacts to `control_logs/TIMESTAMP-ID/`:
-```
-control_logs/20250713-213106-iajsux/
-├── hs-testname-abc123.stderr.log     # Headscale server error logs
-├── hs-testname-abc123.stdout.log     # Headscale server output logs
-├── hs-testname-abc123.db             # Database snapshot for post-mortem
-├── hs-testname-abc123_metrics.txt    # Prometheus metrics dump
-├── hs-testname-abc123-mapresponses/  # Protocol-level debug data
-├── ts-client-xyz789.stderr.log       # Tailscale client error logs
-├── ts-client-xyz789.stdout.log       # Tailscale client output logs
-└── ts-client-xyz789_status.json      # Client network status dump
-```
-
-### Log Analysis Priority Order
-When tests fail, examine artifacts in this specific order:
-
-1. **Headscale server stderr logs** (`hs-*.stderr.log`): Look for errors, panics, database issues, policy evaluation failures
-2. **Tailscale client stderr logs** (`ts-*.stderr.log`): Check for authentication failures, network connectivity issues
-3. **MapResponse JSON files**: Protocol-level debugging for network map generation issues
-4. **Client status dumps** (`*_status.json`): Network state and peer connectivity information
-5. **Database snapshots** (`.db` files): For data consistency and state persistence issues
-
-## Common Failure Patterns and Root Cause Analysis
-
-### CRITICAL MINDSET: Code Issues vs Infrastructure Issues
-
-**⚠️ IMPORTANT**: When tests fail, it is ALMOST ALWAYS a code issue with Headscale, NOT infrastructure problems. Do not immediately blame disk space, Docker issues, or timing unless you have thoroughly investigated the actual error logs first.
-
-### Systematic Debugging Process
-
-1. **Read the actual error message**: Don't assume - read the stderr logs completely
-2. **Check Headscale server logs first**: Most issues originate from server-side logic
-3. **Verify client connectivity**: Only after ruling out server issues
-4. **Check timing patterns**: Use proper `EventuallyWithT` patterns
-5. **Infrastructure as last resort**: Only blame infrastructure after code analysis
-
-### Real Failure Patterns
-
-#### 1. Timing Issues (Common but fixable)
-```go
-// ❌ Wrong: Immediate assertions after async operations
-client.Execute([]string{"tailscale", "set", "--advertise-routes=10.0.0.0/24"})
-nodes, _ := headscale.ListNodes()
-require.Len(t, nodes[0].GetAvailableRoutes(), 1) // WILL FAIL
-
-// ✅ Correct: Wait for async operations
-client.Execute([]string{"tailscale", "set", "--advertise-routes=10.0.0.0/24"})
-require.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes[0].GetAvailableRoutes(), 1)
-}, 10*time.Second, 100*time.Millisecond, "route should be advertised")
-```
-
-**Timeout Guidelines**:
-- Route operations: 3-5 seconds
-- Node state changes: 5-10 seconds
-- Complex scenarios: 10-15 seconds
-- Policy recalculation: 5-10 seconds
-
-#### 2. NodeStore Synchronization Issues
-Route advertisements must propagate through poll requests (`poll.go:420`). NodeStore updates happen at specific synchronization points after Hostinfo changes.
-
-#### 3. Test Data Management Issues
-```go
-// ❌ Wrong: Assuming array ordering
-require.Len(t, nodes[0].GetAvailableRoutes(), 1)
-
-// ✅ Correct: Identify nodes by properties
-expectedRoutes := map[string]string{"1": "10.33.0.0/16"}
-for _, node := range nodes {
-    nodeIDStr := fmt.Sprintf("%d", node.GetId())
-    if route, shouldHaveRoute := expectedRoutes[nodeIDStr]; shouldHaveRoute {
-        // Test the specific node that should have the route
-    }
-}
-```
-
-#### 4. Database Backend Differences
-SQLite vs PostgreSQL have different timing characteristics:
-- Use `--postgres` flag for database-intensive tests
-- PostgreSQL generally has more consistent timing
-- Some race conditions only appear with specific backends
-
-## Resource Management and Cleanup
-
-### Disk Space Management
-Tests consume significant disk space (~100MB per run):
-```bash
-# Check available space before running tests
-df -h
-
-# Clean up test artifacts periodically
-rm -rf control_logs/older-timestamp-dirs/
-
-# Clean Docker resources
-docker system prune -f
-docker volume prune -f
-```
-
-### Container Cleanup
-- Successful tests clean up automatically
-- Failed tests may leave containers running
-- Manually clean if needed: `docker ps -a` and `docker rm -f <containers>`
-
-## Advanced Debugging Techniques
-
-### Protocol-Level Debugging
-MapResponse JSON files in `control_logs/*/hs-*-mapresponses/` contain:
-- Network topology as sent to clients
-- Peer relationships and visibility
-- Route distribution and primary route selection
-- Policy evaluation results
-
-### Database State Analysis
-Use the database snapshots for post-mortem analysis:
-```bash
-# SQLite examination
-sqlite3 control_logs/TIMESTAMP/hs-*.db
-.tables
-.schema nodes
-SELECT * FROM nodes WHERE name LIKE '%problematic%';
-```
-
-### Performance Analysis
-Prometheus metrics dumps show:
-- Request latencies and error rates
-- NodeStore operation timing
-- Database query performance
-- Memory usage patterns
-
-## Test Development and Quality Guidelines
-
-### Proper Test Patterns
-```go
-// Always use EventuallyWithT for async operations
-require.EventuallyWithT(t, func(c *assert.CollectT) {
-    // Test condition that may take time to become true
-}, timeout, interval, "descriptive failure message")
-
-// Handle node identification correctly
-var targetNode *v1.Node
-for _, node := range nodes {
-    if node.GetName() == expectedNodeName {
-        targetNode = node
-        break
-    }
-}
-require.NotNil(t, targetNode, "should find expected node")
-```
-
-### Quality Validation Checklist
-- ✅ Tests use `EventuallyWithT` for asynchronous operations
-- ✅ Tests don't rely on array ordering for node identification
-- ✅ Proper cleanup and resource management
-- ✅ Tests handle both success and failure scenarios
-- ✅ Timing assumptions are realistic for operations being tested
-- ✅ Error messages are descriptive and actionable
-
-## Real-World Test Failure Patterns from HA Debugging
-
-### Infrastructure vs Code Issues - Detailed Examples
-
-**INFRASTRUCTURE FAILURES (Rare but Real)**:
-1. **DNS Resolution in Auth Tests**: `failed to resolve "hs-pingallbyip-jax97k": no DNS fallback candidates remain`
-   - **Pattern**: Client containers can't resolve headscale server hostname during logout
-   - **Detection**: Error messages specifically mention DNS/hostname resolution
-   - **Solution**: Docker networking reset, not code changes
-
-2. **Container Creation Timeouts**: Test gets stuck during client container setup
-   - **Pattern**: Tests hang indefinitely at container startup phase
-   - **Detection**: No progress in logs for >2 minutes during initialization
-   - **Solution**: `docker system prune -f` and retry
-
-3. **Docker Port Conflicts**: Multiple tests trying to use same ports
-   - **Pattern**: "bind: address already in use" errors
-   - **Detection**: Port binding failures in Docker logs
-   - **Solution**: Only run ONE test at a time
-
-**CODE ISSUES (99% of failures)**:
-1. **Route Approval Process Failures**: Routes not getting approved when they should be
-   - **Pattern**: Tests expecting approved routes but finding none
-   - **Detection**: `SubnetRoutes()` returns empty when `AnnouncedRoutes()` shows routes
-   - **Root Cause**: Auto-approval logic bugs, policy evaluation issues
-
-2. **NodeStore Synchronization Issues**: State updates not propagating correctly
-   - **Pattern**: Route changes not reflected in NodeStore or Primary Routes
-   - **Detection**: Logs show route announcements but no tracking updates
-   - **Root Cause**: Missing synchronization points in `poll.go:420` area
-
-3. **HA Failover Architecture Issues**: Routes removed when nodes go offline
-   - **Pattern**: `TestHASubnetRouterFailover` fails because approved routes disappear
-   - **Detection**: Routes available on online nodes but lost when nodes disconnect
-   - **Root Cause**: Conflating route approval with node connectivity
-
-### Critical Test Environment Setup
-
-**Pre-Test Cleanup (MANDATORY)**:
-```bash
-# ALWAYS run this before each test
-rm -rf control_logs/202507*
-docker system prune -f
-df -h  # Verify sufficient disk space
-```
-
-**Environment Verification**:
-```bash
-# Verify system readiness
-go run ./cmd/hi doctor
-
-# Check for running containers that might conflict
-docker ps
-```
-
-### Specific Test Categories and Known Issues
-
-#### Route-Related Tests (Primary Focus)
-```bash
-# Core route functionality - these should work first
-# Note: Generous timeouts are required for reliable execution
-go run ./cmd/hi run "TestSubnetRouteACL" --timeout=1200s
-go run ./cmd/hi run "TestAutoApproveMultiNetwork" --timeout=1800s
-go run ./cmd/hi run "TestHASubnetRouterFailover" --timeout=1800s
-```
-
-**Common Route Test Patterns**:
-- Tests validate route announcement, approval, and distribution workflows
-- Route state changes are asynchronous - may need `EventuallyWithT` wrappers
-- Route approval must respect ACL policies - test expectations encode security requirements
-- HA tests verify route persistence during node connectivity changes
-
-#### Authentication Tests (Infrastructure-Prone)
-```bash
-# These tests are more prone to infrastructure issues
-# Require longer timeouts due to auth flow complexity
-go run ./cmd/hi run "TestAuthKeyLogoutAndReloginSameUser" --timeout=1200s
-go run ./cmd/hi run "TestAuthWebFlowLogoutAndRelogin" --timeout=1200s
-go run ./cmd/hi run "TestOIDCExpireNodesBasedOnTokenExpiry" --timeout=1800s
-```
-
-**Common Auth Test Infrastructure Failures**:
-- DNS resolution during logout operations
-- Container creation timeouts
-- HTTP/2 stream errors (often symptoms, not root cause)
-
-### Security-Critical Debugging Rules
-
-**❌ FORBIDDEN CHANGES (Security & Test Integrity)**:
-1. **Never change expected test outputs** - Tests define correct behavior contracts
-   - Changing `require.Len(t, routes, 3)` to `require.Len(t, routes, 2)` because test fails
-   - Modifying expected status codes, node counts, or route counts
-   - Removing assertions that are "inconvenient"
-   - **Why forbidden**: Test expectations encode business requirements and security policies
-
-2. **Never bypass security mechanisms** - Security must never be compromised for convenience
-   - Using `AnnouncedRoutes()` instead of `SubnetRoutes()` in production code
-   - Skipping authentication or authorization checks
-   - **Why forbidden**: Security bypasses create vulnerabilities in production
-
-3. **Never reduce test coverage** - Tests prevent regressions
-   - Removing test cases or assertions
-   - Commenting out "problematic" test sections
-   - **Why forbidden**: Reduced coverage allows bugs to slip through
-
-**✅ ALLOWED CHANGES (Timing & Observability)**:
-1. **Fix timing issues with proper async patterns**
-   ```go
-   // ✅ GOOD: Add EventuallyWithT for async operations
-   require.EventuallyWithT(t, func(c *assert.CollectT) {
-       nodes, err := headscale.ListNodes()
-       assert.NoError(c, err)
-       assert.Len(c, nodes, expectedCount) // Keep original expectation
-   }, 10*time.Second, 100*time.Millisecond, "nodes should reach expected count")
-   ```
-   - **Why allowed**: Fixes race conditions without changing business logic
-
-2. **Add MORE observability and debugging**
-   - Additional logging statements
-   - More detailed error messages
-   - Extra assertions that verify intermediate states
-   - **Why allowed**: Better observability helps debug without changing behavior
-
-3. **Improve test documentation**
-   - Add godoc comments explaining test purpose and business logic
-   - Document timing requirements and async behavior
-   - **Why encouraged**: Helps future maintainers understand intent
-
-### Advanced Debugging Workflows
-
-#### Route Tracking Debug Flow
-```bash
-# Run test with detailed logging and proper timeout
-go run ./cmd/hi run "TestSubnetRouteACL" --timeout=1200s > test_output.log 2>&1
-
-# Check route approval process
-grep -E "(auto-approval|ApproveRoutesWithPolicy|PolicyManager)" test_output.log
-
-# Check route tracking
-tail -50 control_logs/*/hs-*.stderr.log | grep -E "(announced|tracking|SetNodeRoutes)"
-
-# Check for security violations
-grep -E "(AnnouncedRoutes.*SetNodeRoutes|bypass.*approval)" test_output.log
-```
-
-#### HA Failover Debug Flow
-```bash
-# Test HA failover specifically with adequate timeout
-go run ./cmd/hi run "TestHASubnetRouterFailover" --timeout=1800s
-
-# Check route persistence during disconnect
-grep -E "(Disconnect|NodeWentOffline|PrimaryRoutes)" control_logs/*/hs-*.stderr.log
-
-# Verify routes don't disappear inappropriately
-grep -E "(removing.*routes|SetNodeRoutes.*empty)" control_logs/*/hs-*.stderr.log
-```
-
-### Test Result Interpretation Guidelines
-
-#### Success Patterns to Look For
-- `"updating node routes for tracking"` in logs
-- Routes appearing in `announcedRoutes` logs
-- Proper `ApproveRoutesWithPolicy` calls for auto-approval
-- Routes persisting through node connectivity changes (HA tests)
-
-#### Failure Patterns to Investigate
-- `SubnetRoutes()` returning empty when `AnnouncedRoutes()` has routes
-- Routes disappearing when nodes go offline (HA architectural issue)
-- Missing `EventuallyWithT` causing timing race conditions
-- Security bypass attempts using wrong route methods
-
-### Critical Testing Methodology
-
-**Phase-Based Testing Approach**:
-1. **Phase 1**: Core route tests (ACL, auto-approval, basic functionality)
-2. **Phase 2**: HA and complex route scenarios
-3. **Phase 3**: Auth tests (infrastructure-sensitive, test last)
-
-**Per-Test Process**:
-1. Clean environment before each test
-2. Monitor logs for route tracking and approval messages
-3. Check artifacts in `control_logs/` if test fails
-4. Focus on actual error messages, not assumptions
-5. Document results and patterns discovered
-
-## Test Documentation and Code Quality Standards
-
-### Adding Missing Test Documentation
-When you understand a test's purpose through debugging, always add comprehensive godoc:
-
-```go
-// TestSubnetRoutes validates the complete subnet route lifecycle including
-// advertisement from clients, policy-based approval, and distribution to peers.
-// This test ensures that route security policies are properly enforced and that
-// only approved routes are distributed to the network.
-//
-// The test verifies:
-// - Route announcements are received and tracked
-// - ACL policies control route approval correctly  
-// - Only approved routes appear in peer network maps
-// - Route state persists correctly in the database
-func TestSubnetRoutes(t *testing.T) {
-    // Test implementation...
-}
-```
-
-**Why add documentation**: Future maintainers need to understand business logic and security requirements encoded in tests.
-
-### Comment Guidelines - Focus on WHY, Not WHAT
-
-```go
-// ✅ GOOD: Explains reasoning and business logic
-// Wait for route propagation because NodeStore updates are asynchronous
-// and happen after poll requests complete processing
-require.EventuallyWithT(t, func(c *assert.CollectT) {
-    // Check that security policies are enforced...
-}, timeout, interval, "route approval must respect ACL policies")
-
-// ❌ BAD: Just describes what the code does
-// Wait for routes
-require.EventuallyWithT(t, func(c *assert.CollectT) {
-    // Get routes and check length
-}, timeout, interval, "checking routes")
-```
-
-**Why focus on WHY**: Helps maintainers understand architectural decisions and security requirements.
-
-## EventuallyWithT Pattern for External Calls
-
-### Overview
-EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
-
-### External Calls That Must Be Wrapped
-The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
-- `headscale.ListNodes()` - Queries server state
-- `client.Status()` - Gets client network status
-- `client.Curl()` - Makes HTTP requests through the network
-- `client.Traceroute()` - Performs network diagnostics
-- `client.Execute()` when running commands that query state
-- Any operation that reads from the headscale server or tailscale client
-
-### Five Key Rules for EventuallyWithT
-
-1. **One External Call Per EventuallyWithT Block**
-   - Each EventuallyWithT should make ONE external call (e.g., ListNodes OR Status)
-   - Related assertions based on that single call can be grouped together
-   - Unrelated external calls must be in separate EventuallyWithT blocks
-
-2. **Variable Scoping**
-   - Declare variables that need to be shared across EventuallyWithT blocks at function scope
-   - Use `=` for assignment inside EventuallyWithT, not `:=` (unless the variable is only used within that block)
-   - Variables declared with `:=` inside EventuallyWithT are not accessible outside
-
-3. **No Nested EventuallyWithT**
-   - NEVER put an EventuallyWithT inside another EventuallyWithT
-   - This is a critical anti-pattern that must be avoided
-
-4. **Use CollectT for Assertions**
-   - Inside EventuallyWithT, use `assert` methods with the CollectT parameter
-   - Helper functions called within EventuallyWithT must accept `*assert.CollectT`
-
-5. **Descriptive Messages**
-   - Always provide a descriptive message as the last parameter
-   - Message should explain what condition is being waited for
-
-### Correct Pattern Examples
-
-```go
-// CORRECT: Single external call with related assertions
-var nodes []*v1.Node
-var err error
-
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err = headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 2)
-    // These assertions are all based on the ListNodes() call
-    requireNodeRouteCountWithCollect(c, nodes[0], 2, 2, 2)
-    requireNodeRouteCountWithCollect(c, nodes[1], 1, 1, 1)
-}, 10*time.Second, 500*time.Millisecond, "nodes should have expected route counts")
-
-// CORRECT: Separate EventuallyWithT for different external call
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    // All these assertions are based on the single Status() call
-    for _, peerKey := range status.Peers() {
-        peerStatus := status.Peer[peerKey]
-        requirePeerSubnetRoutesWithCollect(c, peerStatus, expectedPrefixes)
-    }
-}, 10*time.Second, 500*time.Millisecond, "client should see expected routes")
-
-// CORRECT: Variable scoping for sharing between blocks
-var routeNode *v1.Node
-var nodeKey key.NodePublic
-
-// First EventuallyWithT to get the node
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    
-    for _, node := range nodes {
-        if node.GetName() == "router" {
-            routeNode = node
-            nodeKey, _ = key.ParseNodePublicUntyped(mem.S(node.GetNodeKey()))
-            break
-        }
-    }
-    assert.NotNil(c, routeNode, "should find router node")
-}, 10*time.Second, 100*time.Millisecond, "router node should exist")
-
-// Second EventuallyWithT using the nodeKey from first block
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    
-    peerStatus, ok := status.Peer[nodeKey]
-    assert.True(c, ok, "peer should exist in status")
-    requirePeerSubnetRoutesWithCollect(c, peerStatus, expectedPrefixes)
-}, 10*time.Second, 100*time.Millisecond, "routes should be visible to client")
-```
-
-### Incorrect Patterns to Avoid
-
-```go
-// INCORRECT: Multiple unrelated external calls in same EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    // First external call
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 2)
-    
-    // Second unrelated external call - WRONG!
-    status, err := client.Status()
-    assert.NoError(c, err)
-    assert.NotNil(c, status)
-}, 10*time.Second, 500*time.Millisecond, "mixed operations")
-
-// INCORRECT: Nested EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    
-    // NEVER do this!
-    assert.EventuallyWithT(t, func(c2 *assert.CollectT) {
-        status, _ := client.Status()
-        assert.NotNil(c2, status)
-    }, 5*time.Second, 100*time.Millisecond, "nested")
-}, 10*time.Second, 500*time.Millisecond, "outer")
-
-// INCORRECT: Variable scoping error
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes() // This shadows outer 'nodes' variable
-    assert.NoError(c, err)
-}, 10*time.Second, 500*time.Millisecond, "get nodes")
-
-// This will fail - nodes is nil because := created a new variable inside the block
-require.Len(t, nodes, 2) // COMPILATION ERROR or nil pointer
-
-// INCORRECT: Not wrapping external calls
-nodes, err := headscale.ListNodes() // External call not wrapped!
-require.NoError(t, err)
-```
-
-### Helper Functions for EventuallyWithT
-
-When creating helper functions for use within EventuallyWithT:
-
-```go
-// Helper function that accepts CollectT
-func requireNodeRouteCountWithCollect(c *assert.CollectT, node *v1.Node, available, approved, primary int) {
-    assert.Len(c, node.GetAvailableRoutes(), available, "available routes for node %s", node.GetName())
-    assert.Len(c, node.GetApprovedRoutes(), approved, "approved routes for node %s", node.GetName())
-    assert.Len(c, node.GetPrimaryRoutes(), primary, "primary routes for node %s", node.GetName())
-}
-
-// Usage within EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    requireNodeRouteCountWithCollect(c, nodes[0], 2, 2, 2)
-}, 10*time.Second, 500*time.Millisecond, "route counts should match expected")
-```
-
-### Operations That Must NOT Be Wrapped
-
-**CRITICAL**: The following operations are **blocking/mutating operations** that change state and MUST NOT be wrapped in EventuallyWithT:
-- `tailscale set` commands (e.g., `--advertise-routes`, `--accept-routes`)
-- `headscale.ApproveRoute()` - Approves routes on server
-- `headscale.CreateUser()` - Creates users
-- `headscale.CreatePreAuthKey()` - Creates authentication keys
-- `headscale.RegisterNode()` - Registers new nodes
-- Any `client.Execute()` that modifies configuration
-- Any operation that creates, updates, or deletes resources
-
-These operations:
-1. Complete synchronously or fail immediately
-2. Should not be retried automatically
-3. Need explicit error handling with `require.NoError()`
-
-### Correct Pattern for Blocking Operations
-
-```go
-// CORRECT: Blocking operation NOT wrapped
-status := client.MustStatus()
-command := []string{"tailscale", "set", "--advertise-routes=" + expectedRoutes[string(status.Self.ID)]}
-_, _, err = client.Execute(command)
-require.NoErrorf(t, err, "failed to advertise route: %s", err)
-
-// Then wait for the result with EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Contains(c, nodes[0].GetAvailableRoutes(), expectedRoutes[string(status.Self.ID)])
-}, 10*time.Second, 100*time.Millisecond, "route should be advertised")
-
-// INCORRECT: Blocking operation wrapped (DON'T DO THIS)
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    _, _, err = client.Execute([]string{"tailscale", "set", "--advertise-routes=10.0.0.0/24"})
-    assert.NoError(c, err) // This might retry the command multiple times!
-}, 10*time.Second, 100*time.Millisecond, "advertise routes")
-```
-
-### Assert vs Require Pattern
-
-When working within EventuallyWithT blocks where you need to prevent panics:
-
-```go
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    
-    // For array bounds - use require with t to prevent panic
-    assert.Len(c, nodes, 6)  // Test expectation
-    require.GreaterOrEqual(t, len(nodes), 3, "need at least 3 nodes to avoid panic")
-    
-    // For nil pointer access - use require with t before dereferencing
-    assert.NotNil(c, srs1PeerStatus.PrimaryRoutes)  // Test expectation
-    require.NotNil(t, srs1PeerStatus.PrimaryRoutes, "primary routes must be set to avoid panic")
-    assert.Contains(c,
-        srs1PeerStatus.PrimaryRoutes.AsSlice(),
-        pref,
-    )
-}, 5*time.Second, 200*time.Millisecond, "checking route state")
-```
-
-**Key Principle**: 
-- Use `assert` with `c` (*assert.CollectT) for test expectations that can be retried
-- Use `require` with `t` (*testing.T) for MUST conditions that prevent panics
-- Within EventuallyWithT, both are available - choose based on whether failure would cause a panic
-
-### Common Scenarios
-
-1. **Waiting for route advertisement**:
-```go
-client.Execute([]string{"tailscale", "set", "--advertise-routes=10.0.0.0/24"})
-
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Contains(c, nodes[0].GetAvailableRoutes(), "10.0.0.0/24")
-}, 10*time.Second, 100*time.Millisecond, "route should be advertised")
-```
-
-2. **Checking client sees routes**:
-```go
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    
-    // Check all peers have expected routes
-    for _, peerKey := range status.Peers() {
-        peerStatus := status.Peer[peerKey]
-        assert.Contains(c, peerStatus.AllowedIPs, expectedPrefix)
-    }
-}, 10*time.Second, 100*time.Millisecond, "all peers should see route")
-```
-
-3. **Sequential operations**:
-```go
-// First wait for node to appear
-var nodeID uint64
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 1)
-    nodeID = nodes[0].GetId()
-}, 10*time.Second, 100*time.Millisecond, "node should register")
-
-// Then perform operation
-_, err := headscale.ApproveRoute(nodeID, "10.0.0.0/24")
-require.NoError(t, err)
-
-// Then wait for result
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Contains(c, nodes[0].GetApprovedRoutes(), "10.0.0.0/24")
-}, 10*time.Second, 100*time.Millisecond, "route should be approved")
-```
-
-## Your Core Responsibilities
-
-1. **Test Execution Strategy**: Execute integration tests with appropriate configurations, understanding when to use `--postgres` and timing requirements for different test categories. Follow phase-based testing approach prioritizing route tests.
-   - **Why this priority**: Route tests are less infrastructure-sensitive and validate core security logic
-
-2. **Systematic Test Analysis**: When tests fail, systematically examine artifacts starting with Headscale server logs, then client logs, then protocol data. Focus on CODE ISSUES first (99% of cases), not infrastructure. Use real-world failure patterns to guide investigation.
-   - **Why this approach**: Most failures are logic bugs, not environment issues - efficient debugging saves time
-
-3. **Timing & Synchronization Expertise**: Understand asynchronous Headscale operations, particularly route advertisements, NodeStore synchronization at `poll.go:420`, and policy propagation. Fix timing with `EventuallyWithT` while preserving original test expectations.
-   - **Why preserve expectations**: Test assertions encode business requirements and security policies
-   - **Key Pattern**: Apply the EventuallyWithT pattern correctly for all external calls as documented above
-
-4. **Root Cause Analysis**: Distinguish between actual code regressions (route approval logic, HA failover architecture), timing issues requiring `EventuallyWithT` patterns, and genuine infrastructure problems (DNS, Docker, container issues).
-   - **Why this distinction matters**: Different problem types require completely different solution approaches
-   - **EventuallyWithT Issues**: Often manifest as flaky tests or immediate assertion failures after async operations
-
-5. **Security-Aware Quality Validation**: Ensure tests properly validate end-to-end functionality with realistic timing expectations and proper error handling. Never suggest security bypasses or test expectation changes. Add comprehensive godoc when you understand test business logic.
-   - **Why security focus**: Integration tests are the last line of defense against security regressions
-   - **EventuallyWithT Usage**: Proper use prevents race conditions without weakening security assertions
-
-**CRITICAL PRINCIPLE**: Test expectations are sacred contracts that define correct system behavior. When tests fail, fix the code to match the test, never change the test to match broken code. Only timing and observability improvements are allowed - business logic expectations are immutable.
-
-**EventuallyWithT PRINCIPLE**: Every external call to headscale server or tailscale client must be wrapped in EventuallyWithT. Follow the five key rules strictly: one external call per block, proper variable scoping, no nesting, use CollectT for assertions, and provide descriptive messages.
-
-**Remember**: Test failures are usually code issues in Headscale that need to be fixed, not infrastructure problems to be ignored. Use the specific debugging workflows and failure patterns documented above to efficiently identify root causes. Infrastructure issues have very specific signatures - everything else is code-related.

.editorconfig

@@ -0,0 +1,16 @@
+root = true
+
+[*]
+charset = utf-8
+end_of_line = lf
+indent_size = 2
+indent_style = space
+insert_final_newline = true
+trim_trailing_whitespace = true
+max_line_length = 120
+
+[*.go]
+indent_style = tab
+
+[Makefile]
+indent_style = tab

.github/ISSUE_TEMPLATE/bug_report.yaml

@@ -6,8 +6,7 @@ body:
   - type: checkboxes
     attributes:
       label: Is this a support request?
-      description:
-        This issue tracker is for bugs and feature requests only. If you need
+      description: This issue tracker is for bugs and feature requests only. If you need
         help, please use ask in our Discord community
       options:
         - label: This is not a support request
@@ -15,8 +14,7 @@ body:
   - type: checkboxes
     attributes:
       label: Is there an existing issue for this?
-      description:
-        Please search to see if an issue already exists for the bug you
+      description: Please search to see if an issue already exists for the bug you
         encountered.
       options:
         - label: I have searched the existing issues

.github/ISSUE_TEMPLATE/config.yml

@@ -3,9 +3,9 @@ blank_issues_enabled: false
 
 # Contact links
 contact_links:
-  - name: "headscale usage documentation"
-    url: "https://github.com/juanfont/headscale/blob/main/docs"
-    about: "Find documentation about how to configure and run headscale."
   - name: "headscale Discord community"
-    url: "https://discord.gg/xGj2TuqyxY"
+    url: "https://discord.gg/c84AZQhmpx"
     about: "Please ask and answer questions about usage of headscale here."
+  - name: "headscale usage documentation"
+    url: "https://headscale.net/"
+    about: "Find documentation about how to configure and run headscale."

.github/label-response/needs-more-info.md

@@ -0,0 +1,80 @@
+Thank you for taking the time to report this issue.
+
+To help us investigate and resolve this, we need more information. Please provide the following:
+
+> [!TIP]
+> Most issues turn out to be configuration errors rather than bugs. We encourage you to discuss your problem in our [Discord community](https://discord.gg/c84AZQhmpx) **before** opening an issue. The community can often help identify misconfigurations quickly, saving everyone time.
+
+## Required Information
+
+### Environment Details
+
+- **Headscale version**: (run `headscale version`)
+- **Tailscale client version**: (run `tailscale version`)
+- **Operating System**: (e.g., Ubuntu 24.04, macOS 14, Windows 11)
+- **Deployment method**: (binary, Docker, Kubernetes, etc.)
+- **Reverse proxy**: (if applicable: nginx, Traefik, Caddy, etc. - include configuration)
+
+### Debug Information
+
+Please follow our [Debugging and Troubleshooting Guide](https://headscale.net/stable/ref/debug/) and provide:
+
+1. **Client netmap dump** (from affected Tailscale client):
+
+   ```bash
+   tailscale debug netmap > netmap.json
+   ```
+
+2. **Client status dump** (from affected Tailscale client):
+
+   ```bash
+   tailscale status --json > status.json
+   ```
+
+3. **Tailscale client logs** (if experiencing client issues):
+
+   ```bash
+   tailscale debug daemon-logs
+   ```
+
+   > [!IMPORTANT]
+   > We need logs from **multiple nodes** to understand the full picture:
+   >
+   > - The node(s) initiating connections
+   > - The node(s) being connected to
+   >
+   > Without logs from both sides, we cannot diagnose connectivity issues.
+
+4. **Headscale server logs** with `log.level: trace` enabled
+
+5. **Headscale configuration** (with sensitive values redacted - see rules below)
+
+6. **ACL/Policy configuration** (if using ACLs)
+
+7. **Proxy/Docker configuration** (if applicable - nginx.conf, docker-compose.yml, Traefik config, etc.)
+
+## Formatting Requirements
+
+- **Attach long files** - Do not paste large logs or configurations inline. Use GitHub file attachments or GitHub Gists.
+- **Use proper Markdown** - Format code blocks, logs, and configurations with appropriate syntax highlighting.
+- **Structure your response** - Use the headings above to organize your information clearly.
+
+## Redaction Rules
+
+> [!CAUTION]
+> **Replace, do not remove.** Removing information makes debugging impossible.
+
+When redacting sensitive information:
+
+- ✅ **Replace consistently** - If you change `alice@company.com` to `user1@example.com`, use `user1@example.com` everywhere (logs, config, policy, etc.)
+- ✅ **Use meaningful placeholders** - `user1@example.com`, `bob@example.com`, `my-secret-key` are acceptable
+- ❌ **Never remove information** - Gaps in data prevent us from correlating events across logs
+- ❌ **Never redact IP addresses** - We need the actual IPs to trace network paths and identify issues
+
+**If redaction rules are not followed, we will be unable to debug the issue and will have to close it.**
+
+---
+
+**Note:** This issue will be automatically closed in 3 days if no additional information is provided. Once you reply with the requested information, the `needs-more-info` label will be removed automatically.
+
+If you need help gathering this information, please visit our [Discord community](https://discord.gg/c84AZQhmpx).

.github/label-response/support-request.md

@@ -0,0 +1,15 @@
+Thank you for reaching out.
+
+This issue tracker is used for **bug reports and feature requests** only. Your question appears to be a support or configuration question rather than a bug report.
+
+For help with setup, configuration, or general questions, please visit our [Discord community](https://discord.gg/c84AZQhmpx) where the community and maintainers can assist you in real-time.
+
+**Before posting in Discord, please check:**
+
+- [Documentation](https://headscale.net/)
+- [FAQ](https://headscale.net/stable/faq/)
+- [Debugging and Troubleshooting Guide](https://headscale.net/stable/ref/debug/)
+
+If after troubleshooting you determine this is actually a bug, please open a new issue with the required debug information from the troubleshooting guide.
+
+This issue has been automatically closed.

.github/workflows/build.yml

@@ -5,8 +5,6 @@ on:
     branches:
       - main
   pull_request:
-    branches:
-      - main
 
 concurrency:
   group: ${{ github.workflow }}-$${{ github.head_ref || github.run_id }}
@@ -17,7 +15,7 @@ jobs:
     runs-on: ubuntu-latest
     permissions: write-all
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
       - name: Get changed files
@@ -31,13 +29,12 @@ jobs:
               - '**/*.go'
               - 'integration_test/'
               - 'config-example.yaml'
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 
@@ -57,7 +54,7 @@ jobs:
           exit $BUILD_STATUS
 
       - name: Nix gosum diverging
-        uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
+        uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8.0.0
         if: failure() && steps.build.outcome == 'failure'
         with:
           github-token: ${{secrets.GITHUB_TOKEN}}
@@ -69,7 +66,7 @@ jobs:
               body: 'Nix build failed with wrong gosum, please update "vendorSha256" (${{ steps.build.outputs.OLD_HASH }}) for the "headscale" package in flake.nix with the new SHA: ${{ steps.build.outputs.NEW_HASH }}'
             })
 
-      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         if: steps.changed-files.outputs.files == 'true'
         with:
           name: headscale-linux
@@ -84,22 +81,20 @@ jobs:
           - "GOARCH=arm64 GOOS=darwin"
           - "GOARCH=amd64 GOOS=darwin"
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 
       - name: Run go cross compile
         env:
           CGO_ENABLED: 0
-        run:
-          env ${{ matrix.env }} nix develop --command -- go build -o "headscale"
+        run: env ${{ matrix.env }} nix develop --command -- go build -o "headscale"
           ./cmd/headscale
-      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: "headscale-${{ matrix.env }}"
           path: "headscale"

.github/workflows/check-generated.yml

@@ -16,7 +16,7 @@ jobs:
   check-generated:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
       - name: Get changed files
@@ -31,7 +31,7 @@ jobs:
               - '**/*.proto'
               - 'buf.gen.yaml'
               - 'tools/**'
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'

.github/workflows/check-tests.yaml

@@ -10,7 +10,7 @@ jobs:
   check-tests:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
       - name: Get changed files
@@ -24,13 +24,12 @@ jobs:
               - '**/*.go'
               - 'integration_test/'
               - 'config-example.yaml'
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 

.github/workflows/container-main.yml

@@ -0,0 +1,112 @@
+---
+name: Build (main)
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - "*.nix"
+      - "go.*"
+      - "**/*.go"
+      - ".github/workflows/container-main.yml"
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  container:
+    if: github.repository == 'juanfont/headscale'
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - name: Checkout
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+
+      - name: Login to DockerHub
+        uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Login to GHCR
+        uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
+      - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
+        with:
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+            '**/flake.lock') }}
+          restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
+
+      - name: Set commit timestamp
+        run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> $GITHUB_ENV
+
+      - name: Build and push to GHCR
+        env:
+          KO_DOCKER_REPO: ghcr.io/juanfont/headscale
+          KO_DEFAULTBASEIMAGE: gcr.io/distroless/base-debian13
+          CGO_ENABLED: "0"
+        run: |
+          nix develop --command -- ko build \
+            --bare \
+            --platform=linux/amd64,linux/arm64 \
+            --tags=main-${GITHUB_SHA::7},development \
+            ./cmd/headscale
+
+      - name: Push to Docker Hub
+        env:
+          KO_DOCKER_REPO: headscale/headscale
+          KO_DEFAULTBASEIMAGE: gcr.io/distroless/base-debian13
+          CGO_ENABLED: "0"
+        run: |
+          nix develop --command -- ko build \
+            --bare \
+            --platform=linux/amd64,linux/arm64 \
+            --tags=main-${GITHUB_SHA::7},development \
+            ./cmd/headscale
+
+  binaries:
+    if: github.repository == 'juanfont/headscale'
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        include:
+          - goos: linux
+            goarch: amd64
+          - goos: linux
+            goarch: arm64
+          - goos: darwin
+            goarch: amd64
+          - goos: darwin
+            goarch: arm64
+    steps:
+      - name: Checkout
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
+      - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
+        with:
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+            '**/flake.lock') }}
+          restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
+
+      - name: Build binary
+        env:
+          CGO_ENABLED: "0"
+          GOOS: ${{ matrix.goos }}
+          GOARCH: ${{ matrix.goarch }}
+        run: nix develop --command -- go build -o headscale ./cmd/headscale
+
+      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: headscale-${{ matrix.goos }}-${{ matrix.goarch }}
+          path: headscale

.github/workflows/docs-deploy.yml

@@ -21,15 +21,15 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 0
       - name: Install python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0
         with:
           python-version: 3.x
       - name: Setup cache
-        uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+        uses: actions/cache@a7833574556fa59680c1b7cb190c1735db73ebf0 # v5.0.0
         with:
           key: ${{ github.ref }}
           path: .cache

.github/workflows/docs-test.yml

@@ -11,13 +11,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout repository
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
       - name: Install python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        uses: actions/setup-python@83679a892e2d95755f2dac6acb0bfd1e9ac5d548 # v6.1.0
         with:
           python-version: 3.x
       - name: Setup cache
-        uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+        uses: actions/cache@a7833574556fa59680c1b7cb190c1735db73ebf0 # v5.0.0
         with:
           key: ${{ github.ref }}
           path: .cache

.github/workflows/gh-action-integration-generator.go

@@ -10,6 +10,55 @@ import (
 	"strings"
 )
 
+// testsToSplit defines tests that should be split into multiple CI jobs.
+// Key is the test function name, value is a list of subtest prefixes.
+// Each prefix becomes a separate CI job as "TestName/prefix".
+//
+// Example: TestAutoApproveMultiNetwork has subtests like:
+//   - TestAutoApproveMultiNetwork/authkey-tag-advertiseduringup-false-pol-database
+//   - TestAutoApproveMultiNetwork/webauth-user-advertiseduringup-true-pol-file
+//
+// Splitting by approver type (tag, user, group) creates 6 CI jobs with 4 tests each:
+//   - TestAutoApproveMultiNetwork/authkey-tag.* (4 tests)
+//   - TestAutoApproveMultiNetwork/authkey-user.* (4 tests)
+//   - TestAutoApproveMultiNetwork/authkey-group.* (4 tests)
+//   - TestAutoApproveMultiNetwork/webauth-tag.* (4 tests)
+//   - TestAutoApproveMultiNetwork/webauth-user.* (4 tests)
+//   - TestAutoApproveMultiNetwork/webauth-group.* (4 tests)
+//
+// This reduces load per CI job (4 tests instead of 12) to avoid infrastructure
+// flakiness when running many sequential Docker-based integration tests.
+var testsToSplit = map[string][]string{
+	"TestAutoApproveMultiNetwork": {
+		"authkey-tag",
+		"authkey-user",
+		"authkey-group",
+		"webauth-tag",
+		"webauth-user",
+		"webauth-group",
+	},
+}
+
+// expandTests takes a list of test names and expands any that need splitting
+// into multiple subtest patterns.
+func expandTests(tests []string) []string {
+	var expanded []string
+	for _, test := range tests {
+		if prefixes, ok := testsToSplit[test]; ok {
+			// This test should be split into multiple jobs.
+			// We append ".*" to each prefix because the CI runner wraps patterns
+			// with ^...$ anchors. Without ".*", a pattern like "authkey$" wouldn't
+			// match "authkey-tag-advertiseduringup-false-pol-database".
+			for _, prefix := range prefixes {
+				expanded = append(expanded, fmt.Sprintf("%s/%s.*", test, prefix))
+			}
+		} else {
+			expanded = append(expanded, test)
+		}
+	}
+	return expanded
+}
+
 func findTests() []string {
 	rgBin, err := exec.LookPath("rg")
 	if err != nil {
@@ -17,7 +66,9 @@ func findTests() []string {
 	}
 
 	args := []string{
+		"--type", "go",
 		"--regexp", "func (Test.+)\\(.*",
+		"--max-depth", "1",
 		"../../integration/",
 		"--replace", "$1",
 		"--sort", "path",
@@ -66,8 +117,11 @@ func updateYAML(tests []string, jobName string, testPath string) {
 func main() {
 	tests := findTests()
 
-	quotedTests := make([]string, len(tests))
-	for i, test := range tests {
+	// Expand tests that should be split into multiple jobs
+	expandedTests := expandTests(tests)
+
+	quotedTests := make([]string, len(expandedTests))
+	for i, test := range expandedTests {
 		quotedTests[i] = fmt.Sprintf("\"%s\"", test)
 	}
 

.github/workflows/gh-actions-updater.yaml

@@ -11,13 +11,13 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           # [Required] Access token with `workflow` scope.
           token: ${{ secrets.WORKFLOW_SECRET }}
 
       - name: Run GitHub Actions Version Updater
-        uses: saadmk11/github-actions-version-updater@64be81ba69383f81f2be476703ea6570c4c8686e # v0.8.1
+        uses: saadmk11/github-actions-version-updater@d8781caf11d11168579c8e5e94f62b068038f442 # v0.9.0
         with:
           # [Required] Access token with `workflow` scope.
           token: ${{ secrets.WORKFLOW_SECRET }}

.github/workflows/integration-test-template.yml

@@ -16,7 +16,7 @@ on:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04-arm
     env:
       # Github does not allow us to access secrets in pull requests,
       # so this env var is used to check if we have the secret or not.
@@ -28,23 +28,12 @@ jobs:
       # that triggered the build.
       HAS_TAILSCALE_SECRET: ${{ secrets.TS_OAUTH_CLIENT_ID }}
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
-      - name: Get changed files
-        id: changed-files
-        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
-        with:
-          filters: |
-            files:
-              - '*.nix'
-              - 'go.*'
-              - '**/*.go'
-              - 'integration_test/'
-              - 'config-example.yaml'
       - name: Tailscale
         if: ${{ env.HAS_TAILSCALE_SECRET }}
-        uses: tailscale/github-action@6986d2c82a91fbac2949fe01f5bab95cf21b5102 # v3.2.2
+        uses: tailscale/github-action@a392da0a182bba0e9613b6243ebd69529b1878aa # v4.1.0
         with:
           oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
           oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
@@ -52,31 +41,90 @@ jobs:
       - name: Setup SSH server for Actor
         if: ${{ env.HAS_TAILSCALE_SECRET }}
         uses: alexellis/setup-sshd-actor@master
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
-        if: steps.changed-files.outputs.files == 'true'
-      - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
-        if: steps.changed-files.outputs.files == 'true'
+      - name: Download headscale image
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
-            '**/flake.lock') }}
-          restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
+          name: headscale-image
+          path: /tmp/artifacts
+      - name: Download tailscale HEAD image
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
+        with:
+          name: tailscale-head-image
+          path: /tmp/artifacts
+      - name: Download hi binary
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
+        with:
+          name: hi-binary
+          path: /tmp/artifacts
+      - name: Download Go cache
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
+        with:
+          name: go-cache
+          path: /tmp/artifacts
+      - name: Download postgres image
+        if: ${{ inputs.postgres_flag == '--postgres=1' }}
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
+        with:
+          name: postgres-image
+          path: /tmp/artifacts
+      - name: Pin Docker to v28 (avoid v29 breaking changes)
+        run: |
+          # Docker 29 breaks docker build via Go client libraries and
+          # docker load/save with certain tarball formats.
+          # Pin to Docker 28.x until our tooling is updated.
+          # https://github.com/actions/runner-images/issues/13474
+          sudo install -m 0755 -d /etc/apt/keyrings
+          curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
+            | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+            https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
+            | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+          sudo apt-get update -qq
+          VERSION=$(apt-cache madison docker-ce | grep '28\.5' | head -1 | awk '{print $3}')
+          sudo apt-get install -y --allow-downgrades \
+            "docker-ce=${VERSION}" "docker-ce-cli=${VERSION}"
+          sudo systemctl restart docker
+          docker version
+      - name: Load Docker images, Go cache, and prepare binary
+        run: |
+          gunzip -c /tmp/artifacts/headscale-image.tar.gz | docker load
+          gunzip -c /tmp/artifacts/tailscale-head-image.tar.gz | docker load
+          if [ -f /tmp/artifacts/postgres-image.tar.gz ]; then
+            gunzip -c /tmp/artifacts/postgres-image.tar.gz | docker load
+          fi
+          chmod +x /tmp/artifacts/hi
+          docker images
+          # Extract Go cache to host directories for bind mounting
+          mkdir -p /tmp/go-cache
+          tar -xzf /tmp/artifacts/go-cache.tar.gz -C /tmp/go-cache
+          ls -la /tmp/go-cache/ /tmp/go-cache/.cache/
       - name: Run Integration Test
-        if: always() && steps.changed-files.outputs.files == 'true'
-        run:
-          nix develop --command -- hi run --stats --ts-memory-limit=300 --hs-memory-limit=1500 "^${{ inputs.test }}$" \
+        env:
+          HEADSCALE_INTEGRATION_HEADSCALE_IMAGE: headscale:${{ github.sha }}
+          HEADSCALE_INTEGRATION_TAILSCALE_IMAGE: tailscale-head:${{ github.sha }}
+          HEADSCALE_INTEGRATION_POSTGRES_IMAGE: ${{ inputs.postgres_flag == '--postgres=1' && format('postgres:{0}', github.sha) || '' }}
+          HEADSCALE_INTEGRATION_GO_CACHE: /tmp/go-cache/go
+          HEADSCALE_INTEGRATION_GO_BUILD_CACHE: /tmp/go-cache/.cache/go-build
+        run: /tmp/artifacts/hi run --stats --ts-memory-limit=300 --hs-memory-limit=1500 "^${{ inputs.test }}$" \
           --timeout=120m \
           ${{ inputs.postgres_flag }}
-      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
-        if: always() && steps.changed-files.outputs.files == 'true'
+      # Sanitize test name for artifact upload (replace invalid characters: " : < > | * ? \ / with -)
+      - name: Sanitize test name for artifacts
+        if: always()
+        id: sanitize
+        run: echo "name=${TEST_NAME//[\":<>|*?\\\/]/-}" >> $GITHUB_OUTPUT
+        env:
+          TEST_NAME: ${{ inputs.test }}
+      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        if: always()
         with:
-          name: ${{ inputs.database_name }}-${{ inputs.test }}-logs
+          name: ${{ inputs.database_name }}-${{ steps.sanitize.outputs.name }}-logs
           path: "control_logs/*/*.log"
-      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
-        if: always() && steps.changed-files.outputs.files == 'true'
+      - uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        if: always()
         with:
-          name: ${{ inputs.database_name }}-${{ inputs.test }}-archives
-          path: "control_logs/*/*.tar"
+          name: ${{ inputs.database_name }}-${{ steps.sanitize.outputs.name }}-artifacts
+          path: control_logs/
       - name: Setup a blocking tmux session
         if: ${{ env.HAS_TAILSCALE_SECRET }}
         uses: alexellis/block-with-tmux-action@master

.github/workflows/lint.yml

@@ -10,7 +10,7 @@ jobs:
   golangci-lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
       - name: Get changed files
@@ -24,13 +24,12 @@ jobs:
               - '**/*.go'
               - 'integration_test/'
               - 'config-example.yaml'
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 
@@ -46,7 +45,7 @@ jobs:
   prettier-lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
       - name: Get changed files
@@ -65,13 +64,12 @@ jobs:
               - '**/*.css'
               - '**/*.scss'
               - '**/*.html'
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 
@@ -83,12 +81,11 @@ jobs:
   proto-lint:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 

.github/workflows/needs-more-info-comment.yml

@@ -0,0 +1,28 @@
+name: Needs More Info - Post Comment
+
+on:
+  issues:
+    types: [labeled]
+
+jobs:
+  post-comment:
+    if: >-
+      github.event.label.name == 'needs-more-info' &&
+      github.repository == 'juanfont/headscale'
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      contents: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          sparse-checkout: .github/label-response/needs-more-info.md
+          sparse-checkout-cone-mode: false
+
+      - name: Post instruction comment
+        run: gh issue comment "$NUMBER" --body-file .github/label-response/needs-more-info.md
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPO: ${{ github.repository }}
+          NUMBER: ${{ github.event.issue.number }}

.github/workflows/needs-more-info-timer.yml

@@ -0,0 +1,99 @@
+name: Needs More Info - Timer
+
+on:
+  schedule:
+    - cron: "0 0 * * *" # Daily at midnight UTC
+  issue_comment:
+    types: [created]
+  workflow_dispatch:
+
+jobs:
+  # When a non-bot user comments on a needs-more-info issue, remove the label.
+  remove-label-on-response:
+    if: >-
+      github.repository == 'juanfont/headscale' &&
+      github.event_name == 'issue_comment' &&
+      github.event.comment.user.type != 'Bot' &&
+      contains(github.event.issue.labels.*.name, 'needs-more-info')
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+    steps:
+      - name: Remove needs-more-info label
+        run: gh issue edit "$NUMBER" --remove-label needs-more-info
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPO: ${{ github.repository }}
+          NUMBER: ${{ github.event.issue.number }}
+
+  # On schedule, close issues that have had no human response for 3 days.
+  close-stale:
+    if: >-
+      github.repository == 'juanfont/headscale' &&
+      github.event_name != 'issue_comment'
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+    steps:
+      - uses: hustcer/setup-nu@920172d92eb04671776f3ba69d605d3b09351c30 # v3.22
+        with:
+          version: "*"
+
+      - name: Close stale needs-more-info issues
+        shell: nu {0}
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPO: ${{ github.repository }}
+        run: |
+          let issues = (gh issue list
+            --repo $env.GH_REPO
+            --label "needs-more-info"
+            --state open
+            --json number
+            | from json)
+
+          for issue in $issues {
+            let number = $issue.number
+            print $"Checking issue #($number)"
+
+            # Find when needs-more-info was last added
+            let events = (gh api $"repos/($env.GH_REPO)/issues/($number)/events"
+              --paginate | from json)
+            let label_event = ($events
+              | where event == "labeled"
+              | where label.name == "needs-more-info"
+              | last)
+            let label_added_at = ($label_event.created_at | into datetime)
+
+            # Check for non-bot comments after the label was added
+            let comments = (gh api $"repos/($env.GH_REPO)/issues/($number)/comments"
+              --paginate | from json)
+            let human_responses = ($comments
+              | where user.type != "Bot"
+              | where { ($in.created_at | into datetime) > $label_added_at })
+
+            if ($human_responses | length) > 0 {
+              print $"  Human responded, removing label"
+              gh issue edit $number --repo $env.GH_REPO --remove-label needs-more-info
+              continue
+            }
+
+            # Check if 3 days have passed
+            let elapsed = (date now) - $label_added_at
+            if $elapsed < 3day {
+              print $"  Only ($elapsed | format duration day) elapsed, skipping"
+              continue
+            }
+
+            print $"  No response for ($elapsed | format duration day), closing"
+            let message = [
+              "This issue has been automatically closed because no additional information was provided within 3 days."
+              ""
+              "If you have the requested information, please open a new issue and include the debug information requested above."
+              ""
+              "Thank you for your understanding."
+            ] | str join "\n"
+            gh issue comment $number --repo $env.GH_REPO --body $message
+            gh issue close $number --repo $env.GH_REPO --reason "not planned"
+            gh issue edit $number --repo $env.GH_REPO --remove-label needs-more-info
+          }

.github/workflows/nix-module-test.yml

@@ -0,0 +1,55 @@
+name: NixOS Module Tests
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+concurrency:
+  group: ${{ github.workflow }}-$${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  nix-module-check:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+
+    steps:
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+        with:
+          fetch-depth: 2
+
+      - name: Get changed files
+        id: changed-files
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+        with:
+          filters: |
+            nix:
+              - 'nix/**'
+              - 'flake.nix'
+              - 'flake.lock'
+            go:
+              - 'go.*'
+              - '**/*.go'
+              - 'cmd/**'
+              - 'hscontrol/**'
+
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
+        if: steps.changed-files.outputs.nix == 'true' || steps.changed-files.outputs.go == 'true'
+
+      - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
+        if: steps.changed-files.outputs.nix == 'true' || steps.changed-files.outputs.go == 'true'
+        with:
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+            '**/flake.lock') }}
+          restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
+
+      - name: Run NixOS module tests
+        if: steps.changed-files.outputs.nix == 'true' || steps.changed-files.outputs.go == 'true'
+        run: |
+          echo "Running NixOS module integration test..."
+          nix build .#checks.x86_64-linux.headscale -L

.github/workflows/release.yml

@@ -13,28 +13,46 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 0
 
+      - name: Pin Docker to v28 (avoid v29 breaking changes)
+        run: |
+          # Docker 29 breaks docker build via Go client libraries and
+          # docker load/save with certain tarball formats.
+          # Pin to Docker 28.x until our tooling is updated.
+          # https://github.com/actions/runner-images/issues/13474
+          sudo install -m 0755 -d /etc/apt/keyrings
+          curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
+            | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+            https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
+            | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+          sudo apt-get update -qq
+          VERSION=$(apt-cache madison docker-ce | grep '28\.5' | head -1 | awk '{print $3}')
+          sudo apt-get install -y --allow-downgrades \
+            "docker-ce=${VERSION}" "docker-ce-cli=${VERSION}"
+          sudo systemctl restart docker
+          docker version
+
       - name: Login to DockerHub
-        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
+        uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_TOKEN }}
 
       - name: Login to GHCR
-        uses: docker/login-action@74a5d142397b4f367a81961eba4e8cd7edddf772 # v3.4.0
+        uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
         with:
           registry: ghcr.io
           username: ${{ github.repository_owner }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 

.github/workflows/stale.yml

@@ -12,18 +12,16 @@ jobs:
       issues: write
       pull-requests: write
     steps:
-      - uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9.1.0
+      - uses: actions/stale@997185467fa4f803885201cee163a9f38240193d # v10.1.1
         with:
           days-before-issue-stale: 90
           days-before-issue-close: 7
           stale-issue-label: "stale"
-          stale-issue-message:
-            "This issue is stale because it has been open for 90 days with no
+          stale-issue-message: "This issue is stale because it has been open for 90 days with no
             activity."
-          close-issue-message:
-            "This issue was closed because it has been inactive for 14 days
+          close-issue-message: "This issue was closed because it has been inactive for 14 days
             since being marked as stale."
           days-before-pr-stale: -1
           days-before-pr-close: -1
-          exempt-issue-labels: "no-stale-bot"
+          exempt-issue-labels: "no-stale-bot,needs-more-info"
           repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/support-request.yml

@@ -0,0 +1,30 @@
+name: Support Request - Close Issue
+
+on:
+  issues:
+    types: [labeled]
+
+jobs:
+  close-support-request:
+    if: >-
+      github.event.label.name == 'support-request' &&
+      github.repository == 'juanfont/headscale'
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      contents: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          sparse-checkout: .github/label-response/support-request.md
+          sparse-checkout-cone-mode: false
+
+      - name: Post comment and close issue
+        run: |
+          gh issue comment "$NUMBER" --body-file .github/label-response/support-request.md
+          gh issue close "$NUMBER" --reason "not planned"
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_REPO: ${{ github.repository }}
+          NUMBER: ${{ github.event.issue.number }}

.github/workflows/test-integration.yaml

@@ -7,7 +7,154 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
   cancel-in-progress: true
 jobs:
+  # build: Builds binaries and Docker images once, uploads as artifacts for reuse.
+  # build-postgres: Pulls postgres image separately to avoid Docker Hub rate limits.
+  # sqlite: Runs all integration tests with SQLite backend.
+  # postgres: Runs a subset of tests with PostgreSQL to verify database compatibility.
+  build:
+    runs-on: ubuntu-24.04-arm
+    outputs:
+      files-changed: ${{ steps.changed-files.outputs.files }}
+    steps:
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+        with:
+          fetch-depth: 2
+      - name: Get changed files
+        id: changed-files
+        uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+        with:
+          filters: |
+            files:
+              - '*.nix'
+              - 'go.*'
+              - '**/*.go'
+              - 'integration/**'
+              - 'config-example.yaml'
+              - '.github/workflows/test-integration.yaml'
+              - '.github/workflows/integration-test-template.yml'
+              - 'Dockerfile.*'
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
+        if: steps.changed-files.outputs.files == 'true'
+      - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
+        if: steps.changed-files.outputs.files == 'true'
+        with:
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix', '**/flake.lock') }}
+          restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
+      - name: Build binaries and warm Go cache
+        if: steps.changed-files.outputs.files == 'true'
+        run: |
+          # Build all Go binaries in one nix shell to maximize cache reuse
+          nix develop --command -- bash -c '
+            go build -o hi ./cmd/hi
+            CGO_ENABLED=0 GOOS=linux go build -o headscale ./cmd/headscale
+            # Build integration test binary to warm the cache with all dependencies
+            go test -c ./integration -o /dev/null 2>/dev/null || true
+          '
+      - name: Upload hi binary
+        if: steps.changed-files.outputs.files == 'true'
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: hi-binary
+          path: hi
+          retention-days: 10
+      - name: Package Go cache
+        if: steps.changed-files.outputs.files == 'true'
+        run: |
+          # Package Go module cache and build cache
+          tar -czf go-cache.tar.gz -C ~ go .cache/go-build
+      - name: Upload Go cache
+        if: steps.changed-files.outputs.files == 'true'
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: go-cache
+          path: go-cache.tar.gz
+          retention-days: 10
+      - name: Pin Docker to v28 (avoid v29 breaking changes)
+        if: steps.changed-files.outputs.files == 'true'
+        run: |
+          # Docker 29 breaks docker build via Go client libraries and
+          # docker load/save with certain tarball formats.
+          # Pin to Docker 28.x until our tooling is updated.
+          # https://github.com/actions/runner-images/issues/13474
+          sudo install -m 0755 -d /etc/apt/keyrings
+          curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
+            | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+            https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
+            | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+          sudo apt-get update -qq
+          VERSION=$(apt-cache madison docker-ce | grep '28\.5' | head -1 | awk '{print $3}')
+          sudo apt-get install -y --allow-downgrades \
+            "docker-ce=${VERSION}" "docker-ce-cli=${VERSION}"
+          sudo systemctl restart docker
+          docker version
+      - name: Build headscale image
+        if: steps.changed-files.outputs.files == 'true'
+        run: |
+          docker build \
+            --file Dockerfile.integration-ci \
+            --tag headscale:${{ github.sha }} \
+            .
+          docker save headscale:${{ github.sha }} | gzip > headscale-image.tar.gz
+      - name: Build tailscale HEAD image
+        if: steps.changed-files.outputs.files == 'true'
+        run: |
+          docker build \
+            --file Dockerfile.tailscale-HEAD \
+            --tag tailscale-head:${{ github.sha }} \
+            .
+          docker save tailscale-head:${{ github.sha }} | gzip > tailscale-head-image.tar.gz
+      - name: Upload headscale image
+        if: steps.changed-files.outputs.files == 'true'
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: headscale-image
+          path: headscale-image.tar.gz
+          retention-days: 10
+      - name: Upload tailscale HEAD image
+        if: steps.changed-files.outputs.files == 'true'
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: tailscale-head-image
+          path: tailscale-head-image.tar.gz
+          retention-days: 10
+  build-postgres:
+    runs-on: ubuntu-24.04-arm
+    needs: build
+    if: needs.build.outputs.files-changed == 'true'
+    steps:
+      - name: Pin Docker to v28 (avoid v29 breaking changes)
+        run: |
+          # Docker 29 breaks docker build via Go client libraries and
+          # docker load/save with certain tarball formats.
+          # Pin to Docker 28.x until our tooling is updated.
+          # https://github.com/actions/runner-images/issues/13474
+          sudo install -m 0755 -d /etc/apt/keyrings
+          curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
+            | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
+            https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
+            | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+          sudo apt-get update -qq
+          VERSION=$(apt-cache madison docker-ce | grep '28\.5' | head -1 | awk '{print $3}')
+          sudo apt-get install -y --allow-downgrades \
+            "docker-ce=${VERSION}" "docker-ce-cli=${VERSION}"
+          sudo systemctl restart docker
+          docker version
+      - name: Pull and save postgres image
+        run: |
+          docker pull postgres:latest
+          docker tag postgres:latest postgres:${{ github.sha }}
+          docker save postgres:${{ github.sha }} | gzip > postgres-image.tar.gz
+      - name: Upload postgres image
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
+        with:
+          name: postgres-image
+          path: postgres-image.tar.gz
+          retention-days: 10
   sqlite:
+    needs: build
+    if: needs.build.outputs.files-changed == 'true'
     strategy:
       fail-fast: false
       matrix:
@@ -25,6 +172,13 @@ jobs:
           - TestACLAutogroupTagged
           - TestACLAutogroupSelf
           - TestACLPolicyPropagationOverTime
+          - TestACLTagPropagation
+          - TestACLTagPropagationPortSpecific
+          - TestACLGroupWithUnknownUser
+          - TestACLGroupAfterUserDeletion
+          - TestACLGroupDeletionExactReproduction
+          - TestACLDynamicUnknownUserAddition
+          - TestACLDynamicUnknownUserRemoval
           - TestAPIAuthenticationBypass
           - TestAPIAuthenticationBypassCurl
           - TestGRPCAuthenticationBypass
@@ -32,6 +186,8 @@ jobs:
           - TestAuthKeyLogoutAndReloginSameUser
           - TestAuthKeyLogoutAndReloginNewUser
           - TestAuthKeyLogoutAndReloginSameUserExpiredKey
+          - TestAuthKeyDeleteKey
+          - TestAuthKeyLogoutAndReloginRoutesPreserved
           - TestOIDCAuthenticationPingAll
           - TestOIDCExpireNodesBasedOnTokenExpiry
           - TestOIDC024UserCreation
@@ -41,6 +197,8 @@ jobs:
           - TestOIDCMultipleOpenedLoginUrls
           - TestOIDCReloginSameNodeSameUser
           - TestOIDCExpiryAfterRestart
+          - TestOIDCACLPolicyOnJoin
+          - TestOIDCReloginSameUserRoutesPreserved
           - TestAuthWebFlowAuthenticationPingAll
           - TestAuthWebFlowLogoutAndReloginSameUser
           - TestAuthWebFlowLogoutAndReloginNewUser
@@ -49,13 +207,11 @@ jobs:
           - TestPreAuthKeyCommandWithoutExpiry
           - TestPreAuthKeyCommandReusableEphemeral
           - TestPreAuthKeyCorrectUserLoggedInCommand
+          - TestTaggedNodesCLIOutput
           - TestApiKeyCommand
-          - TestNodeTagCommand
-          - TestNodeAdvertiseTagCommand
           - TestNodeCommand
           - TestNodeExpireCommand
           - TestNodeRenameCommand
-          - TestNodeMoveCommand
           - TestPolicyCommand
           - TestPolicyBrokenConfigCommand
           - TestDERPVerifyEndpoint
@@ -73,17 +229,31 @@ jobs:
           - TestUpdateHostnameFromClient
           - TestExpireNode
           - TestSetNodeExpiryInFuture
+          - TestDisableNodeExpiry
           - TestNodeOnlineStatus
           - TestPingAllByIPManyUpDown
           - Test2118DeletingOnlineNodePanics
+          - TestGrantCapRelay
+          - TestGrantCapDrive
           - TestEnablingRoutes
           - TestHASubnetRouterFailover
           - TestSubnetRouteACL
           - TestEnablingExitRoutes
+          - TestExitRoutesWithAutogroupInternetACL
           - TestSubnetRouterMultiNetwork
           - TestSubnetRouterMultiNetworkExitNode
-          - TestAutoApproveMultiNetwork
+          - TestAutoApproveMultiNetwork/authkey-tag.*
+          - TestAutoApproveMultiNetwork/authkey-user.*
+          - TestAutoApproveMultiNetwork/authkey-group.*
+          - TestAutoApproveMultiNetwork/webauth-tag.*
+          - TestAutoApproveMultiNetwork/webauth-user.*
+          - TestAutoApproveMultiNetwork/webauth-group.*
           - TestSubnetRouteACLFiltering
+          - TestGrantViaSubnetSteering
+          - TestHASubnetRouterPingFailover
+          - TestHASubnetRouterFailoverBothOffline
+          - TestHASubnetRouterFailoverBothOfflineCablePull
+          - TestHASubnetRouterFailoverDockerDisconnect
           - TestHeadscale
           - TestTailscaleNodesJoiningHeadcale
           - TestSSHOneUserToAll
@@ -92,12 +262,55 @@ jobs:
           - TestSSHIsBlockedInACL
           - TestSSHUserOnlyIsolation
           - TestSSHAutogroupSelf
+          - TestSSHOneUserToOneCheckModeCLI
+          - TestSSHOneUserToOneCheckModeOIDC
+          - TestSSHCheckModeUnapprovedTimeout
+          - TestSSHCheckModeCheckPeriodCLI
+          - TestSSHCheckModeAutoApprove
+          - TestSSHCheckModeNegativeCLI
+          - TestSSHLocalpart
+          - TestTagsAuthKeyWithTagRequestDifferentTag
+          - TestTagsAuthKeyWithTagNoAdvertiseFlag
+          - TestTagsAuthKeyWithTagCannotAddViaCLI
+          - TestTagsAuthKeyWithTagCannotChangeViaCLI
+          - TestTagsAuthKeyWithTagAdminOverrideReauthPreserves
+          - TestTagsAuthKeyWithTagCLICannotModifyAdminTags
+          - TestTagsAuthKeyWithoutTagCannotRequestTags
+          - TestTagsAuthKeyWithoutTagRegisterNoTags
+          - TestTagsAuthKeyWithoutTagCannotAddViaCLI
+          - TestTagsAuthKeyWithoutTagCLINoOpAfterAdminWithReset
+          - TestTagsAuthKeyWithoutTagCLINoOpAfterAdminWithEmptyAdvertise
+          - TestTagsAuthKeyWithoutTagCLICannotReduceAdminMultiTag
+          - TestTagsUserLoginOwnedTagAtRegistration
+          - TestTagsUserLoginNonExistentTagAtRegistration
+          - TestTagsUserLoginUnownedTagAtRegistration
+          - TestTagsUserLoginAddTagViaCLIReauth
+          - TestTagsUserLoginRemoveTagViaCLIReauth
+          - TestTagsUserLoginCLINoOpAfterAdminAssignment
+          - TestTagsUserLoginCLICannotRemoveAdminTags
+          - TestTagsAuthKeyWithTagRequestNonExistentTag
+          - TestTagsAuthKeyWithTagRequestUnownedTag
+          - TestTagsAuthKeyWithoutTagRequestNonExistentTag
+          - TestTagsAuthKeyWithoutTagRequestUnownedTag
+          - TestTagsAdminAPICannotSetNonExistentTag
+          - TestTagsAdminAPICanSetUnownedTag
+          - TestTagsAdminAPICannotRemoveAllTags
+          - TestTagsIssue2978ReproTagReplacement
+          - TestTagsAdminAPICannotSetInvalidFormat
+          - TestTagsUserLoginReauthWithEmptyTagsRemovesAllTags
+          - TestTagsAuthKeyWithoutUserInheritsTags
+          - TestTagsAuthKeyWithoutUserRejectsAdvertisedTags
+          - TestTagsAuthKeyConvertToUserViaCLIRegister
+          - TestTailscaleRustAxum
     uses: ./.github/workflows/integration-test-template.yml
+    secrets: inherit
     with:
       test: ${{ matrix.test }}
       postgres_flag: "--postgres=0"
       database_name: "sqlite"
   postgres:
+    needs: [build, build-postgres]
+    if: needs.build.outputs.files-changed == 'true'
     strategy:
       fail-fast: false
       matrix:
@@ -108,6 +321,7 @@ jobs:
           - TestPingAllByIPManyUpDown
           - TestSubnetRouterMultiNetwork
     uses: ./.github/workflows/integration-test-template.yml
+    secrets: inherit
     with:
       test: ${{ matrix.test }}
       postgres_flag: "--postgres=1"

.github/workflows/test.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
         with:
           fetch-depth: 2
 
@@ -27,13 +27,12 @@ jobs:
               - 'integration_test/'
               - 'config-example.yaml'
 
-      - uses: nixbuild/nix-quick-install-action@889f3180bb5f064ee9e3201428d04ae9e41d54ad # v31
+      - uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
         if: steps.changed-files.outputs.files == 'true'
       - uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
         if: steps.changed-files.outputs.files == 'true'
         with:
-          primary-key:
-            nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
+          primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
             '**/flake.lock') }}
           restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
 

.gitignore

@@ -2,6 +2,7 @@ ignored/
 tailscale/
 .vscode/
 .claude/
+logs/
 
 *.prof
 
@@ -28,6 +29,7 @@ config*.yaml
 !config-example.yaml
 derp.yaml
 *.hujson
+!hscontrol/policy/v2/testdata/*/*.hujson
 *.key
 /db.sqlite
 *.sqlite3

.golangci.yaml

@@ -7,6 +7,7 @@ linters:
     - depguard
     - dupl
     - exhaustruct
+    - funcorder
     - funlen
     - gochecknoglobals
     - gochecknoinits
@@ -17,6 +18,7 @@ linters:
     - lll
     - maintidx
     - makezero
+    - mnd
     - musttag
     - nestif
     - nolintlint
@@ -28,6 +30,32 @@ linters:
     - wrapcheck
     - wsl
   settings:
+    forbidigo:
+      forbid:
+        # Forbid time.Sleep everywhere with context-appropriate alternatives
+        - pattern: 'time\.Sleep'
+          msg: >-
+            time.Sleep is forbidden.
+            In tests: use assert.EventuallyWithT for polling/waiting patterns.
+            In production code: use a backoff strategy (e.g., cenkalti/backoff) or proper synchronization primitives.
+        # Forbid inline string literals in zerolog field methods - use zf.* constants
+        - pattern: '\.(Str|Int|Int8|Int16|Int32|Int64|Uint|Uint8|Uint16|Uint32|Uint64|Float32|Float64|Bool|Dur|Time|TimeDiff|Strs|Ints|Uints|Floats|Bools|Any|Interface)\("[^"]+"'
+          msg: >-
+            Use zf.* constants for zerolog field names instead of string literals.
+            Import "github.com/juanfont/headscale/hscontrol/util/zlog/zf" and use
+            constants like zf.NodeID, zf.UserName, etc. Add new constants to
+            hscontrol/util/zlog/zf/fields.go if needed.
+        # Forbid ptr.To - use Go 1.26 new(expr) instead
+        - pattern: 'ptr\.To\('
+          msg: >-
+            ptr.To is forbidden. Use Go 1.26's new(expr) syntax instead.
+            Example: ptr.To(value) → new(value)
+        # Forbid tsaddr.SortPrefixes - use slices.SortFunc with netip.Prefix.Compare
+        - pattern: 'tsaddr\.SortPrefixes'
+          msg: >-
+            tsaddr.SortPrefixes is forbidden. Use Go 1.26's netip.Prefix.Compare instead.
+            Example: slices.SortFunc(prefixes, netip.Prefix.Compare)
+      analyze-types: true
     gocritic:
       disabled-checks:
         - appendAssign

.goreleaser.yml

@@ -2,7 +2,7 @@
 version: 2
 before:
   hooks:
-    - go mod tidy -compat=1.25
+    - go mod tidy -compat=1.26
     - go mod vendor
 
 release:
@@ -13,29 +13,6 @@ release:
 
     Please follow the steps outlined in the [upgrade guide](https://headscale.net/stable/setup/upgrade/) to update your existing Headscale installation.
 
-    **It's best to update from one stable version to the next** (e.g., 0.24.0 → 0.25.1 → 0.26.1) in case you are multiple releases behind. You should always pick the latest available patch release.
-
-    Be sure to check the changelog above for version-specific upgrade instructions and breaking changes.
-
-    ### Backup Your Database
-
-    **Always backup your database before upgrading.** Here's how to backup a SQLite database:
-
-    ```bash
-    # Stop headscale
-    systemctl stop headscale
-
-    # Backup sqlite database
-    cp /var/lib/headscale/db.sqlite /var/lib/headscale/db.sqlite.backup
-
-    # Backup sqlite WAL/SHM files (if they exist)
-    cp /var/lib/headscale/db.sqlite-wal /var/lib/headscale/db.sqlite-wal.backup
-    cp /var/lib/headscale/db.sqlite-shm /var/lib/headscale/db.sqlite-shm.backup
-
-    # Start headscale (migration will run automatically)
-    systemctl start headscale
-    ```
-
 builds:
   - id: headscale
     main: ./cmd/headscale
@@ -50,8 +27,6 @@ builds:
       - linux_arm64
     flags:
       - -mod=readonly
-    tags:
-      - ts2019
 
 archives:
   - id: golang-cross
@@ -67,10 +42,9 @@ source:
     - "vendor/"
 
 nfpms:
-  # Configure nFPM for .deb and .rpm releases
+  # Configure nFPM for .deb releases
   #
-  # See https://nfpm.goreleaser.com/configuration/
-  # and https://goreleaser.com/customization/nfpm/
+  # See https://goreleaser.com/customization/package/nfpm/
   #
   # Useful tools for debugging .debs:
   # List file contents: dpkg -c dist/headscale...deb
@@ -104,6 +78,8 @@ nfpms:
         dst: /usr/lib/systemd/system/headscale.service
       - dst: /var/lib/headscale
         type: dir
+      - src: ./config-example.yaml
+        dst: /usr/share/doc/headscale/examples/config-example.yaml
       - src: LICENSE
         dst: /usr/share/doc/headscale/copyright
     scripts:
@@ -125,7 +101,7 @@ kos:
     # bare tells KO to only use the repository
     # for tagging and naming the container.
     bare: true
-    base_image: gcr.io/distroless/base-debian12
+    base_image: gcr.io/distroless/base-debian13
     build: headscale
     main: ./cmd/headscale
     env:
@@ -154,7 +130,7 @@ kos:
       - headscale/headscale
 
     bare: true
-    base_image: gcr.io/distroless/base-debian12:debug
+    base_image: gcr.io/distroless/base-debian13:debug
     build: headscale
     main: ./cmd/headscale
     env:

.mcp.json

@@ -3,45 +3,31 @@
     "claude-code-mcp": {
       "type": "stdio",
       "command": "npx",
-      "args": [
-        "-y",
-        "@steipete/claude-code-mcp@latest"
-      ],
+      "args": ["-y", "@steipete/claude-code-mcp@latest"],
       "env": {}
     },
     "sequential-thinking": {
       "type": "stdio",
       "command": "npx",
-      "args": [
-        "-y",
-        "@modelcontextprotocol/server-sequential-thinking"
-      ],
+      "args": ["-y", "@modelcontextprotocol/server-sequential-thinking"],
       "env": {}
     },
     "nixos": {
       "type": "stdio",
       "command": "uvx",
-      "args": [
-        "mcp-nixos"
-      ],
+      "args": ["mcp-nixos"],
       "env": {}
     },
     "context7": {
       "type": "stdio",
       "command": "npx",
-      "args": [
-        "-y",
-        "@upstash/context7-mcp"
-      ],
+      "args": ["-y", "@upstash/context7-mcp"],
       "env": {}
     },
     "git": {
       "type": "stdio",
       "command": "npx",
-      "args": [
-        "-y",
-        "@cyanheads/git-mcp-server"
-      ],
+      "args": ["-y", "@cyanheads/git-mcp-server"],
       "env": {}
     }
   }

.mdformat.toml

@@ -0,0 +1,2 @@
+[plugin.mkdocs]
+align_semantic_breaks_in_lists = true

.pre-commit-config.yaml

@@ -0,0 +1,63 @@
+# prek/pre-commit configuration for headscale
+# See: https://prek.j178.dev/quickstart/
+# See: https://prek.j178.dev/builtin/
+
+# Global exclusions - ignore generated code
+exclude: ^gen/
+
+repos:
+  # Built-in hooks from pre-commit/pre-commit-hooks
+  # prek will use fast-path optimized versions automatically
+  # See: https://prek.j178.dev/builtin/
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+      - id: check-added-large-files
+        args: [--maxkb=1024]
+      - id: check-case-conflict
+      - id: check-executables-have-shebangs
+      - id: check-json
+      - id: check-merge-conflict
+      - id: check-symlinks
+      - id: check-toml
+      - id: check-xml
+      - id: check-yaml
+      - id: detect-private-key
+      - id: end-of-file-fixer
+      - id: fix-byte-order-marker
+      - id: mixed-line-ending
+      - id: trailing-whitespace
+
+  # Local hooks for project-specific tooling
+  - repo: local
+    hooks:
+      # nixpkgs-fmt for Nix files
+      - id: nixpkgs-fmt
+        name: nixpkgs-fmt
+        entry: nixpkgs-fmt
+        language: system
+        files: \.nix$
+
+      # Prettier for formatting
+      - id: prettier
+        name: prettier
+        entry: prettier --write --list-different
+        language: system
+        exclude: ^docs/
+        types_or: [javascript, jsx, ts, tsx, yaml, json, toml, html, css, scss, sass, markdown]
+
+      # mdformat for docs
+      - id: mdformat
+        name: mdformat
+        entry: mdformat
+        language: system
+        types_or: [markdown]
+        files: ^docs/
+
+      # golangci-lint for Go code quality
+      - id: golangci-lint
+        name: golangci-lint
+        entry: nix develop --command -- golangci-lint run --new-from-rev=HEAD~1 --timeout=5m --fix
+        language: system
+        types: [go]
+        pass_filenames: false

.prettierignore

@@ -1,5 +1,2 @@
 .github/workflows/test-integration-v2*
-docs/about/features.md
-docs/ref/configuration.md
-docs/ref/oidc.md
-docs/ref/remote-cli.md
+docs/

AGENTS.md

@@ -0,0 +1,291 @@
+# AGENTS.md
+
+Behavioural guidance for AI agents working in this repository. Reference
+material for complex procedures lives next to the code — integration
+testing is documented in [`cmd/hi/README.md`](cmd/hi/README.md) and
+[`integration/README.md`](integration/README.md). Read those files
+before running tests or writing new ones.
+
+Headscale is an open-source implementation of the Tailscale control server
+written in Go. It manages node registration, IP allocation, policy
+enforcement, and DERP routing for self-hosted tailnets.
+
+## Interaction Rules
+
+These rules govern how you work in this repo. They are listed first
+because they shape every other decision.
+
+### Ask with comprehensive multiple-choice options
+
+When you need to clarify intent, scope, or approach, use the
+`AskUserQuestion` tool (or a numbered list fallback) and present the user
+with a comprehensive set of options. Cover the likely branches explicitly
+and include an "other — please describe" escape.
+
+- Bad: _"How should I handle expired nodes?"_
+- Good: _"How should expired nodes be handled? (a) Remain visible to peers
+  but marked expired (current behaviour); (b) Hidden from peers entirely;
+  (c) Hidden from peers but visible in admin API; (d) Other."_
+
+This matters more than you think — open-ended questions waste a round
+trip and often produce a misaligned answer.
+
+### Read the documented procedure before running complex commands
+
+Before invoking any `hi` command, integration test, generator, or
+migration tool, read the referenced README in full —
+`cmd/hi/README.md` for running tests, `integration/README.md` for
+writing them. Never guess flags. If the procedure is not documented
+anywhere, ask the user rather than inventing one.
+
+### Map once, then act
+
+Use `Glob` / `Grep` to understand file structure, then execute. Do not
+re-explore the same area to "double-check" once you have a plan. Do not
+re-read files you edited in this session — the harness tracks state for
+you.
+
+### Fail fast, report up
+
+If a command fails twice with the same error, stop and report the exact
+error to the user with context. Do not loop through variants or
+"try one more thing". A repeated failure means your model of the problem
+is wrong.
+
+### Confirm scope for multi-file changes
+
+Before touching more than three files, show the user which files will
+change and why. Use plan mode (`ExitPlanMode`) for non-trivial work.
+
+### Prefer editing existing files
+
+Do not create new files unless strictly necessary. Do not generate helper
+abstractions, wrapper utilities, or "just in case" configuration. Three
+similar lines of code is better than a premature abstraction.
+
+## Quick Start
+
+```bash
+# Enter the nix dev shell (Go 1.26.1, buf, golangci-lint, prek)
+nix develop
+
+# Full development workflow: fmt + lint + test + build
+make dev
+
+# Individual targets
+make build           # build the headscale binary
+make test            # go test ./...
+make fmt             # format Go, docs, proto
+make lint            # lint Go, proto
+make generate        # regenerate protobuf code (after changes to proto/)
+make clean           # remove build artefacts
+
+# Direct go test invocations
+go test ./...
+go test -race ./...
+
+# Integration tests — read cmd/hi/README.md first
+go run ./cmd/hi doctor
+go run ./cmd/hi run "TestName"
+```
+
+Go 1.26.1 minimum (per `go.mod:3`). `nix develop` pins the exact toolchain
+used in CI.
+
+## Pre-Commit with prek
+
+`prek` installs git hooks that run the same checks as CI.
+
+```bash
+nix develop
+prek install            # one-time setup
+prek run                # run hooks on staged files
+prek run --all-files    # run hooks on the full tree
+```
+
+Hooks cover: file hygiene (trailing whitespace, line endings, BOM),
+syntax validation (JSON/YAML/TOML/XML), merge-conflict markers, private
+key detection, nixpkgs-fmt, prettier, and `golangci-lint` via
+`--new-from-rev=HEAD~1` (see `.pre-commit-config.yaml:59`). A manual
+invocation with an `upstream/main` remote is equivalent:
+
+```bash
+golangci-lint run --new-from-rev=upstream/main --timeout=5m --fix
+```
+
+`git commit --no-verify` is acceptable only for WIP commits on feature
+branches — never on `main`.
+
+## Project Layout
+
+```
+headscale/
+├── cmd/
+│   ├── headscale/    # Main headscale server binary
+│   └── hi/           # Integration test runner (see cmd/hi/README.md)
+├── hscontrol/        # Core control plane
+├── integration/      # End-to-end Docker-based tests (see integration/README.md)
+├── proto/            # Protocol buffer definitions
+├── gen/              # Generated code (buf output — do not edit)
+├── docs/             # User and ACL reference documentation
+└── packaging/        # Distribution packaging
+```
+
+### `hscontrol/` packages
+
+- `app.go`, `handlers.go`, `grpcv1.go`, `noise.go`, `auth.go`, `oidc.go`,
+  `poll.go`, `metrics.go`, `debug.go`, `tailsql.go`, `platform_config.go`
+  — top-level server files
+- `state/` — central coordinator (`state.go`) and the copy-on-write
+  `NodeStore` (`node_store.go`). All cross-subsystem operations go
+  through `State`.
+- `db/` — GORM layer, migrations, schema. `node.go`, `users.go`,
+  `api_key.go`, `preauth_keys.go`, `ip.go`, `policy.go`.
+- `mapper/` — streaming batcher that distributes MapResponses to
+  clients: `batcher.go`, `node_conn.go`, `builder.go`, `mapper.go`.
+  Performance-critical.
+- `policy/` — `policy/v2/` is **the** policy implementation. The
+  top-level `policy.go` is thin wrappers. There is no v1 directory.
+- `routes/`, `dns/`, `derp/`, `types/`, `util/`, `templates/`, `capver/`
+  — routing, MagicDNS, relay, core types, helpers, client templates,
+  capability versioning.
+- `servertest/` — in-memory test harness for server-level tests that
+  don't need Docker. Prefer this over `integration/` when possible.
+- `assets/` — embedded UI assets.
+
+### `cmd/hi/` files
+
+`main.go`, `run.go`, `doctor.go`, `docker.go`, `cleanup.go`, `stats.go`,
+`README.md`. **Read `cmd/hi/README.md` before running any `hi` command.**
+
+## Architecture Essentials
+
+- **`hscontrol/state/state.go`** is the central coordinator. Cross-cutting
+  operations (node updates, policy evaluation, IP allocation) go through
+  the `State` type, not directly to the database.
+- **`NodeStore`** in `hscontrol/state/node_store.go` is a copy-on-write
+  in-memory cache backed by `atomic.Pointer[Snapshot]`. Every read is a
+  pointer load; writes rebuild a new snapshot and atomically swap. It is
+  the hot path for `MapRequest` processing and peer visibility.
+- **The map-request sync point** is
+  `State.UpdateNodeFromMapRequest()` in
+  `hscontrol/state/state.go:2351`. This is where Hostinfo changes,
+  endpoint updates, and route advertisements land in the NodeStore.
+- **Mapper subsystem** streams MapResponses via `batcher.go` and
+  `node_conn.go`. Changes here affect all connected clients.
+- **Node registration flow**: noise handshake (`noise.go`) → auth
+  (`auth.go`) → state/DB persistence (`state/`, `db/`) → initial map
+  (`mapper/`).
+
+## Database Migration Rules
+
+These rules are load-bearing — violating them corrupts production
+databases. The `migrationsRequiringFKDisabled` map in
+`hscontrol/db/db.go:962` is frozen as of 2025-07-02 (see the comment at
+`db.go:989`). All new migrations must:
+
+1. **Never reorder existing migrations.** Migration order is immutable
+   once committed.
+2. **Only add new migrations to the end** of the migrations array.
+3. **Never disable foreign keys.** No new entries in
+   `migrationsRequiringFKDisabled`.
+4. **Use the migration ID format** `YYYYMMDDHHMM-short-description`
+   (timestamp + descriptive suffix). Example: `202602201200-clear-tagged-node-user-id`.
+5. **Never rename columns** that later migrations reference. Let
+   `AutoMigrate` create a new column if needed.
+
+## Tags-as-Identity
+
+Headscale enforces **tags XOR user ownership**: every node is either
+tagged (owned by tags) or user-owned (owned by a user namespace), never
+both. This is a load-bearing architectural rule.
+
+- **Use `node.IsTagged()`** (`hscontrol/types/node.go:221`) to determine
+  ownership, not `node.UserID().Valid()`. A tagged node may still have
+  `UserID` set for "created by" tracking — `IsTagged()` is authoritative.
+- `IsUserOwned()` (`node.go:227`) returns `!IsTagged()`.
+- Tagged nodes are presented to Tailscale as the special
+  `TaggedDevices` user (`hscontrol/types/users.go`, ID `2147455555`).
+- `SetTags` validation is enforced by `validateNodeOwnership()` in
+  `hscontrol/state/tags.go`.
+- Examples and edge cases live in `hscontrol/types/node_tags_test.go`
+  and `hscontrol/grpcv1_test.go` (`TestSetTags_*`).
+
+**Don't do this**:
+
+```go
+if node.UserID().Valid() { /* assume user-owned */ }       // WRONG
+if node.UserID().Valid() && !node.IsTagged() { /* ok */ }  // correct
+```
+
+## Policy Engine
+
+`hscontrol/policy/v2/policy.go` is the policy implementation. The
+top-level `hscontrol/policy/policy.go` contains only wrapper functions
+around v2. There is no v1 directory.
+
+Key concepts an agent will encounter:
+
+- **Autogroups**: `autogroup:self`, `autogroup:member`, `autogroup:internet`
+- **Tag owners**: IP-based authorization for who can claim a tag
+- **Route approvals**: auto-approval of subnet routes by policy
+- **SSH policies**: SSH access control via grants
+- **HuJSON** parsing for policy files
+
+For usage examples, read `hscontrol/policy/v2/policy_test.go`. For ACL
+reference documentation, see `docs/`.
+
+## Integration Testing
+
+**Before running any `hi` command, read `cmd/hi/README.md` in full.**
+Guessing at `hi` flags leads to broken runs and stale containers.
+
+Test-authoring patterns (`EventuallyWithT`, `IntegrationSkip`, helper
+variants, scenario setup) are documented in `integration/README.md`.
+
+Key reminders:
+
+- Integration test functions **must** start with `IntegrationSkip(t)`.
+- External calls (`client.Status`, `headscale.ListNodes`, etc.) belong
+  inside `EventuallyWithT`; state-mutating commands (`tailscale set`)
+  must not.
+- Tests generate ~100 MB of logs per run under `control_logs/{runID}/`.
+  Prune old runs if disk is tight.
+- Flakes are almost always code, not infrastructure. Read `hs-*.stderr.log`
+  before blaming Docker.
+
+## Code Conventions
+
+- **Commit messages** follow Go-style `package: imperative description`.
+  Recent examples from `git log`:
+  - `db: scope DestroyUser to only delete the target user's pre-auth keys`
+  - `state: fix policy change race in UpdateNodeFromMapRequest`
+  - `integration: fix ACL tests for address-family-specific resolve`
+
+  Not Conventional Commits. No `feat:`/`chore:`/`docs:` prefixes.
+
+- **Protobuf regeneration**: changes under `proto/` require
+  `make generate` (which runs `buf generate`) and should land in a
+  **separate commit** from the callers that use the regenerated types.
+- **Formatting** is enforced by `golangci-lint` with `golines` (width 88)
+  and `gofumpt`. Run `make fmt` or rely on the pre-commit hook.
+- **Logging** uses `zerolog`. Prefer single-line chains
+  (`log.Info().Str(...).Msg(...)`). For 4+ fields or conditional fields,
+  build incrementally and **reassign** the event variable:
+  `e = e.Str("k", v)`. Forgetting to reassign silently drops the field.
+- **Tests**: prefer `hscontrol/servertest/` for server-level tests that
+  don't need Docker — faster than full integration tests.
+
+## Gotchas
+
+- **Database**: SQLite for local dev, PostgreSQL for integration-heavy
+  tests (`go run ./cmd/hi run "..." --postgres`). Some race conditions
+  only surface on one backend.
+- **NodeStore writes** rebuild a full snapshot. Measure before changing
+  hot-path code.
+- **`.claude/agents/` is deprecated.** Do not create new agent files
+  there. Put behavioural guidance in this file and procedural guidance
+  in the nearest README.
+- **Do not edit `gen/`** — it is regenerated from `proto/` by
+  `make generate`.
+- **Proto changes + code changes should be two commits**, not one.

CLAUDE.md

@@ -1,531 +1 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Overview
-
-Headscale is an open-source implementation of the Tailscale control server written in Go. It provides self-hosted coordination for Tailscale networks (tailnets), managing node registration, IP allocation, policy enforcement, and DERP routing.
-
-## Development Commands
-
-### Quick Setup
-```bash
-# Recommended: Use Nix for dependency management
-nix develop
-
-# Full development workflow
-make dev  # runs fmt + lint + test + build
-```
-
-### Essential Commands
-```bash
-# Build headscale binary
-make build
-
-# Run tests
-make test
-go test ./...                    # All unit tests
-go test -race ./...              # With race detection
-
-# Run specific integration test
-go run ./cmd/hi run "TestName" --postgres
-
-# Code formatting and linting
-make fmt         # Format all code (Go, docs, proto)
-make lint        # Lint all code (Go, proto)
-make fmt-go      # Format Go code only
-make lint-go     # Lint Go code only
-
-# Protocol buffer generation (after modifying proto/)
-make generate
-
-# Clean build artifacts  
-make clean
-```
-
-### Integration Testing
-```bash
-# Use the hi (Headscale Integration) test runner
-go run ./cmd/hi doctor                    # Check system requirements
-go run ./cmd/hi run "TestPattern"         # Run specific test
-go run ./cmd/hi run "TestPattern" --postgres  # With PostgreSQL backend
-
-# Test artifacts are saved to control_logs/ with logs and debug data
-```
-
-## Project Structure & Architecture
-
-### Top-Level Organization
-
-```
-headscale/
-├── cmd/                    # Command-line applications
-│   ├── headscale/         # Main headscale server binary
-│   └── hi/               # Headscale Integration test runner
-├── hscontrol/            # Core control plane logic
-├── integration/          # End-to-end Docker-based tests
-├── proto/               # Protocol buffer definitions
-├── gen/                 # Generated code (protobuf)
-├── docs/                # Documentation
-└── packaging/           # Distribution packaging
-```
-
-### Core Packages (`hscontrol/`)
-
-**Main Server (`hscontrol/`)**
-- `app.go`: Application setup, dependency injection, server lifecycle
-- `handlers.go`: HTTP/gRPC API endpoints for management operations
-- `grpcv1.go`: gRPC service implementation for headscale API
-- `poll.go`: **Critical** - Handles Tailscale MapRequest/MapResponse protocol
-- `noise.go`: Noise protocol implementation for secure client communication
-- `auth.go`: Authentication flows (web, OIDC, command-line)
-- `oidc.go`: OpenID Connect integration for user authentication
-
-**State Management (`hscontrol/state/`)**
-- `state.go`: Central coordinator for all subsystems (database, policy, IP allocation, DERP)
-- `node_store.go`: **Performance-critical** - In-memory cache with copy-on-write semantics
-- Thread-safe operations with deadlock detection
-- Coordinates between database persistence and real-time operations
-
-**Database Layer (`hscontrol/db/`)**
-- `db.go`: Database abstraction, GORM setup, migration management
-- `node.go`: Node lifecycle, registration, expiration, IP assignment
-- `users.go`: User management, namespace isolation
-- `api_key.go`: API authentication tokens
-- `preauth_keys.go`: Pre-authentication keys for automated node registration
-- `ip.go`: IP address allocation and management
-- `policy.go`: Policy storage and retrieval
-- Schema migrations in `schema.sql` with extensive test data coverage
-
-**Policy Engine (`hscontrol/policy/`)**
-- `policy.go`: Core ACL evaluation logic, HuJSON parsing
-- `v2/`: Next-generation policy system with improved filtering
-- `matcher/`: ACL rule matching and evaluation engine
-- Determines peer visibility, route approval, and network access rules
-- Supports both file-based and database-stored policies
-
-**Network Management (`hscontrol/`)**
-- `derp/`: DERP (Designated Encrypted Relay for Packets) server implementation
-  - NAT traversal when direct connections fail
-  - Fallback relay for firewall-restricted environments
-- `mapper/`: Converts internal Headscale state to Tailscale's wire protocol format
-  - `tail.go`: Tailscale-specific data structure generation
-- `routes/`: Subnet route management and primary route selection
-- `dns/`: DNS record management and MagicDNS implementation
-
-**Utilities & Support (`hscontrol/`)**
-- `types/`: Core data structures, configuration, validation
-- `util/`: Helper functions for networking, DNS, key management
-- `templates/`: Client configuration templates (Apple, Windows, etc.)
-- `notifier/`: Event notification system for real-time updates
-- `metrics.go`: Prometheus metrics collection
-- `capver/`: Tailscale capability version management
-
-### Key Subsystem Interactions
-
-**Node Registration Flow**
-1. **Client Connection**: `noise.go` handles secure protocol handshake
-2. **Authentication**: `auth.go` validates credentials (web/OIDC/preauth)
-3. **State Creation**: `state.go` coordinates IP allocation via `db/ip.go`
-4. **Storage**: `db/node.go` persists node, `NodeStore` caches in memory
-5. **Network Setup**: `mapper/` generates initial Tailscale network map
-
-**Ongoing Operations**
-1. **Poll Requests**: `poll.go` receives periodic client updates
-2. **State Updates**: `NodeStore` maintains real-time node information
-3. **Policy Application**: `policy/` evaluates ACL rules for peer relationships
-4. **Map Distribution**: `mapper/` sends network topology to all affected clients
-
-**Route Management**
-1. **Advertisement**: Clients announce routes via `poll.go` Hostinfo updates
-2. **Storage**: `db/` persists routes, `NodeStore` caches for performance
-3. **Approval**: `policy/` auto-approves routes based on ACL rules
-4. **Distribution**: `routes/` selects primary routes, `mapper/` distributes to peers
-
-### Command-Line Tools (`cmd/`)
-
-**Main Server (`cmd/headscale/`)**
-- `headscale.go`: CLI parsing, configuration loading, server startup
-- Supports daemon mode, CLI operations (user/node management), database operations
-
-**Integration Test Runner (`cmd/hi/`)**
-- `main.go`: Test execution framework with Docker orchestration
-- `run.go`: Individual test execution with artifact collection
-- `doctor.go`: System requirements validation
-- `docker.go`: Container lifecycle management
-- Essential for validating changes against real Tailscale clients
-
-### Generated & External Code
-
-**Protocol Buffers (`proto/` → `gen/`)**
-- Defines gRPC API for headscale management operations
-- Client libraries can generate from these definitions
-- Run `make generate` after modifying `.proto` files
-
-**Integration Testing (`integration/`)**
-- `scenario.go`: Docker test environment setup
-- `tailscale.go`: Tailscale client container management
-- Individual test files for specific functionality areas
-- Real end-to-end validation with network isolation
-
-### Critical Performance Paths
-
-**High-Frequency Operations**
-1. **MapRequest Processing** (`poll.go`): Every 15-60 seconds per client
-2. **NodeStore Reads** (`node_store.go`): Every operation requiring node data
-3. **Policy Evaluation** (`policy/`): On every peer relationship calculation
-4. **Route Lookups** (`routes/`): During network map generation
-
-**Database Write Patterns**
-- **Frequent**: Node heartbeats, endpoint updates, route changes
-- **Moderate**: User operations, policy updates, API key management
-- **Rare**: Schema migrations, bulk operations
-
-### Configuration & Deployment
-
-**Configuration** (`hscontrol/types/config.go`)**
-- Database connection settings (SQLite/PostgreSQL)
-- Network configuration (IP ranges, DNS settings)
-- Policy mode (file vs database)
-- DERP relay configuration
-- OIDC provider settings
-
-**Key Dependencies**
-- **GORM**: Database ORM with migration support
-- **Tailscale Libraries**: Core networking and protocol code
-- **Zerolog**: Structured logging throughout the application
-- **Buf**: Protocol buffer toolchain for code generation
-
-### Development Workflow Integration
-
-The architecture supports incremental development:
-- **Unit Tests**: Focus on individual packages (`*_test.go` files)
-- **Integration Tests**: Validate cross-component interactions
-- **Database Tests**: Extensive migration and data integrity validation
-- **Policy Tests**: ACL rule evaluation and edge cases
-- **Performance Tests**: NodeStore and high-frequency operation validation
-
-## Integration Testing System
-
-### Overview
-Headscale uses Docker-based integration tests with real Tailscale clients to validate end-to-end functionality. The integration test system is complex and requires specialized knowledge for effective execution and debugging.
-
-### **MANDATORY: Use the headscale-integration-tester Agent**
-
-**CRITICAL REQUIREMENT**: For ANY integration test execution, analysis, troubleshooting, or validation, you MUST use the `headscale-integration-tester` agent. This agent contains specialized knowledge about:
-
-- Test execution strategies and timing requirements  
-- Infrastructure vs code issue distinction (99% vs 1% failure patterns)
-- Security-critical debugging rules and forbidden practices
-- Comprehensive artifact analysis workflows
-- Real-world failure patterns from HA debugging experiences
-
-### Quick Reference Commands
-
-```bash
-# Check system requirements (always run first)
-go run ./cmd/hi doctor
-
-# Run single test (recommended for development)  
-go run ./cmd/hi run "TestName"
-
-# Use PostgreSQL for database-heavy tests
-go run ./cmd/hi run "TestName" --postgres
-
-# Pattern matching for related tests
-go run ./cmd/hi run "TestPattern*"
-```
-
-**Critical Notes**:
-- Only ONE test can run at a time (Docker port conflicts)
-- Tests generate ~100MB of logs per run in `control_logs/`
-- Clean environment before each test: `rm -rf control_logs/202507* && docker system prune -f`
-
-### Test Artifacts Location
-All test runs save comprehensive debugging artifacts to `control_logs/TIMESTAMP-ID/` including server logs, client logs, database dumps, MapResponse protocol data, and Prometheus metrics.
-
-**For all integration test work, use the headscale-integration-tester agent - it contains the complete knowledge needed for effective testing and debugging.**
-
-## NodeStore Implementation Details
-
-**Key Insight from Recent Work**: The NodeStore is a critical performance optimization that caches node data in memory while ensuring consistency with the database. When working with route advertisements or node state changes:
-
-1. **Timing Considerations**: Route advertisements need time to propagate from clients to server. Use `require.EventuallyWithT()` patterns in tests instead of immediate assertions.
-
-2. **Synchronization Points**: NodeStore updates happen at specific points like `poll.go:420` after Hostinfo changes. Ensure these are maintained when modifying the polling logic.
-
-3. **Peer Visibility**: The NodeStore's `peersFunc` determines which nodes are visible to each other. Policy-based filtering is separate from monitoring visibility - expired nodes should remain visible for debugging but marked as expired.
-
-## Testing Guidelines
-
-### Integration Test Patterns
-
-#### **CRITICAL: EventuallyWithT Pattern for External Calls**
-
-**All external calls in integration tests MUST be wrapped in EventuallyWithT blocks** to handle eventual consistency in distributed systems. External calls include:
-- `client.Status()` - Getting Tailscale client status
-- `client.Curl()` - Making HTTP requests through clients
-- `client.Traceroute()` - Running network diagnostics
-- `headscale.ListNodes()` - Querying headscale server state
-- Any other calls that interact with external systems or network operations
-
-**Key Rules**:
-1. **Never use bare `require.NoError(t, err)` with external calls** - Always wrap in EventuallyWithT
-2. **Keep related assertions together** - If multiple assertions depend on the same external call, keep them in the same EventuallyWithT block
-3. **Split unrelated external calls** - Different external calls should be in separate EventuallyWithT blocks
-4. **Never nest EventuallyWithT calls** - Each EventuallyWithT should be at the same level
-5. **Declare shared variables at function scope** - Variables used across multiple EventuallyWithT blocks must be declared before first use
-
-**Examples**:
-
-```go
-// CORRECT: External call wrapped in EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    
-    // Related assertions using the same status call
-    for _, peerKey := range status.Peers() {
-        peerStatus := status.Peer[peerKey]
-        assert.NotNil(c, peerStatus.PrimaryRoutes)
-        requirePeerSubnetRoutesWithCollect(c, peerStatus, expectedRoutes)
-    }
-}, 5*time.Second, 200*time.Millisecond, "Verifying client status and routes")
-
-// INCORRECT: Bare external call without EventuallyWithT
-status, err := client.Status()  // ❌ Will fail intermittently
-require.NoError(t, err)
-
-// CORRECT: Separate EventuallyWithT for different external calls
-// First external call - headscale.ListNodes()
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 2)
-    requireNodeRouteCountWithCollect(c, nodes[0], 2, 2, 2)
-}, 10*time.Second, 500*time.Millisecond, "route state changes should propagate to nodes")
-
-// Second external call - client.Status() 
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    
-    for _, peerKey := range status.Peers() {
-        peerStatus := status.Peer[peerKey]
-        requirePeerSubnetRoutesWithCollect(c, peerStatus, []netip.Prefix{tsaddr.AllIPv4(), tsaddr.AllIPv6()})
-    }
-}, 10*time.Second, 500*time.Millisecond, "routes should be visible to client")
-
-// INCORRECT: Multiple unrelated external calls in same EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err := headscale.ListNodes()  // ❌ First external call
-    assert.NoError(c, err)
-    
-    status, err := client.Status()  // ❌ Different external call - should be separate
-    assert.NoError(c, err)
-}, 10*time.Second, 500*time.Millisecond, "mixed calls")
-
-// CORRECT: Variable scoping for shared data
-var (
-    srs1, srs2, srs3       *ipnstate.Status
-    clientStatus           *ipnstate.Status
-    srs1PeerStatus         *ipnstate.PeerStatus
-)
-
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    srs1 = subRouter1.MustStatus()  // = not :=
-    srs2 = subRouter2.MustStatus()
-    clientStatus = client.MustStatus()
-    
-    srs1PeerStatus = clientStatus.Peer[srs1.Self.PublicKey]
-    // assertions...
-}, 5*time.Second, 200*time.Millisecond, "checking router status")
-
-// CORRECT: Wrapping client operations
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    result, err := client.Curl(weburl)
-    assert.NoError(c, err)
-    assert.Len(c, result, 13)
-}, 5*time.Second, 200*time.Millisecond, "Verifying HTTP connectivity")
-
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    tr, err := client.Traceroute(webip)
-    assert.NoError(c, err)
-    assertTracerouteViaIPWithCollect(c, tr, expectedRouter.MustIPv4())
-}, 5*time.Second, 200*time.Millisecond, "Verifying network path")
-```
-
-**Helper Functions**:
-- Use `requirePeerSubnetRoutesWithCollect` instead of `requirePeerSubnetRoutes` inside EventuallyWithT
-- Use `requireNodeRouteCountWithCollect` instead of `requireNodeRouteCount` inside EventuallyWithT
-- Use `assertTracerouteViaIPWithCollect` instead of `assertTracerouteViaIP` inside EventuallyWithT
-
-```go
-// Node route checking by actual node properties, not array position
-var routeNode *v1.Node
-for _, node := range nodes {
-    if nodeIDStr := fmt.Sprintf("%d", node.GetId()); expectedRoutes[nodeIDStr] != "" {
-        routeNode = node
-        break
-    }
-}
-```
-
-### Running Problematic Tests
-- Some tests require significant time (e.g., `TestNodeOnlineStatus` runs for 12 minutes)
-- Infrastructure issues like disk space can cause test failures unrelated to code changes  
-- Use `--postgres` flag when testing database-heavy scenarios
-
-## Quality Assurance and Testing Requirements
-
-### **MANDATORY: Always Use Specialized Testing Agents**
-
-**CRITICAL REQUIREMENT**: For ANY task involving testing, quality assurance, review, or validation, you MUST use the appropriate specialized agent at the END of your task list. This ensures comprehensive quality validation and prevents regressions.
-
-**Required Agents for Different Task Types**:
-
-1. **Integration Testing**: Use `headscale-integration-tester` agent for:
-   - Running integration tests with `cmd/hi`
-   - Analyzing test failures and artifacts
-   - Troubleshooting Docker-based test infrastructure
-   - Validating end-to-end functionality changes
-
-2. **Quality Control**: Use `quality-control-enforcer` agent for:
-   - Code review and validation
-   - Ensuring best practices compliance  
-   - Preventing common pitfalls and anti-patterns
-   - Validating architectural decisions
-
-**Agent Usage Pattern**: Always add the appropriate agent as the FINAL step in any task list to ensure quality validation occurs after all work is complete.
-
-### Integration Test Debugging Reference
-
-Test artifacts are preserved in `control_logs/TIMESTAMP-ID/` including:
-- Headscale server logs (stderr/stdout)
-- Tailscale client logs and status  
-- Database dumps and network captures
-- MapResponse JSON files for protocol debugging
-
-**For integration test issues, ALWAYS use the headscale-integration-tester agent - do not attempt manual debugging.**
-
-## EventuallyWithT Pattern for Integration Tests
-
-### Overview
-EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
-
-### External Calls That Must Be Wrapped
-The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
-- `headscale.ListNodes()` - Queries server state
-- `client.Status()` - Gets client network status
-- `client.Curl()` - Makes HTTP requests through the network
-- `client.Traceroute()` - Performs network diagnostics
-- `client.Execute()` when running commands that query state
-- Any operation that reads from the headscale server or tailscale client
-
-### Operations That Must NOT Be Wrapped
-The following are **blocking operations** that modify state and should NOT be wrapped in EventuallyWithT:
-- `tailscale set` commands (e.g., `--advertise-routes`, `--exit-node`)
-- Any command that changes configuration or state
-- Use `client.MustStatus()` instead of `client.Status()` when you just need the ID for a blocking operation
-
-### Five Key Rules for EventuallyWithT
-
-1. **One External Call Per EventuallyWithT Block**
-   - Each EventuallyWithT should make ONE external call (e.g., ListNodes OR Status)
-   - Related assertions based on that single call can be grouped together
-   - Unrelated external calls must be in separate EventuallyWithT blocks
-
-2. **Variable Scoping**
-   - Declare variables that need to be shared across EventuallyWithT blocks at function scope
-   - Use `=` for assignment inside EventuallyWithT, not `:=` (unless the variable is only used within that block)
-   - Variables declared with `:=` inside EventuallyWithT are not accessible outside
-
-3. **No Nested EventuallyWithT**
-   - NEVER put an EventuallyWithT inside another EventuallyWithT
-   - This is a critical anti-pattern that must be avoided
-
-4. **Use CollectT for Assertions**
-   - Inside EventuallyWithT, use `assert` methods with the CollectT parameter
-   - Helper functions called within EventuallyWithT must accept `*assert.CollectT`
-
-5. **Descriptive Messages**
-   - Always provide a descriptive message as the last parameter
-   - Message should explain what condition is being waited for
-
-### Correct Pattern Examples
-
-```go
-// CORRECT: Blocking operation NOT wrapped
-for _, client := range allClients {
-    status := client.MustStatus()
-    command := []string{
-        "tailscale",
-        "set",
-        "--advertise-routes=" + expectedRoutes[string(status.Self.ID)],
-    }
-    _, _, err = client.Execute(command)
-    require.NoErrorf(t, err, "failed to advertise route: %s", err)
-}
-
-// CORRECT: Single external call with related assertions
-var nodes []*v1.Node
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    nodes, err = headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 2)
-    requireNodeRouteCountWithCollect(c, nodes[0], 2, 2, 2)
-}, 10*time.Second, 500*time.Millisecond, "nodes should have expected route counts")
-
-// CORRECT: Separate EventuallyWithT for different external call
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    for _, peerKey := range status.Peers() {
-        peerStatus := status.Peer[peerKey]
-        requirePeerSubnetRoutesWithCollect(c, peerStatus, expectedPrefixes)
-    }
-}, 10*time.Second, 500*time.Millisecond, "client should see expected routes")
-```
-
-### Incorrect Patterns to Avoid
-
-```go
-// INCORRECT: Blocking operation wrapped in EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    status, err := client.Status()
-    assert.NoError(c, err)
-    
-    // This is a blocking operation - should NOT be in EventuallyWithT!
-    command := []string{
-        "tailscale",
-        "set",
-        "--advertise-routes=" + expectedRoutes[string(status.Self.ID)],
-    }
-    _, _, err = client.Execute(command)
-    assert.NoError(c, err)
-}, 5*time.Second, 200*time.Millisecond, "wrong pattern")
-
-// INCORRECT: Multiple unrelated external calls in same EventuallyWithT
-assert.EventuallyWithT(t, func(c *assert.CollectT) {
-    // First external call
-    nodes, err := headscale.ListNodes()
-    assert.NoError(c, err)
-    assert.Len(c, nodes, 2)
-    
-    // Second unrelated external call - WRONG!
-    status, err := client.Status()
-    assert.NoError(c, err)
-    assert.NotNil(c, status)
-}, 10*time.Second, 500*time.Millisecond, "mixed operations")
-```
-
-## Important Notes
-
-- **Dependencies**: Use `nix develop` for consistent toolchain (Go, buf, protobuf tools, linting)
-- **Protocol Buffers**: Changes to `proto/` require `make generate` and should be committed separately
-- **Code Style**: Enforced via golangci-lint with golines (width 88) and gofumpt formatting
-- **Database**: Supports both SQLite (development) and PostgreSQL (production/testing)
-- **Integration Tests**: Require Docker and can consume significant disk space - use headscale-integration-tester agent
-- **Performance**: NodeStore optimizations are critical for scale - be careful with changes to state management
-- **Quality Assurance**: Always use appropriate specialized agents for testing and validation tasks
-- **NEVER create gists in the user's name**: Do not use the `create_gist` tool - present information directly in the response instead
+@AGENTS.md

Dockerfile.derper

@@ -1,6 +1,6 @@
 # For testing purposes only
 
-FROM golang:alpine AS build-env
+FROM golang:1.26.3-alpine AS build-env
 
 WORKDIR /go/src
 

Dockerfile.integration

@@ -2,28 +2,43 @@
 # and are in no way endorsed by Headscale's maintainers as an
 # official nor supported release or distribution.
 
-FROM docker.io/golang:1.25-trixie
+FROM docker.io/golang:1.26.1-trixie AS builder
 ARG VERSION=dev
 ENV GOPATH /go
 WORKDIR /go/src/headscale
 
-RUN apt-get --update install --no-install-recommends --yes less jq sqlite3 dnsutils \
-  && rm -rf /var/lib/apt/lists/* \
-  && apt-get clean
-RUN mkdir -p /var/run/headscale
-
-# Install delve debugger
+# Install delve debugger first - rarely changes, good cache candidate
 RUN go install github.com/go-delve/delve/cmd/dlv@latest
 
+# Download dependencies - only invalidated when go.mod/go.sum change
 COPY go.mod go.sum /go/src/headscale/
 RUN go mod download
 
+# Copy source and build - invalidated on any source change
 COPY . .
 
 # Build debug binary with debug symbols for delve
 RUN CGO_ENABLED=0 GOOS=linux go build -gcflags="all=-N -l" -o /go/bin/headscale ./cmd/headscale
 
+# Runtime stage
+FROM debian:trixie-slim
+
+RUN apt-get --update install --no-install-recommends --yes \
+    bash ca-certificates curl dnsutils findutils iproute2 jq less procps python3 sqlite3 \
+  && apt-get dist-clean
+
+RUN mkdir -p /var/run/headscale
+
+# Copy binaries from builder
+COPY --from=builder /go/bin/headscale /usr/local/bin/headscale
+COPY --from=builder /go/bin/dlv /usr/local/bin/dlv
+
+# Copy source code for delve source-level debugging
+COPY --from=builder /go/src/headscale /go/src/headscale
+
+WORKDIR /go/src/headscale
+
 # Need to reset the entrypoint or everything will run as a busybox script
 ENTRYPOINT []
 EXPOSE 8080/tcp 40000/tcp
-CMD ["/go/bin/dlv", "--listen=0.0.0.0:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/go/bin/headscale", "--"]
+CMD ["dlv", "--listen=0.0.0.0:40000", "--headless=true", "--api-version=2", "--accept-multiclient", "exec", "/usr/local/bin/headscale", "--"]

Dockerfile.integration-ci

@@ -0,0 +1,17 @@
+# Minimal CI image - expects pre-built headscale binary in build context
+# For local development with delve debugging, use Dockerfile.integration instead
+
+FROM debian:trixie-slim
+
+RUN apt-get --update install --no-install-recommends --yes \
+    bash ca-certificates curl dnsutils findutils iproute2 jq less procps python3 sqlite3 \
+  && apt-get dist-clean
+
+RUN mkdir -p /var/run/headscale
+
+# Copy pre-built headscale binary from build context
+COPY headscale /usr/local/bin/headscale
+
+ENTRYPOINT []
+EXPOSE 8080/tcp
+CMD ["/usr/local/bin/headscale"]

Dockerfile.tailscale-HEAD

@@ -4,7 +4,7 @@
 # This Dockerfile is more or less lifted from tailscale/tailscale
 # to ensure a similar build process when testing the HEAD of tailscale.
 
-FROM golang:1.25-alpine AS build-env
+FROM golang:1.26.3-alpine AS build-env
 
 WORKDIR /go/src
 
@@ -37,7 +37,9 @@ RUN GOARCH=$TARGETARCH go install -tags="${BUILD_TAGS}" -ldflags="\
       -v ./cmd/tailscale ./cmd/tailscaled ./cmd/containerboot
 
 FROM alpine:3.22
-RUN apk add --no-cache ca-certificates iptables iproute2 ip6tables curl
+# Upstream: ca-certificates ip6tables iptables iproute2
+# Tests: curl python3 (traceroute via BusyBox)
+RUN apk add --no-cache ca-certificates curl ip6tables iptables iproute2 python3
 
 COPY --from=build-env /go/bin/* /usr/local/bin/
 # For compat with the previous run.sh, although ideally you should be

Dockerfile.tailscale-rs

@@ -0,0 +1,26 @@
+FROM rust:1.94-bookworm AS builder
+
+ARG TAILSCALE_RS_REPO=https://github.com/tailscale/tailscale-rs.git
+ARG TAILSCALE_RS_REF=main
+
+WORKDIR /app
+RUN git clone --depth 1 --branch "$TAILSCALE_RS_REF" "$TAILSCALE_RS_REPO" .
+
+# Re-export ts_control's insecure-keyfetch feature through the tailscale
+# crate so the axum example can fetch the headscale control key over
+# plain HTTP. The integration harness serves the control plane without
+# TLS, and upstream only allows plain-HTTP key fetches when this Cargo
+# feature is compiled in.
+RUN sed -i '/^axum = \["dep:axum"\]/a insecure-keyfetch = ["ts_control/insecure-keyfetch"]' Cargo.toml
+
+RUN cargo build --release --features axum,insecure-keyfetch --example axum
+
+FROM debian:bookworm-slim
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    ca-certificates \
+    iproute2 \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY --from=builder /app/target/release/examples/axum /usr/local/bin/axum

Makefile

@@ -21,7 +21,7 @@ endef
 # Source file collections using shell find for better performance
 GO_SOURCES := $(shell find . -name '*.go' -not -path './gen/*' -not -path './vendor/*')
 PROTO_SOURCES := $(shell find . -name '*.proto' -not -path './gen/*' -not -path './vendor/*')
-DOC_SOURCES := $(shell find . \( -name '*.md' -o -name '*.yaml' -o -name '*.yml' -o -name '*.ts' -o -name '*.js' -o -name '*.html' -o -name '*.css' -o -name '*.scss' -o -name '*.sass' \) -not -path './gen/*' -not -path './vendor/*' -not -path './node_modules/*')
+PRETTIER_SOURCES := $(shell find . \( -name '*.md' -o -name '*.yaml' -o -name '*.yml' -o -name '*.ts' -o -name '*.js' -o -name '*.html' -o -name '*.css' -o -name '*.scss' -o -name '*.sass' \) -not -path './gen/*' -not -path './vendor/*' -not -path './node_modules/*')
 
 # Default target
 .PHONY: all
@@ -33,6 +33,7 @@ check-deps:
 	$(call check_tool,go)
 	$(call check_tool,golangci-lint)
 	$(call check_tool,gofumpt)
+	$(call check_tool,mdformat)
 	$(call check_tool,prettier)
 	$(call check_tool,clang-format)
 	$(call check_tool,buf)
@@ -52,7 +53,7 @@ test: check-deps $(GO_SOURCES) go.mod go.sum
 
 # Formatting targets
 .PHONY: fmt
-fmt: fmt-go fmt-prettier fmt-proto
+fmt: fmt-go fmt-mdformat fmt-prettier fmt-proto
 
 .PHONY: fmt-go
 fmt-go: check-deps $(GO_SOURCES)
@@ -60,11 +61,15 @@ fmt-go: check-deps $(GO_SOURCES)
 	gofumpt -l -w .
 	golangci-lint run --fix
 
+.PHONY: fmt-mdformat
+fmt-mdformat: check-deps
+	@echo "Formatting documentation..."
+	mdformat docs/
+
 .PHONY: fmt-prettier
-fmt-prettier: check-deps $(DOC_SOURCES)
-	@echo "Formatting documentation and config files..."
+fmt-prettier: check-deps $(PRETTIER_SOURCES)
+	@echo "Formatting markup and config files..."
 	prettier --write '**/*.{ts,js,md,yaml,yml,sass,css,scss,html}'
-	prettier --write --print-width 80 --prose-wrap always CHANGELOG.md
 
 .PHONY: fmt-proto
 fmt-proto: check-deps $(PROTO_SOURCES)
@@ -100,6 +105,11 @@ clean:
 .PHONY: dev
 dev: fmt lint test build
 
+# Start a local headscale dev server (use mts to add nodes)
+.PHONY: dev-server
+dev-server:
+	go run ./cmd/dev
+
 # Help target
 .PHONY: help
 help:
@@ -117,7 +127,8 @@ help:
 	@echo ""
 	@echo "Specific targets:"
 	@echo "  fmt-go       - Format Go code only"
-	@echo "  fmt-prettier - Format documentation only" 
+	@echo "  fmt-mdformat - Format documentation only"
+	@echo "  fmt-prettier - Format markup and config files only"
 	@echo "  fmt-proto    - Format Protocol Buffer files only"
 	@echo "  lint-go      - Lint Go code only"
 	@echo "  lint-proto   - Lint Protocol Buffer files only"
@@ -126,4 +137,4 @@ help:
 	@echo "  check-deps   - Verify required tools are available"
 	@echo ""
 	@echo "Note: If not running in a nix shell, ensure dependencies are available:"
-	@echo "  nix develop"
+	@echo "  nix develop"

README.md

@@ -1,4 +1,4 @@
-![headscale logo](./docs/logo/headscale3_header_stacked_left.png)
+![headscale logo](./docs/assets/logo/headscale3_header_stacked_left.png)
 
 ![ci](https://github.com/juanfont/headscale/actions/workflows/test.yml/badge.svg)
 
@@ -30,8 +30,8 @@ nodes in the Tailscale network. It assigns the IP addresses of the clients,
 creates the boundaries between each user, enables sharing machines between users,
 and exposes the advertised routes of your nodes.
 
-A [Tailscale network (tailnet)](https://tailscale.com/kb/1136/tailnet/) is private
-network which Tailscale assigns to a user in terms of private users or an
+A [Tailscale network (tailnet)](https://tailscale.com/docs/concepts/tailnet) is
+private network which Tailscale assigns to a user in terms of private users or an
 organisation.
 
 ## Design goal
@@ -63,8 +63,18 @@ and container to run Headscale.**
 
 Please have a look at the [`documentation`](https://headscale.net/stable/).
 
+For NixOS users, a module is available in [`nix/`](./nix/).
+
+## Builds from `main`
+
+Development builds from the `main` branch are available as container images and
+binaries. See the [development builds](https://headscale.net/stable/setup/install/main/)
+documentation for details.
+
 ## Talks
 
+- Fosdem 2026 (video): [Headscale & Tailscale: The complementary open source clone](https://fosdem.org/2026/schedule/event/KYQ3LL-headscale-the-complementary-open-source-clone/)
+  - presented by Kristoffer Dalby
 - Fosdem 2023 (video): [Headscale: How we are using integration testing to reimplement Tailscale](https://fosdem.org/2023/schedule/event/goheadscale/)
   - presented by Juan Font Alonso and Kristoffer Dalby
 
@@ -103,6 +113,8 @@ run `make lint` and `make fmt` before committing any code.
 The **Proto** code is linted with [`buf`](https://docs.buf.build/lint/overview) and
 formatted with [`clang-format`](https://clang.llvm.org/docs/ClangFormat.html).
 
+The **docs** are formatted with [`mdformat`](https://mdformat.readthedocs.io).
+
 The **rest** (Markdown, YAML, etc) is formatted with [`prettier`](https://prettier.io).
 
 Check out the `.golangci.yaml` and `Makefile` to see the specific configuration.
@@ -147,6 +159,7 @@ make build
 We recommend using Nix for dependency management to ensure you have all required tools. If you prefer to manage dependencies yourself, you can use Make directly:
 
 **With Nix (recommended):**
+
 ```shell
 nix develop
 make test
@@ -154,6 +167,7 @@ make build
 ```
 
 **With your own dependencies:**
+
 ```shell
 make test
 make build

cmd/dev/README.md

@@ -0,0 +1,96 @@
+# cmd/dev -- Local Development Environment
+
+Starts a headscale server on localhost with a pre-created user and
+pre-auth key. Pair with `mts` to add real tailscale nodes.
+
+## Quick start
+
+```bash
+# Terminal 1: start headscale
+go run ./cmd/dev
+
+# Terminal 2: start mts server
+go tool mts server run
+
+# Terminal 3: add and connect nodes
+go tool mts server add node1
+go tool mts server add node2
+
+# Disable logtail (avoids startup delays, see "Known issues" below)
+for n in node1 node2; do
+  cat > ~/.config/multi-tailscale-dev/$n/env.txt << 'EOF'
+TS_NO_LOGS_NO_SUPPORT=true
+EOF
+done
+
+# Restart nodes so env.txt takes effect
+go tool mts server stop node1 && go tool mts server start node1
+go tool mts server stop node2 && go tool mts server start node2
+
+# Connect to headscale (use the auth key printed by cmd/dev)
+go tool mts node1 up --login-server=http://127.0.0.1:8080 --authkey=<KEY> --reset
+go tool mts node2 up --login-server=http://127.0.0.1:8080 --authkey=<KEY> --reset
+
+# Verify
+go tool mts node1 status
+```
+
+## Flags
+
+| Flag     | Default | Description                  |
+| -------- | ------- | ---------------------------- |
+| `--port` | 8080    | Headscale listen port        |
+| `--keep` | false   | Keep state directory on exit |
+
+The metrics/debug port is `port + 1010` (default 9090) and the gRPC
+port is `port + 42363` (default 50443).
+
+## What it does
+
+1. Builds the headscale binary into a temp directory
+2. Writes a minimal dev config (SQLite, public DERP, debug logging)
+3. Starts `headscale serve` as a subprocess
+4. Creates a "dev" user and a reusable 24h pre-auth key via the CLI
+5. Prints a banner with server URL, auth key, and usage instructions
+6. Blocks until Ctrl+C, then kills headscale
+
+State lives in `/tmp/headscale-dev-*/`. Pass `--keep` to preserve it
+across restarts (useful for inspecting the database or reusing keys).
+
+## Useful endpoints
+
+- `http://127.0.0.1:8080/health` -- health check
+- `http://127.0.0.1:9090/debug/ping` -- interactive ping UI
+- `http://127.0.0.1:9090/debug/ping?node=1` -- quick-ping a node
+- `POST http://127.0.0.1:9090/debug/ping` with `node=<id>` -- trigger ping
+
+## Managing headscale
+
+The banner prints the full path to the built binary and config. Use it
+for any headscale CLI command:
+
+```bash
+/tmp/headscale-dev-*/headscale -c /tmp/headscale-dev-*/config.yaml nodes list
+/tmp/headscale-dev-*/headscale -c /tmp/headscale-dev-*/config.yaml users list
+```
+
+## Known issues
+
+### Logtail delays on mts nodes
+
+Freshly created `mts` instances may take 30+ seconds to start if
+`~/.local/share/tailscale/` contains stale logtail cache from previous
+tailscaled runs. The daemon blocks trying to upload old logs before
+creating its socket.
+
+Fix: write `TS_NO_LOGS_NO_SUPPORT=true` to each instance's `env.txt`
+before starting (or restart after writing). See the quick start above.
+
+### mts node cleanup
+
+`mts` stores state in `~/.config/multi-tailscale-dev/`. Old instances
+accumulate over time. Clean them with:
+
+```bash
+go tool mts server rm <name>
+```

cmd/dev/main.go

@@ -0,0 +1,314 @@
+// cmd/dev starts a local headscale development server with a pre-created
+// user and pre-auth key, ready for connecting tailscale nodes via mts.
+package main
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"flag"
+	"fmt"
+	"log"
+	"net/http"
+	"os"
+	"os/exec"
+	"os/signal"
+	"path/filepath"
+	"strconv"
+	"syscall"
+	"time"
+)
+
+var (
+	port = flag.Int("port", 8080, "headscale listen port")
+	keep = flag.Bool("keep", false, "keep state directory on exit")
+)
+
+var errHealthTimeout = errors.New("health check timed out")
+
+var errEmptyAuthKey = errors.New("empty auth key in response")
+
+// maxDevPort is the highest --port value that keeps both the derived
+// metrics port (port+1010) and gRPC port (port+42363) inside the valid
+// 1..65535 TCP range.
+const maxDevPort = 23172
+
+const devConfig = `---
+server_url: http://127.0.0.1:%d
+listen_addr: 127.0.0.1:%d
+metrics_listen_addr: 127.0.0.1:%d
+grpc_listen_addr: 127.0.0.1:%d
+grpc_allow_insecure: true
+
+noise:
+  private_key_path: %s/noise_private.key
+
+prefixes:
+  v4: 100.64.0.0/10
+  v6: fd7a:115c:a1e0::/48
+  allocation: sequential
+
+database:
+  type: sqlite
+  sqlite:
+    path: %s/db.sqlite
+    write_ahead_log: true
+
+derp:
+  server:
+    enabled: false
+  urls:
+    - https://controlplane.tailscale.com/derpmap/default
+  auto_update_enabled: false
+
+dns:
+  magic_dns: true
+  base_domain: headscale.dev
+  override_local_dns: false
+
+log:
+  level: debug
+  format: text
+
+policy:
+  mode: database
+
+unix_socket: %s/headscale.sock
+unix_socket_permission: "0770"
+`
+
+func main() {
+	flag.Parse()
+	log.SetFlags(0)
+
+	if *port < 1 || *port > maxDevPort {
+		log.Fatalf(
+			"--port must be in 1..%d (higher values overflow the derived gRPC port); got %d",
+			maxDevPort, *port,
+		)
+	}
+
+	http.DefaultClient.Timeout = 2 * time.Second
+	http.DefaultClient.CheckRedirect = func(*http.Request, []*http.Request) error {
+		return http.ErrUseLastResponse
+	}
+
+	err := run()
+	if err != nil {
+		log.Fatal(err)
+	}
+}
+
+func run() error {
+	metricsPort := *port + 1010 // default 9090
+	grpcPort := *port + 42363   // default 50443
+
+	tmpDir, err := os.MkdirTemp("", "headscale-dev-")
+	if err != nil {
+		return fmt.Errorf("creating temp dir: %w", err)
+	}
+
+	if !*keep {
+		defer os.RemoveAll(tmpDir)
+	}
+
+	// Write config.
+	configPath := filepath.Join(tmpDir, "config.yaml")
+	configContent := fmt.Sprintf(devConfig,
+		*port, *port, metricsPort, grpcPort,
+		tmpDir, tmpDir, tmpDir,
+	)
+
+	err = os.WriteFile(configPath, []byte(configContent), 0o600)
+	if err != nil {
+		return fmt.Errorf("writing config: %w", err)
+	}
+
+	// Build headscale.
+	fmt.Println("Building headscale...")
+
+	hsBin := filepath.Join(tmpDir, "headscale")
+
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	build := exec.CommandContext(ctx, "go", "build", "-o", hsBin, "./cmd/headscale")
+	build.Stdout = os.Stdout
+	build.Stderr = os.Stderr
+
+	err = build.Run()
+	if err != nil {
+		return fmt.Errorf("building headscale: %w", err)
+	}
+
+	// Start headscale serve.
+	fmt.Println("Starting headscale server...")
+
+	serve := exec.CommandContext(ctx, hsBin, "serve", "-c", configPath)
+	serve.Stdout = os.Stdout
+	serve.Stderr = os.Stderr
+
+	err = serve.Start()
+	if err != nil {
+		return fmt.Errorf("starting headscale: %w", err)
+	}
+
+	// Wait for server to be ready.
+	healthURL := fmt.Sprintf("http://127.0.0.1:%d/health", *port)
+
+	err = waitForHealth(ctx, healthURL, 30*time.Second)
+	if err != nil {
+		return fmt.Errorf("waiting for headscale: %w", err)
+	}
+
+	// Create user.
+	fmt.Println("Creating user and pre-auth key...")
+
+	userJSON, err := runHS(ctx, hsBin, configPath, "users", "create", "dev", "-o", "json")
+	if err != nil {
+		return fmt.Errorf("creating user: %w", err)
+	}
+
+	userID, err := extractUserID(userJSON)
+	if err != nil {
+		return fmt.Errorf("parsing user: %w", err)
+	}
+
+	// Create pre-auth key.
+	keyJSON, err := runHS(
+		ctx, hsBin, configPath,
+		"preauthkeys", "create",
+		"-u", strconv.FormatUint(userID, 10),
+		"--reusable",
+		"-e", "24h",
+		"-o", "json",
+	)
+	if err != nil {
+		return fmt.Errorf("creating pre-auth key: %w", err)
+	}
+
+	authKey, err := extractAuthKey(keyJSON)
+	if err != nil {
+		return fmt.Errorf("parsing pre-auth key: %w", err)
+	}
+
+	// Print banner.
+	fmt.Printf(`
+=== Headscale Dev Environment ===
+  Server:  http://127.0.0.1:%d
+  Metrics: http://127.0.0.1:%d
+  Debug:   http://127.0.0.1:%d/debug/ping
+  Config:  %s
+  State:   %s
+
+Pre-auth key: %s
+
+Connect nodes with mts:
+  go tool mts server run                  # start mts (once, another terminal)
+  go tool mts server add node1            # create a node
+  go tool mts node1 up --login-server=http://127.0.0.1:%d --authkey=%s
+  go tool mts node1 status                # check connection
+
+Manage headscale:
+  %s -c %s nodes list
+  %s -c %s users list
+
+Press Ctrl+C to stop.
+`,
+		*port, metricsPort, metricsPort,
+		configPath, tmpDir,
+		authKey,
+		*port, authKey,
+		hsBin, configPath,
+		hsBin, configPath,
+	)
+
+	// Wait for headscale to exit.
+	err = serve.Wait()
+	if err != nil {
+		// Context cancellation is expected on Ctrl+C.
+		if ctx.Err() != nil {
+			fmt.Println("\nShutting down...")
+
+			return nil
+		}
+
+		return fmt.Errorf("headscale exited: %w", err)
+	}
+
+	return nil
+}
+
+// waitForHealth polls the health endpoint until it returns 200 or the
+// timeout expires.
+func waitForHealth(ctx context.Context, url string, timeout time.Duration) error {
+	deadline := time.Now().Add(timeout)
+
+	for time.Now().Before(deadline) {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+
+		req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+		if err != nil {
+			return fmt.Errorf("creating request: %w", err)
+		}
+
+		resp, err := http.DefaultClient.Do(req)
+		if err == nil {
+			resp.Body.Close()
+
+			if resp.StatusCode == http.StatusOK {
+				return nil
+			}
+		}
+
+		// Busy-wait is acceptable for a dev tool polling a local server.
+		time.Sleep(200 * time.Millisecond) //nolint:forbidigo
+	}
+
+	return errHealthTimeout
+}
+
+// runHS executes a headscale CLI command and returns its stdout.
+func runHS(ctx context.Context, bin, config string, args ...string) ([]byte, error) {
+	fullArgs := append([]string{"-c", config}, args...)
+	cmd := exec.CommandContext(ctx, bin, fullArgs...)
+	cmd.Stderr = os.Stderr
+
+	return cmd.Output()
+}
+
+// extractUserID parses the JSON output of "users create" and returns the
+// user ID.
+func extractUserID(data []byte) (uint64, error) {
+	var user struct {
+		ID uint64 `json:"id"`
+	}
+
+	err := json.Unmarshal(data, &user)
+	if err != nil {
+		return 0, fmt.Errorf("unmarshalling user JSON: %w (raw: %s)", err, data)
+	}
+
+	return user.ID, nil
+}
+
+// extractAuthKey parses the JSON output of "preauthkeys create" and
+// returns the key string.
+func extractAuthKey(data []byte) (string, error) {
+	var key struct {
+		Key string `json:"key"`
+	}
+
+	err := json.Unmarshal(data, &key)
+	if err != nil {
+		return "", fmt.Errorf("unmarshalling key JSON: %w (raw: %s)", err, data)
+	}
+
+	if key.Key == "" {
+		return "", errEmptyAuthKey
+	}
+
+	return key.Key, nil
+}

cmd/headscale/cli/api_key.go

@@ -1,21 +1,18 @@
 package cli
 
 import (
+	"context"
 	"fmt"
 	"strconv"
-	"time"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/juanfont/headscale/hscontrol/util"
-	"github.com/prometheus/common/model"
 	"github.com/pterm/pterm"
-	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
-	"google.golang.org/protobuf/types/known/timestamppb"
 )
 
 const (
-	// 90 days.
+	// DefaultAPIKeyExpiry is 90 days.
 	DefaultAPIKeyExpiry = "90d"
 )
 
@@ -29,15 +26,11 @@ func init() {
 	apiKeysCmd.AddCommand(createAPIKeyCmd)
 
 	expireAPIKeyCmd.Flags().StringP("prefix", "p", "", "ApiKey prefix")
-	if err := expireAPIKeyCmd.MarkFlagRequired("prefix"); err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
+	expireAPIKeyCmd.Flags().Uint64P("id", "i", 0, "ApiKey ID")
 	apiKeysCmd.AddCommand(expireAPIKeyCmd)
 
 	deleteAPIKeyCmd.Flags().StringP("prefix", "p", "", "ApiKey prefix")
-	if err := deleteAPIKeyCmd.MarkFlagRequired("prefix"); err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
+	deleteAPIKeyCmd.Flags().Uint64P("id", "i", 0, "ApiKey ID")
 	apiKeysCmd.AddCommand(deleteAPIKeyCmd)
 }
 
@@ -51,55 +44,35 @@ var listAPIKeys = &cobra.Command{
 	Use:     "list",
 	Short:   "List the Api keys for headscale",
 	Aliases: []string{"ls", "show"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		request := &v1.ListApiKeysRequest{}
-
-		response, err := client.ListApiKeys(ctx, request)
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		response, err := client.ListApiKeys(ctx, &v1.ListApiKeysRequest{})
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting the list of keys: %s", err),
-				output,
-			)
+			return fmt.Errorf("listing api keys: %w", err)
 		}
 
-		if output != "" {
-			SuccessOutput(response.GetApiKeys(), "", output)
-		}
-
-		tableData := pterm.TableData{
-			{"ID", "Prefix", "Expiration", "Created"},
-		}
-		for _, key := range response.GetApiKeys() {
-			expiration := "-"
-
-			if key.GetExpiration() != nil {
-				expiration = ColourTime(key.GetExpiration().AsTime())
+		return printListOutput(cmd, response.GetApiKeys(), func() error {
+			tableData := pterm.TableData{
+				{"ID", "Prefix", "Expiration", "Created"},
 			}
 
-			tableData = append(tableData, []string{
-				strconv.FormatUint(key.GetId(), util.Base10),
-				key.GetPrefix(),
-				expiration,
-				key.GetCreatedAt().AsTime().Format(HeadscaleDateTimeFormat),
-			})
+			for _, key := range response.GetApiKeys() {
+				expiration := "-"
 
-		}
-		err = pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to render pterm table: %s", err),
-				output,
-			)
-		}
-	},
+				if key.GetExpiration() != nil {
+					expiration = ColourTime(key.GetExpiration().AsTime())
+				}
+
+				tableData = append(tableData, []string{
+					strconv.FormatUint(key.GetId(), util.Base10),
+					key.GetPrefix(),
+					expiration,
+					key.GetCreatedAt().AsTime().Format(HeadscaleDateTimeFormat),
+				})
+			}
+
+			return pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
+		})
+	}),
 }
 
 var createAPIKeyCmd = &cobra.Command{
@@ -108,115 +81,81 @@ var createAPIKeyCmd = &cobra.Command{
 	Long: `
 Creates a new Api key, the Api key is only visible on creation
 and cannot be retrieved again.
-If you loose a key, create a new one and revoke (expire) the old one.`,
+If you lose a key, create a new one and revoke (expire) the old one.`,
 	Aliases: []string{"c", "new"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		request := &v1.CreateApiKeyRequest{}
-
-		durationStr, _ := cmd.Flags().GetString("expiration")
-
-		duration, err := model.ParseDuration(durationStr)
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		expiration, err := expirationFromFlag(cmd)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Could not parse duration: %s\n", err),
-				output,
-			)
+			return err
 		}
 
-		expiration := time.Now().UTC().Add(time.Duration(duration))
-
-		request.Expiration = timestamppb.New(expiration)
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		response, err := client.CreateApiKey(ctx, request)
+		response, err := client.CreateApiKey(ctx, &v1.CreateApiKeyRequest{
+			Expiration: expiration,
+		})
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Cannot create Api Key: %s\n", err),
-				output,
-			)
+			return fmt.Errorf("creating api key: %w", err)
 		}
 
-		SuccessOutput(response.GetApiKey(), response.GetApiKey(), output)
-	},
+		return printOutput(cmd, response.GetApiKey(), response.GetApiKey())
+	}),
+}
+
+// apiKeyIDOrPrefix reads --id and --prefix from cmd and validates that
+// exactly one is provided.
+func apiKeyIDOrPrefix(cmd *cobra.Command) (uint64, string, error) {
+	id, _ := cmd.Flags().GetUint64("id")
+	prefix, _ := cmd.Flags().GetString("prefix")
+
+	switch {
+	case id == 0 && prefix == "":
+		return 0, "", fmt.Errorf("either --id or --prefix must be provided: %w", errMissingParameter)
+	case id != 0 && prefix != "":
+		return 0, "", fmt.Errorf("only one of --id or --prefix can be provided: %w", errMissingParameter)
+	}
+
+	return id, prefix, nil
 }
 
 var expireAPIKeyCmd = &cobra.Command{
 	Use:     "expire",
 	Short:   "Expire an ApiKey",
 	Aliases: []string{"revoke", "exp", "e"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		prefix, err := cmd.Flags().GetString("prefix")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, prefix, err := apiKeyIDOrPrefix(cmd)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting prefix from CLI flag: %s", err),
-				output,
-			)
+			return err
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		request := &v1.ExpireApiKeyRequest{
+		response, err := client.ExpireApiKey(ctx, &v1.ExpireApiKeyRequest{
+			Id:     id,
 			Prefix: prefix,
-		}
-
-		response, err := client.ExpireApiKey(ctx, request)
+		})
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Cannot expire Api Key: %s\n", err),
-				output,
-			)
+			return fmt.Errorf("expiring api key: %w", err)
 		}
 
-		SuccessOutput(response, "Key expired", output)
-	},
+		return printOutput(cmd, response, "Key expired")
+	}),
 }
 
 var deleteAPIKeyCmd = &cobra.Command{
 	Use:     "delete",
 	Short:   "Delete an ApiKey",
 	Aliases: []string{"remove", "del"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		prefix, err := cmd.Flags().GetString("prefix")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, prefix, err := apiKeyIDOrPrefix(cmd)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting prefix from CLI flag: %s", err),
-				output,
-			)
+			return err
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		request := &v1.DeleteApiKeyRequest{
+		response, err := client.DeleteApiKey(ctx, &v1.DeleteApiKeyRequest{
+			Id:     id,
 			Prefix: prefix,
-		}
-
-		response, err := client.DeleteApiKey(ctx, request)
+		})
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Cannot delete Api Key: %s\n", err),
-				output,
-			)
+			return fmt.Errorf("deleting api key: %w", err)
 		}
 
-		SuccessOutput(response, "Key deleted", output)
-	},
+		return printOutput(cmd, response, "Key deleted")
+	}),
 }

cmd/headscale/cli/auth.go

@@ -0,0 +1,93 @@
+package cli
+
+import (
+	"context"
+	"fmt"
+
+	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
+	"github.com/spf13/cobra"
+)
+
+func init() {
+	rootCmd.AddCommand(authCmd)
+
+	authRegisterCmd.Flags().StringP("user", "u", "", "User")
+	authRegisterCmd.Flags().String("auth-id", "", "Auth ID")
+	mustMarkRequired(authRegisterCmd, "user", "auth-id")
+	authCmd.AddCommand(authRegisterCmd)
+
+	authApproveCmd.Flags().String("auth-id", "", "Auth ID")
+	mustMarkRequired(authApproveCmd, "auth-id")
+	authCmd.AddCommand(authApproveCmd)
+
+	authRejectCmd.Flags().String("auth-id", "", "Auth ID")
+	mustMarkRequired(authRejectCmd, "auth-id")
+	authCmd.AddCommand(authRejectCmd)
+}
+
+var authCmd = &cobra.Command{
+	Use:   "auth",
+	Short: "Manage node authentication and approval",
+}
+
+var authRegisterCmd = &cobra.Command{
+	Use:   "register",
+	Short: "Register a node to your network",
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		user, _ := cmd.Flags().GetString("user")
+		authID, _ := cmd.Flags().GetString("auth-id")
+
+		request := &v1.AuthRegisterRequest{
+			AuthId: authID,
+			User:   user,
+		}
+
+		response, err := client.AuthRegister(ctx, request)
+		if err != nil {
+			return fmt.Errorf("registering node: %w", err)
+		}
+
+		return printOutput(
+			cmd,
+			response.GetNode(),
+			fmt.Sprintf("Node %s registered", response.GetNode().GetGivenName()))
+	}),
+}
+
+var authApproveCmd = &cobra.Command{
+	Use:   "approve",
+	Short: "Approve a pending authentication request",
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		authID, _ := cmd.Flags().GetString("auth-id")
+
+		request := &v1.AuthApproveRequest{
+			AuthId: authID,
+		}
+
+		response, err := client.AuthApprove(ctx, request)
+		if err != nil {
+			return fmt.Errorf("approving auth request: %w", err)
+		}
+
+		return printOutput(cmd, response, "Auth request approved")
+	}),
+}
+
+var authRejectCmd = &cobra.Command{
+	Use:   "reject",
+	Short: "Reject a pending authentication request",
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		authID, _ := cmd.Flags().GetString("auth-id")
+
+		request := &v1.AuthRejectRequest{
+			AuthId: authID,
+		}
+
+		response, err := client.AuthReject(ctx, request)
+		if err != nil {
+			return fmt.Errorf("rejecting auth request: %w", err)
+		}
+
+		return printOutput(cmd, response, "Auth request rejected")
+	}),
+}

cmd/headscale/cli/configtest.go

@@ -1,7 +1,8 @@
 package cli
 
 import (
-	"github.com/rs/zerolog/log"
+	"fmt"
+
 	"github.com/spf13/cobra"
 )
 
@@ -13,10 +14,12 @@ var configTestCmd = &cobra.Command{
 	Use:   "configtest",
 	Short: "Test the configuration.",
 	Long:  "Run a test of the configuration and exit.",
-	Run: func(cmd *cobra.Command, args []string) {
+	RunE: func(cmd *cobra.Command, args []string) error {
 		_, err := newHeadscaleServerWithConfig()
 		if err != nil {
-			log.Fatal().Caller().Err(err).Msg("Error initializing")
+			return fmt.Errorf("configuration error: %w", err)
 		}
+
+		return nil
 	},
 }

cmd/headscale/cli/debug.go

@@ -1,48 +1,22 @@
 package cli
 
 import (
+	"context"
 	"fmt"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/juanfont/headscale/hscontrol/types"
-	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
-	"google.golang.org/grpc/status"
 )
 
-const (
-	errPreAuthKeyMalformed = Error("key is malformed. expected 64 hex characters with `nodekey` prefix")
-)
-
-// Error is used to compare errors as per https://dave.cheney.net/2016/04/07/constant-errors
-type Error string
-
-func (e Error) Error() string { return string(e) }
-
 func init() {
 	rootCmd.AddCommand(debugCmd)
 
 	createNodeCmd.Flags().StringP("name", "", "", "Name")
-	err := createNodeCmd.MarkFlagRequired("name")
-	if err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
 	createNodeCmd.Flags().StringP("user", "u", "", "User")
-
-	createNodeCmd.Flags().StringP("namespace", "n", "", "User")
-	createNodeNamespaceFlag := createNodeCmd.Flags().Lookup("namespace")
-	createNodeNamespaceFlag.Deprecated = deprecateNamespaceMessage
-	createNodeNamespaceFlag.Hidden = true
-
-	err = createNodeCmd.MarkFlagRequired("user")
-	if err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
 	createNodeCmd.Flags().StringP("key", "k", "", "Key")
-	err = createNodeCmd.MarkFlagRequired("key")
-	if err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
+	mustMarkRequired(createNodeCmd, "name", "user", "key")
+
 	createNodeCmd.Flags().
 		StringSliceP("route", "r", []string{}, "List (or repeated flags) of routes to advertise")
 
@@ -57,54 +31,18 @@ var debugCmd = &cobra.Command{
 
 var createNodeCmd = &cobra.Command{
 	Use:   "create-node",
-	Short: "Create a node that can be registered with `nodes register <>` command",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	Short: "Create a node that can be registered with `auth register <>` command",
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		user, _ := cmd.Flags().GetString("user")
+		name, _ := cmd.Flags().GetString("name")
+		registrationID, _ := cmd.Flags().GetString("key")
 
-		user, err := cmd.Flags().GetString("user")
+		_, err := types.AuthIDFromString(registrationID)
 		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
+			return fmt.Errorf("parsing machine key: %w", err)
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		name, err := cmd.Flags().GetString("name")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting node from flag: %s", err),
-				output,
-			)
-		}
-
-		registrationID, err := cmd.Flags().GetString("key")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting key from flag: %s", err),
-				output,
-			)
-		}
-
-		_, err = types.RegistrationIDFromString(registrationID)
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to parse machine key from flag: %s", err),
-				output,
-			)
-		}
-
-		routes, err := cmd.Flags().GetStringSlice("route")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting routes from flag: %s", err),
-				output,
-			)
-		}
+		routes, _ := cmd.Flags().GetStringSlice("route")
 
 		request := &v1.DebugCreateNodeRequest{
 			Key:    registrationID,
@@ -115,13 +53,9 @@ var createNodeCmd = &cobra.Command{
 
 		response, err := client.DebugCreateNode(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot create node: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("creating node: %w", err)
 		}
 
-		SuccessOutput(response.GetNode(), "Node created", output)
-	},
+		return printOutput(cmd, response.GetNode(), "Node created")
+	}),
 }

cmd/headscale/cli/dump_config.go

@@ -15,14 +15,12 @@ var dumpConfigCmd = &cobra.Command{
 	Use:    "dumpConfig",
 	Short:  "dump current config to /etc/headscale/config.dump.yaml, integration test only",
 	Hidden: true,
-	Args: func(cmd *cobra.Command, args []string) error {
-		return nil
-	},
-	Run: func(cmd *cobra.Command, args []string) {
+	RunE: func(cmd *cobra.Command, args []string) error {
 		err := viper.WriteConfigAs("/etc/headscale/config.dump.yaml")
 		if err != nil {
-			//nolint
-			fmt.Println("Failed to dump config")
+			return fmt.Errorf("dumping config: %w", err)
 		}
+
+		return nil
 	},
 }

cmd/headscale/cli/generate.go

@@ -21,22 +21,17 @@ var generateCmd = &cobra.Command{
 var generatePrivateKeyCmd = &cobra.Command{
 	Use:   "private-key",
 	Short: "Generate a private key for the headscale server",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	RunE: func(cmd *cobra.Command, args []string) error {
 		machineKey := key.NewMachine()
 
 		machineKeyStr, err := machineKey.MarshalText()
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting machine key from flag: %s", err),
-				output,
-			)
+			return fmt.Errorf("marshalling machine key: %w", err)
 		}
 
-		SuccessOutput(map[string]string{
+		return printOutput(cmd, map[string]string{
 			"private_key": string(machineKeyStr),
 		},
-			string(machineKeyStr), output)
+			string(machineKeyStr))
 	},
 }

cmd/headscale/cli/health.go

@@ -1,6 +1,9 @@
 package cli
 
 import (
+	"context"
+	"fmt"
+
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/spf13/cobra"
 )
@@ -13,17 +16,12 @@ var healthCmd = &cobra.Command{
 	Use:   "health",
 	Short: "Check the health of the Headscale server",
 	Long:  "Check the health of the Headscale server. This command will return an exit code of 0 if the server is healthy, or 1 if it is not.",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
 		response, err := client.Health(ctx, &v1.HealthRequest{})
 		if err != nil {
-			ErrorOutput(err, "Error checking health", output)
+			return fmt.Errorf("checking health: %w", err)
 		}
 
-		SuccessOutput(response, "", output)
-	},
+		return printOutput(cmd, response, "")
+	}),
 }

cmd/headscale/cli/mockoidc.go

@@ -1,8 +1,8 @@
 package cli
 
 import (
+	"context"
 	"encoding/json"
-	"errors"
 	"fmt"
 	"net"
 	"net/http"
@@ -10,15 +10,22 @@ import (
 	"strconv"
 	"time"
 
+	"github.com/juanfont/headscale/hscontrol/util/zlog/zf"
 	"github.com/oauth2-proxy/mockoidc"
 	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
 )
 
+// Error is used to compare errors as per https://dave.cheney.net/2016/04/07/constant-errors
+type Error string
+
+func (e Error) Error() string { return string(e) }
+
 const (
 	errMockOidcClientIDNotDefined     = Error("MOCKOIDC_CLIENT_ID not defined")
 	errMockOidcClientSecretNotDefined = Error("MOCKOIDC_CLIENT_SECRET not defined")
 	errMockOidcPortNotDefined         = Error("MOCKOIDC_PORT not defined")
+	errMockOidcUsersNotDefined        = Error("MOCKOIDC_USERS not defined")
 	refreshTTL                        = 60 * time.Minute
 )
 
@@ -32,12 +39,13 @@ var mockOidcCmd = &cobra.Command{
 	Use:   "mockoidc",
 	Short: "Runs a mock OIDC server for testing",
 	Long:  "This internal command runs a OpenID Connect for testing purposes",
-	Run: func(cmd *cobra.Command, args []string) {
+	RunE: func(cmd *cobra.Command, args []string) error {
 		err := mockOIDC()
 		if err != nil {
-			log.Error().Err(err).Msgf("Error running mock OIDC server")
-			os.Exit(1)
+			return fmt.Errorf("running mock OIDC server: %w", err)
 		}
+
+		return nil
 	},
 }
 
@@ -46,41 +54,47 @@ func mockOIDC() error {
 	if clientID == "" {
 		return errMockOidcClientIDNotDefined
 	}
+
 	clientSecret := os.Getenv("MOCKOIDC_CLIENT_SECRET")
 	if clientSecret == "" {
 		return errMockOidcClientSecretNotDefined
 	}
+
 	addrStr := os.Getenv("MOCKOIDC_ADDR")
 	if addrStr == "" {
 		return errMockOidcPortNotDefined
 	}
+
 	portStr := os.Getenv("MOCKOIDC_PORT")
 	if portStr == "" {
 		return errMockOidcPortNotDefined
 	}
+
 	accessTTLOverride := os.Getenv("MOCKOIDC_ACCESS_TTL")
 	if accessTTLOverride != "" {
 		newTTL, err := time.ParseDuration(accessTTLOverride)
 		if err != nil {
 			return err
 		}
+
 		accessTTL = newTTL
 	}
 
 	userStr := os.Getenv("MOCKOIDC_USERS")
 	if userStr == "" {
-		return errors.New("MOCKOIDC_USERS not defined")
+		return errMockOidcUsersNotDefined
 	}
 
 	var users []mockoidc.MockUser
+
 	err := json.Unmarshal([]byte(userStr), &users)
 	if err != nil {
 		return fmt.Errorf("unmarshalling users: %w", err)
 	}
 
-	log.Info().Interface("users", users).Msg("loading users from JSON")
+	log.Info().Interface(zf.Users, users).Msg("loading users from JSON")
 
-	log.Info().Msgf("Access token TTL: %s", accessTTL)
+	log.Info().Msgf("access token TTL: %s", accessTTL)
 
 	port, err := strconv.Atoi(portStr)
 	if err != nil {
@@ -92,7 +106,7 @@ func mockOIDC() error {
 		return err
 	}
 
-	listener, err := net.Listen("tcp", fmt.Sprintf("%s:%d", addrStr, port))
+	listener, err := new(net.ListenConfig).Listen(context.Background(), "tcp", fmt.Sprintf("%s:%d", addrStr, port))
 	if err != nil {
 		return err
 	}
@@ -101,8 +115,10 @@ func mockOIDC() error {
 	if err != nil {
 		return err
 	}
-	log.Info().Msgf("Mock OIDC server listening on %s", listener.Addr().String())
-	log.Info().Msgf("Issuer: %s", mock.Issuer())
+
+	log.Info().Msgf("mock OIDC server listening on %s", listener.Addr().String())
+	log.Info().Msgf("issuer: %s", mock.Issuer())
+
 	c := make(chan struct{})
 	<-c
 
@@ -133,12 +149,13 @@ func getMockOIDC(clientID string, clientSecret string, users []mockoidc.MockUser
 		ErrorQueue:                    &mockoidc.ErrorQueue{},
 	}
 
-	mock.AddMiddleware(func(h http.Handler) http.Handler {
+	_ = mock.AddMiddleware(func(h http.Handler) http.Handler {
 		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-			log.Info().Msgf("Request: %+v", r)
+			log.Info().Msgf("request: %+v", r)
 			h.ServeHTTP(w, r)
+
 			if r.Response != nil {
-				log.Info().Msgf("Response: %+v", r.Response)
+				log.Info().Msgf("response: %+v", r.Response)
 			}
 		})
 	})

cmd/headscale/cli/nodes.go

@@ -1,10 +1,9 @@
 package cli
 
 import (
+	"context"
 	"fmt"
-	"log"
 	"net/netip"
-	"slices"
 	"strconv"
 	"strings"
 	"time"
@@ -14,7 +13,6 @@ import (
 	"github.com/pterm/pterm"
 	"github.com/samber/lo"
 	"github.com/spf13/cobra"
-	"google.golang.org/grpc/status"
 	"google.golang.org/protobuf/types/known/timestamppb"
 	"tailscale.com/types/key"
 )
@@ -22,84 +20,37 @@ import (
 func init() {
 	rootCmd.AddCommand(nodeCmd)
 	listNodesCmd.Flags().StringP("user", "u", "", "Filter by user")
-	listNodesCmd.Flags().BoolP("tags", "t", false, "Show tags")
-
-	listNodesCmd.Flags().StringP("namespace", "n", "", "User")
-	listNodesNamespaceFlag := listNodesCmd.Flags().Lookup("namespace")
-	listNodesNamespaceFlag.Deprecated = deprecateNamespaceMessage
-	listNodesNamespaceFlag.Hidden = true
 	nodeCmd.AddCommand(listNodesCmd)
 
 	listNodeRoutesCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
 	nodeCmd.AddCommand(listNodeRoutesCmd)
 
 	registerNodeCmd.Flags().StringP("user", "u", "", "User")
-
-	registerNodeCmd.Flags().StringP("namespace", "n", "", "User")
-	registerNodeNamespaceFlag := registerNodeCmd.Flags().Lookup("namespace")
-	registerNodeNamespaceFlag.Deprecated = deprecateNamespaceMessage
-	registerNodeNamespaceFlag.Hidden = true
-
-	err := registerNodeCmd.MarkFlagRequired("user")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
 	registerNodeCmd.Flags().StringP("key", "k", "", "Key")
-	err = registerNodeCmd.MarkFlagRequired("key")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
+	mustMarkRequired(registerNodeCmd, "user", "key")
 	nodeCmd.AddCommand(registerNodeCmd)
 
 	expireNodeCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
 	expireNodeCmd.Flags().StringP("expiry", "e", "", "Set expire to (RFC3339 format, e.g. 2025-08-27T10:00:00Z), or leave empty to expire immediately.")
-	err = expireNodeCmd.MarkFlagRequired("identifier")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
+	expireNodeCmd.Flags().BoolP("disable", "d", false, "Disable key expiry (node will never expire)")
+	mustMarkRequired(expireNodeCmd, "identifier")
 	nodeCmd.AddCommand(expireNodeCmd)
 
 	renameNodeCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
-	err = renameNodeCmd.MarkFlagRequired("identifier")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
+	mustMarkRequired(renameNodeCmd, "identifier")
 	nodeCmd.AddCommand(renameNodeCmd)
 
 	deleteNodeCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
-	err = deleteNodeCmd.MarkFlagRequired("identifier")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
+	mustMarkRequired(deleteNodeCmd, "identifier")
 	nodeCmd.AddCommand(deleteNodeCmd)
 
-	moveNodeCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
-
-	err = moveNodeCmd.MarkFlagRequired("identifier")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
-
-	moveNodeCmd.Flags().Uint64P("user", "u", 0, "New user")
-
-	moveNodeCmd.Flags().StringP("namespace", "n", "", "User")
-	moveNodeNamespaceFlag := moveNodeCmd.Flags().Lookup("namespace")
-	moveNodeNamespaceFlag.Deprecated = deprecateNamespaceMessage
-	moveNodeNamespaceFlag.Hidden = true
-
-	err = moveNodeCmd.MarkFlagRequired("user")
-	if err != nil {
-		log.Fatal(err.Error())
-	}
-	nodeCmd.AddCommand(moveNodeCmd)
-
 	tagCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
-	tagCmd.MarkFlagRequired("identifier")
+	mustMarkRequired(tagCmd, "identifier")
 	tagCmd.Flags().StringSliceP("tags", "t", []string{}, "List of tags to add to the node")
 	nodeCmd.AddCommand(tagCmd)
 
 	approveRoutesCmd.Flags().Uint64P("identifier", "i", 0, "Node identifier (ID)")
-	approveRoutesCmd.MarkFlagRequired("identifier")
+	mustMarkRequired(approveRoutesCmd, "identifier")
 	approveRoutesCmd.Flags().StringSliceP("routes", "r", []string{}, `List of routes that will be approved (comma-separated, e.g. "10.0.0.0/8,192.168.0.0/24" or empty string to remove all approved routes)`)
 	nodeCmd.AddCommand(approveRoutesCmd)
 
@@ -109,31 +60,16 @@ func init() {
 var nodeCmd = &cobra.Command{
 	Use:     "nodes",
 	Short:   "Manage the nodes of Headscale",
-	Aliases: []string{"node", "machine", "machines"},
+	Aliases: []string{"node"},
 }
 
 var registerNodeCmd = &cobra.Command{
-	Use:   "register",
-	Short: "Registers a node to your network",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		user, err := cmd.Flags().GetString("user")
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
-		}
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		registrationID, err := cmd.Flags().GetString("key")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting node key from flag: %s", err),
-				output,
-			)
-		}
+	Use:        "register",
+	Short:      "Registers a node to your network",
+	Deprecated: "use 'headscale auth register --auth-id <id> --user <user>' instead",
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		user, _ := cmd.Flags().GetString("user")
+		registrationID, _ := cmd.Flags().GetString("key")
 
 		request := &v1.RegisterNodeRequest{
 			Key:  registrationID,
@@ -142,106 +78,49 @@ var registerNodeCmd = &cobra.Command{
 
 		response, err := client.RegisterNode(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf(
-					"Cannot register node: %s\n",
-					status.Convert(err).Message(),
-				),
-				output,
-			)
+			return fmt.Errorf("registering node: %w", err)
 		}
 
-		SuccessOutput(
+		return printOutput(
+			cmd,
 			response.GetNode(),
-			fmt.Sprintf("Node %s registered", response.GetNode().GetGivenName()), output)
-	},
+			fmt.Sprintf("Node %s registered", response.GetNode().GetGivenName()))
+	}),
 }
 
 var listNodesCmd = &cobra.Command{
 	Use:     "list",
 	Short:   "List nodes",
 	Aliases: []string{"ls", "show"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		user, err := cmd.Flags().GetString("user")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		user, _ := cmd.Flags().GetString("user")
+
+		response, err := client.ListNodes(ctx, &v1.ListNodesRequest{User: user})
 		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
-		}
-		showTags, err := cmd.Flags().GetBool("tags")
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting tags flag: %s", err), output)
+			return fmt.Errorf("listing nodes: %w", err)
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
+		return printListOutput(cmd, response.GetNodes(), func() error {
+			tableData, err := nodesToPtables(user, response.GetNodes())
+			if err != nil {
+				return fmt.Errorf("converting to table: %w", err)
+			}
 
-		request := &v1.ListNodesRequest{
-			User: user,
-		}
-
-		response, err := client.ListNodes(ctx, request)
-		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot get nodes: "+status.Convert(err).Message(),
-				output,
-			)
-		}
-
-		if output != "" {
-			SuccessOutput(response.GetNodes(), "", output)
-		}
-
-		tableData, err := nodesToPtables(user, showTags, response.GetNodes())
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error converting to table: %s", err), output)
-		}
-
-		err = pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to render pterm table: %s", err),
-				output,
-			)
-		}
-	},
+			return pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
+		})
+	}),
 }
 
 var listNodeRoutesCmd = &cobra.Command{
 	Use:     "list-routes",
 	Short:   "List routes available on nodes",
 	Aliases: []string{"lsr", "routes"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		identifier, err := cmd.Flags().GetUint64("identifier")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
+
+		response, err := client.ListNodes(ctx, &v1.ListNodesRequest{})
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
-		}
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		request := &v1.ListNodesRequest{}
-
-		response, err := client.ListNodes(ctx, request)
-		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot get nodes: "+status.Convert(err).Message(),
-				output,
-			)
-		}
-
-		if output != "" {
-			SuccessOutput(response.GetNodes(), "", output)
+			return fmt.Errorf("listing nodes: %w", err)
 		}
 
 		nodes := response.GetNodes()
@@ -249,6 +128,7 @@ var listNodeRoutesCmd = &cobra.Command{
 			for _, node := range response.GetNodes() {
 				if node.GetId() == identifier {
 					nodes = []*v1.Node{node}
+
 					break
 				}
 			}
@@ -258,67 +138,53 @@ var listNodeRoutesCmd = &cobra.Command{
 			return (n.GetSubnetRoutes() != nil && len(n.GetSubnetRoutes()) > 0) || (n.GetApprovedRoutes() != nil && len(n.GetApprovedRoutes()) > 0) || (n.GetAvailableRoutes() != nil && len(n.GetAvailableRoutes()) > 0)
 		})
 
-		tableData, err := nodeRoutesToPtables(nodes)
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error converting to table: %s", err), output)
-		}
-
-		err = pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to render pterm table: %s", err),
-				output,
-			)
-		}
-	},
+		return printListOutput(cmd, nodes, func() error {
+			return pterm.DefaultTable.WithHasHeader().WithData(nodeRoutesToPtables(nodes)).Render()
+		})
+	}),
 }
 
 var expireNodeCmd = &cobra.Command{
-	Use:     "expire",
-	Short:   "Expire (log out) a node in your network",
-	Long:    "Expiring a node will keep the node in the database and force it to reauthenticate.",
+	Use:   "expire",
+	Short: "Expire (log out) a node in your network",
+	Long: `Expiring a node will keep the node in the database and force it to reauthenticate.
+
+Use --disable to disable key expiry (node will never expire).`,
 	Aliases: []string{"logout", "exp", "e"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
+		disableExpiry, _ := cmd.Flags().GetBool("disable")
 
-		identifier, err := cmd.Flags().GetUint64("identifier")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
+		// Handle disable expiry - node will never expire.
+		if disableExpiry {
+			request := &v1.ExpireNodeRequest{
+				NodeId:        identifier,
+				DisableExpiry: true,
+			}
+
+			response, err := client.ExpireNode(ctx, request)
+			if err != nil {
+				return fmt.Errorf("disabling node expiry: %w", err)
+			}
+
+			return printOutput(cmd, response.GetNode(), "Node expiry disabled")
 		}
 
-		expiry, err := cmd.Flags().GetString("expiry")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting expiry to string: %s", err),
-				output,
-			)
+		expiry, _ := cmd.Flags().GetString("expiry")
+
+		now := time.Now()
+
+		expiryTime := now
 
-			return
-		}
-		expiryTime := time.Now()
 		if expiry != "" {
+			var err error
+
 			expiryTime, err = time.Parse(time.RFC3339, expiry)
 			if err != nil {
-				ErrorOutput(
-					err,
-					fmt.Sprintf("Error converting expiry to string: %s", err),
-					output,
-				)
-
-				return
+				return fmt.Errorf("parsing expiry time: %w", err)
 			}
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
 		request := &v1.ExpireNodeRequest{
 			NodeId: identifier,
 			Expiry: timestamppb.New(expiryTime),
@@ -326,43 +192,28 @@ var expireNodeCmd = &cobra.Command{
 
 		response, err := client.ExpireNode(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf(
-					"Cannot expire node: %s\n",
-					status.Convert(err).Message(),
-				),
-				output,
-			)
+			return fmt.Errorf("expiring node: %w", err)
 		}
 
-		SuccessOutput(response.GetNode(), "Node expired", output)
-	},
+		if now.Equal(expiryTime) || now.After(expiryTime) {
+			return printOutput(cmd, response.GetNode(), "Node expired")
+		}
+
+		return printOutput(cmd, response.GetNode(), "Node expiration updated")
+	}),
 }
 
 var renameNodeCmd = &cobra.Command{
 	Use:   "rename NEW_NAME",
 	Short: "Renames a node in your network",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		identifier, err := cmd.Flags().GetUint64("identifier")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
-		}
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
 
 		newName := ""
 		if len(args) > 0 {
 			newName = args[0]
 		}
+
 		request := &v1.RenameNodeRequest{
 			NodeId:  identifier,
 			NewName: newName,
@@ -370,39 +221,19 @@ var renameNodeCmd = &cobra.Command{
 
 		response, err := client.RenameNode(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf(
-					"Cannot rename node: %s\n",
-					status.Convert(err).Message(),
-				),
-				output,
-			)
+			return fmt.Errorf("renaming node: %w", err)
 		}
 
-		SuccessOutput(response.GetNode(), "Node renamed", output)
-	},
+		return printOutput(cmd, response.GetNode(), "Node renamed")
+	}),
 }
 
 var deleteNodeCmd = &cobra.Command{
 	Use:     "delete",
 	Short:   "Delete a node",
 	Aliases: []string{"del"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		identifier, err := cmd.Flags().GetUint64("identifier")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
-		}
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
 
 		getRequest := &v1.GetNodeRequest{
 			NodeId: identifier,
@@ -410,109 +241,31 @@ var deleteNodeCmd = &cobra.Command{
 
 		getResponse, err := client.GetNode(ctx, getRequest)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Error getting node node: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("getting node: %w", err)
 		}
 
 		deleteRequest := &v1.DeleteNodeRequest{
 			NodeId: identifier,
 		}
 
-		confirm := false
-		force, _ := cmd.Flags().GetBool("force")
-		if !force {
-			confirm = util.YesNo(fmt.Sprintf(
-				"Do you want to remove the node %s?",
-				getResponse.GetNode().GetName(),
-			))
+		if !confirmAction(cmd, fmt.Sprintf(
+			"Do you want to remove the node %s?",
+			getResponse.GetNode().GetName(),
+		)) {
+			return printOutput(cmd, map[string]string{"Result": "Node not deleted"}, "Node not deleted")
 		}
 
-		if confirm || force {
-			response, err := client.DeleteNode(ctx, deleteRequest)
-			if output != "" {
-				SuccessOutput(response, "", output)
-
-				return
-			}
-			if err != nil {
-				ErrorOutput(
-					err,
-					"Error deleting node: "+status.Convert(err).Message(),
-					output,
-				)
-			}
-			SuccessOutput(
-				map[string]string{"Result": "Node deleted"},
-				"Node deleted",
-				output,
-			)
-		} else {
-			SuccessOutput(map[string]string{"Result": "Node not deleted"}, "Node not deleted", output)
-		}
-	},
-}
-
-var moveNodeCmd = &cobra.Command{
-	Use:     "move",
-	Short:   "Move node to another user",
-	Aliases: []string{"mv"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		identifier, err := cmd.Flags().GetUint64("identifier")
+		_, err = client.DeleteNode(ctx, deleteRequest)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
+			return fmt.Errorf("deleting node: %w", err)
 		}
 
-		user, err := cmd.Flags().GetUint64("user")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting user: %s", err),
-				output,
-			)
-		}
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		getRequest := &v1.GetNodeRequest{
-			NodeId: identifier,
-		}
-
-		_, err = client.GetNode(ctx, getRequest)
-		if err != nil {
-			ErrorOutput(
-				err,
-				"Error getting node: "+status.Convert(err).Message(),
-				output,
-			)
-		}
-
-		moveRequest := &v1.MoveNodeRequest{
-			NodeId: identifier,
-			User:   user,
-		}
-
-		moveResponse, err := client.MoveNode(ctx, moveRequest)
-		if err != nil {
-			ErrorOutput(
-				err,
-				"Error moving node: "+status.Convert(err).Message(),
-				output,
-			)
-		}
-
-		SuccessOutput(moveResponse.GetNode(), "Node moved to another user", output)
-	},
+		return printOutput(
+			cmd,
+			map[string]string{"Result": "Node deleted"},
+			"Node deleted",
+		)
+	}),
 }
 
 var backfillNodeIPsCmd = &cobra.Command{
@@ -530,38 +283,29 @@ all nodes that are missing.
 If you remove IPv4 or IPv6 prefixes from the config,
 it can be run to remove the IPs that should no longer
 be assigned to nodes.`,
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		confirm := false
-
-		force, _ := cmd.Flags().GetBool("force")
-		if !force {
-			confirm = util.YesNo("Are you sure that you want to assign/remove IPs to/from nodes?")
+	RunE: func(cmd *cobra.Command, args []string) error {
+		if !confirmAction(cmd, "Are you sure that you want to assign/remove IPs to/from nodes?") {
+			return nil
 		}
 
-		if confirm || force {
-			ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-			defer cancel()
-			defer conn.Close()
-
-			changes, err := client.BackfillNodeIPs(ctx, &v1.BackfillNodeIPsRequest{Confirmed: confirm || force})
-			if err != nil {
-				ErrorOutput(
-					err,
-					"Error backfilling IPs: "+status.Convert(err).Message(),
-					output,
-				)
-			}
-
-			SuccessOutput(changes, "Node IPs backfilled successfully", output)
+		ctx, client, conn, cancel, err := newHeadscaleCLIWithConfig()
+		if err != nil {
+			return fmt.Errorf("connecting to headscale: %w", err)
 		}
+		defer cancel()
+		defer conn.Close()
+
+		changes, err := client.BackfillNodeIPs(ctx, &v1.BackfillNodeIPsRequest{Confirmed: true})
+		if err != nil {
+			return fmt.Errorf("backfilling IPs: %w", err)
+		}
+
+		return printOutput(cmd, changes, "Node IPs backfilled successfully")
 	},
 }
 
 func nodesToPtables(
 	currentUser string,
-	showTags bool,
 	nodes []*v1.Node,
 ) (pterm.TableData, error) {
 	tableHeader := []string{
@@ -571,6 +315,7 @@ func nodesToPtables(
 		"MachineKey",
 		"NodeKey",
 		"User",
+		"Tags",
 		"IP addresses",
 		"Ephemeral",
 		"Last seen",
@@ -578,13 +323,6 @@ func nodesToPtables(
 		"Connected",
 		"Expired",
 	}
-	if showTags {
-		tableHeader = append(tableHeader, []string{
-			"ForcedTags",
-			"InvalidTags",
-			"ValidTags",
-		}...)
-	}
 	tableData := pterm.TableData{tableHeader}
 
 	for _, node := range nodes {
@@ -593,23 +331,30 @@ func nodesToPtables(
 			ephemeral = true
 		}
 
-		var lastSeen time.Time
-		var lastSeenTime string
+		var (
+			lastSeen     time.Time
+			lastSeenTime string
+		)
+
 		if node.GetLastSeen() != nil {
 			lastSeen = node.GetLastSeen().AsTime()
-			lastSeenTime = lastSeen.Format("2006-01-02 15:04:05")
+			lastSeenTime = lastSeen.Format(HeadscaleDateTimeFormat)
 		}
 
-		var expiry time.Time
-		var expiryTime string
+		var (
+			expiry     time.Time
+			expiryTime string
+		)
+
 		if node.GetExpiry() != nil {
 			expiry = node.GetExpiry().AsTime()
-			expiryTime = expiry.Format("2006-01-02 15:04:05")
+			expiryTime = expiry.Format(HeadscaleDateTimeFormat)
 		} else {
 			expiryTime = "N/A"
 		}
 
 		var machineKey key.MachinePublic
+
 		err := machineKey.UnmarshalText(
 			[]byte(node.GetMachineKey()),
 		)
@@ -618,6 +363,7 @@ func nodesToPtables(
 		}
 
 		var nodeKey key.NodePublic
+
 		err = nodeKey.UnmarshalText(
 			[]byte(node.GetNodeKey()),
 		)
@@ -633,50 +379,40 @@ func nodesToPtables(
 		}
 
 		var expired string
-		if expiry.IsZero() || expiry.After(time.Now()) {
-			expired = pterm.LightGreen("no")
-		} else {
+		if node.GetExpiry() != nil && node.GetExpiry().AsTime().Before(time.Now()) {
 			expired = pterm.LightRed("yes")
+		} else {
+			expired = pterm.LightGreen("no")
 		}
 
-		var forcedTags string
-		for _, tag := range node.GetForcedTags() {
-			forcedTags += "," + tag
+		var tagsBuilder strings.Builder
+
+		for _, tag := range node.GetTags() {
+			tagsBuilder.WriteString("\n" + tag)
 		}
-		forcedTags = strings.TrimLeft(forcedTags, ",")
-		var invalidTags string
-		for _, tag := range node.GetInvalidTags() {
-			if !slices.Contains(node.GetForcedTags(), tag) {
-				invalidTags += "," + pterm.LightRed(tag)
-			}
-		}
-		invalidTags = strings.TrimLeft(invalidTags, ",")
-		var validTags string
-		for _, tag := range node.GetValidTags() {
-			if !slices.Contains(node.GetForcedTags(), tag) {
-				validTags += "," + pterm.LightGreen(tag)
-			}
-		}
-		validTags = strings.TrimLeft(validTags, ",")
+
+		tags := strings.TrimLeft(tagsBuilder.String(), "\n")
 
 		var user string
-		if currentUser == "" || (currentUser == node.GetUser().GetName()) {
-			user = pterm.LightMagenta(node.GetUser().GetName())
-		} else {
-			// Shared into this user
-			user = pterm.LightYellow(node.GetUser().GetName())
+		if node.GetUser() != nil {
+			user = node.GetUser().GetName()
 		}
 
-		var IPV4Address string
-		var IPV6Address string
+		var ipBuilder strings.Builder
+
 		for _, addr := range node.GetIpAddresses() {
-			if netip.MustParseAddr(addr).Is4() {
-				IPV4Address = addr
-			} else {
-				IPV6Address = addr
+			ip, err := netip.ParseAddr(addr)
+			if err == nil {
+				if ipBuilder.Len() > 0 {
+					ipBuilder.WriteString("\n")
+				}
+
+				ipBuilder.WriteString(ip.String())
 			}
 		}
 
+		ipAddresses := ipBuilder.String()
+
 		nodeData := []string{
 			strconv.FormatUint(node.GetId(), util.Base10),
 			node.GetName(),
@@ -684,16 +420,14 @@ func nodesToPtables(
 			machineKey.ShortString(),
 			nodeKey.ShortString(),
 			user,
-			strings.Join([]string{IPV4Address, IPV6Address}, ", "),
+			tags,
+			ipAddresses,
 			strconv.FormatBool(ephemeral),
 			lastSeenTime,
 			expiryTime,
 			online,
 			expired,
 		}
-		if showTags {
-			nodeData = append(nodeData, []string{forcedTags, invalidTags, validTags}...)
-		}
 		tableData = append(
 			tableData,
 			nodeData,
@@ -705,7 +439,7 @@ func nodesToPtables(
 
 func nodeRoutesToPtables(
 	nodes []*v1.Node,
-) (pterm.TableData, error) {
+) pterm.TableData {
 	tableHeader := []string{
 		"ID",
 		"Hostname",
@@ -719,9 +453,9 @@ func nodeRoutesToPtables(
 		nodeData := []string{
 			strconv.FormatUint(node.GetId(), util.Base10),
 			node.GetGivenName(),
-			strings.Join(node.GetApprovedRoutes(), ", "),
-			strings.Join(node.GetAvailableRoutes(), ", "),
-			strings.Join(node.GetSubnetRoutes(), ", "),
+			strings.Join(node.GetApprovedRoutes(), "\n"),
+			strings.Join(node.GetAvailableRoutes(), "\n"),
+			strings.Join(node.GetSubnetRoutes(), "\n"),
 		}
 		tableData = append(
 			tableData,
@@ -729,108 +463,50 @@ func nodeRoutesToPtables(
 		)
 	}
 
-	return tableData, nil
+	return tableData
 }
 
 var tagCmd = &cobra.Command{
 	Use:     "tag",
 	Short:   "Manage the tags of a node",
 	Aliases: []string{"tags", "t"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		// retrieve flags from CLI
-		identifier, err := cmd.Flags().GetUint64("identifier")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
-		}
-		tagsToSet, err := cmd.Flags().GetStringSlice("tags")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error retrieving list of tags to add to node, %v", err),
-				output,
-			)
-		}
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
+		tagsToSet, _ := cmd.Flags().GetStringSlice("tags")
 
 		// Sending tags to node
 		request := &v1.SetTagsRequest{
 			NodeId: identifier,
 			Tags:   tagsToSet,
 		}
+
 		resp, err := client.SetTags(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error while sending tags to headscale: %s", err),
-				output,
-			)
+			return fmt.Errorf("setting tags: %w", err)
 		}
 
-		if resp != nil {
-			SuccessOutput(
-				resp.GetNode(),
-				"Node updated",
-				output,
-			)
-		}
-	},
+		return printOutput(cmd, resp.GetNode(), "Node updated")
+	}),
 }
 
 var approveRoutesCmd = &cobra.Command{
 	Use:   "approve-routes",
 	Short: "Manage the approved routes of a node",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		// retrieve flags from CLI
-		identifier, err := cmd.Flags().GetUint64("identifier")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error converting ID to integer: %s", err),
-				output,
-			)
-		}
-		routes, err := cmd.Flags().GetStringSlice("routes")
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error retrieving list of routes to add to node, %v", err),
-				output,
-			)
-		}
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		identifier, _ := cmd.Flags().GetUint64("identifier")
+		routes, _ := cmd.Flags().GetStringSlice("routes")
 
 		// Sending routes to node
 		request := &v1.SetApprovedRoutesRequest{
 			NodeId: identifier,
 			Routes: routes,
 		}
+
 		resp, err := client.SetApprovedRoutes(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error while sending routes to headscale: %s", err),
-				output,
-			)
+			return fmt.Errorf("setting approved routes: %w", err)
 		}
 
-		if resp != nil {
-			SuccessOutput(
-				resp.GetNode(),
-				"Node updated",
-				output,
-			)
-		}
-	},
+		return printOutput(cmd, resp.GetNode(), "Node updated")
+	}),
 }

cmd/headscale/cli/policy.go

@@ -1,24 +1,41 @@
 package cli
 
 import (
+	"errors"
 	"fmt"
-	"io"
 	"os"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/juanfont/headscale/hscontrol/db"
 	"github.com/juanfont/headscale/hscontrol/policy"
 	"github.com/juanfont/headscale/hscontrol/types"
-	"github.com/juanfont/headscale/hscontrol/util"
-	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
 	"tailscale.com/types/views"
 )
 
 const (
-	bypassFlag = "bypass-grpc-and-access-database-directly"
+	bypassFlag = "bypass-grpc-and-access-database-directly" //nolint:gosec // not a credential
 )
 
+var errAborted = errors.New("command aborted by user")
+
+// bypassDatabase loads the server config and opens the database directly,
+// bypassing the gRPC server. The caller is responsible for closing the
+// returned database handle.
+func bypassDatabase() (*db.HSDatabase, error) {
+	cfg, err := types.LoadServerConfig()
+	if err != nil {
+		return nil, fmt.Errorf("loading config: %w", err)
+	}
+
+	d, err := db.NewHeadscaleDatabase(cfg)
+	if err != nil {
+		return nil, fmt.Errorf("opening database: %w", err)
+	}
+
+	return d, nil
+}
+
 func init() {
 	rootCmd.AddCommand(policyCmd)
 
@@ -26,16 +43,13 @@ func init() {
 	policyCmd.AddCommand(getPolicy)
 
 	setPolicy.Flags().StringP("file", "f", "", "Path to a policy file in HuJSON format")
-	if err := setPolicy.MarkFlagRequired("file"); err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
 	setPolicy.Flags().BoolP(bypassFlag, "", false, "Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
+	mustMarkRequired(setPolicy, "file")
 	policyCmd.AddCommand(setPolicy)
 
 	checkPolicy.Flags().StringP("file", "f", "", "Path to a policy file in HuJSON format")
-	if err := checkPolicy.MarkFlagRequired("file"); err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
+	checkPolicy.Flags().BoolP(bypassFlag, "", false, "Open the database directly (no gRPC, no running server) to validate user@ token references and to evaluate the policy's tests block. Required when those checks are needed.")
+	mustMarkRequired(checkPolicy, "file")
 	policyCmd.AddCommand(checkPolicy)
 }
 
@@ -48,60 +62,47 @@ var getPolicy = &cobra.Command{
 	Use:     "get",
 	Short:   "Print the current ACL Policy",
 	Aliases: []string{"show", "view", "fetch"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		var policy string
+	RunE: func(cmd *cobra.Command, args []string) error {
+		var policyData string
+
 		if bypass, _ := cmd.Flags().GetBool(bypassFlag); bypass {
-			confirm := false
-			force, _ := cmd.Flags().GetBool("force")
-			if !force {
-				confirm = util.YesNo("DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?")
+			if !confirmAction(cmd, "DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?") {
+				return errAborted
 			}
 
-			if !confirm && !force {
-				ErrorOutput(nil, "Aborting command", output)
-				return
-			}
-
-			cfg, err := types.LoadServerConfig()
+			d, err := bypassDatabase()
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed loading config: %s", err), output)
-			}
-
-			d, err := db.NewHeadscaleDatabase(
-				cfg.Database,
-				cfg.BaseDomain,
-				nil,
-			)
-			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed to open database: %s", err), output)
+				return err
 			}
+			defer d.Close()
 
 			pol, err := d.GetPolicy()
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed loading Policy from database: %s", err), output)
+				return fmt.Errorf("loading policy from database: %w", err)
 			}
 
-			policy = pol.Data
+			policyData = pol.Data
 		} else {
-			ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
+			ctx, client, conn, cancel, err := newHeadscaleCLIWithConfig()
+			if err != nil {
+				return fmt.Errorf("connecting to headscale: %w", err)
+			}
 			defer cancel()
 			defer conn.Close()
 
-			request := &v1.GetPolicyRequest{}
-
-			response, err := client.GetPolicy(ctx, request)
+			response, err := client.GetPolicy(ctx, &v1.GetPolicyRequest{})
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed loading ACL Policy: %s", err), output)
+				return fmt.Errorf("loading ACL policy: %w", err)
 			}
 
-			policy = response.GetPolicy()
+			policyData = response.GetPolicy()
 		}
 
-		// TODO(pallabpain): Maybe print this better?
-		// This does not pass output as we dont support yaml, json or json-line
-		// output for this command. It is HuJSON already.
-		SuccessOutput("", policy, "")
+		// This does not pass output format as we don't support yaml, json or
+		// json-line output for this command. It is HuJSON already.
+		fmt.Println(policyData)
+
+		return nil
 	},
 }
 
@@ -112,101 +113,131 @@ var setPolicy = &cobra.Command{
 	Updates the existing ACL Policy with the provided policy. The policy must be a valid HuJSON object.
 	This command only works when the acl.policy_mode is set to "db", and the policy will be stored in the database.`,
 	Aliases: []string{"put", "update"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	RunE: func(cmd *cobra.Command, args []string) error {
 		policyPath, _ := cmd.Flags().GetString("file")
 
-		f, err := os.Open(policyPath)
+		policyBytes, err := os.ReadFile(policyPath)
 		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error opening the policy file: %s", err), output)
-		}
-		defer f.Close()
-
-		policyBytes, err := io.ReadAll(f)
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error reading the policy file: %s", err), output)
+			return fmt.Errorf("reading policy file: %w", err)
 		}
 
 		if bypass, _ := cmd.Flags().GetBool(bypassFlag); bypass {
-			confirm := false
-			force, _ := cmd.Flags().GetBool("force")
-			if !force {
-				confirm = util.YesNo("DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?")
+			if !confirmAction(cmd, "DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?") {
+				return errAborted
 			}
 
-			if !confirm && !force {
-				ErrorOutput(nil, "Aborting command", output)
-				return
-			}
-
-			cfg, err := types.LoadServerConfig()
+			d, err := bypassDatabase()
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed loading config: %s", err), output)
-			}
-
-			d, err := db.NewHeadscaleDatabase(
-				cfg.Database,
-				cfg.BaseDomain,
-				nil,
-			)
-			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed to open database: %s", err), output)
+				return err
 			}
+			defer d.Close()
 
 			users, err := d.ListUsers()
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed to load users for policy validation: %s", err), output)
+				return fmt.Errorf("loading users for policy validation: %w", err)
 			}
 
 			_, err = policy.NewPolicyManager(policyBytes, users, views.Slice[types.NodeView]{})
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Error parsing the policy file: %s", err), output)
-				return
+				return fmt.Errorf("parsing policy file: %w", err)
 			}
 
 			_, err = d.SetPolicy(string(policyBytes))
 			if err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed to set ACL Policy: %s", err), output)
+				return fmt.Errorf("setting ACL policy: %w", err)
 			}
 		} else {
 			request := &v1.SetPolicyRequest{Policy: string(policyBytes)}
 
-			ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
+			ctx, client, conn, cancel, err := newHeadscaleCLIWithConfig()
+			if err != nil {
+				return fmt.Errorf("connecting to headscale: %w", err)
+			}
 			defer cancel()
 			defer conn.Close()
 
-			if _, err := client.SetPolicy(ctx, request); err != nil {
-				ErrorOutput(err, fmt.Sprintf("Failed to set ACL Policy: %s", err), output)
+			_, err = client.SetPolicy(ctx, request)
+			if err != nil {
+				return fmt.Errorf("setting ACL policy: %w", err)
 			}
 		}
 
-		SuccessOutput(nil, "Policy updated.", "")
+		fmt.Println("Policy updated.")
+
+		return nil
 	},
 }
 
 var checkPolicy = &cobra.Command{
 	Use:   "check",
 	Short: "Check the Policy file for errors",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	Long: `
+	Check validates the policy against the server's live users and nodes,
+	running any "tests" block. By default the command is a thin frontend
+	for a gRPC call to a running headscale; pass --` + bypassFlag + ` to
+	open the database directly when headscale is not running.`,
+	RunE: func(cmd *cobra.Command, args []string) error {
 		policyPath, _ := cmd.Flags().GetString("file")
 
-		f, err := os.Open(policyPath)
+		policyBytes, err := os.ReadFile(policyPath)
 		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error opening the policy file: %s", err), output)
-		}
-		defer f.Close()
-
-		policyBytes, err := io.ReadAll(f)
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error reading the policy file: %s", err), output)
+			return fmt.Errorf("reading policy file: %w", err)
 		}
 
-		_, err = policy.NewPolicyManager(policyBytes, nil, views.Slice[types.NodeView]{})
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error parsing the policy file: %s", err), output)
+		if bypass, _ := cmd.Flags().GetBool(bypassFlag); bypass {
+			if !confirmAction(cmd, "DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?") {
+				return errAborted
+			}
+
+			d, err := bypassDatabase()
+			if err != nil {
+				return err
+			}
+			defer d.Close()
+
+			users, err := d.ListUsers()
+			if err != nil {
+				return fmt.Errorf("loading users: %w", err)
+			}
+
+			nodes, err := d.ListNodes()
+			if err != nil {
+				return fmt.Errorf("loading nodes: %w", err)
+			}
+
+			// NewPolicyManager validates structure and user references
+			// but intentionally skips test evaluation (boot path).
+			// SetPolicy is the user-write boundary and is what runs the
+			// tests block.
+			pm, err := policy.NewPolicyManager(policyBytes, users, nodes.ViewSlice())
+			if err != nil {
+				return fmt.Errorf("parsing policy file: %w", err)
+			}
+
+			_, err = pm.SetPolicy(policyBytes)
+			if err != nil {
+				return err
+			}
+
+			fmt.Println("Policy is valid")
+
+			return nil
 		}
 
-		SuccessOutput(nil, "Policy is valid", "")
+		ctx, client, conn, cancel, err := newHeadscaleCLIWithConfig()
+		if err != nil {
+			return fmt.Errorf("connecting to headscale: %w", err)
+		}
+		defer cancel()
+		defer conn.Close()
+
+		_, err = client.CheckPolicy(ctx, &v1.CheckPolicyRequest{Policy: string(policyBytes)})
+		if err != nil {
+			return err
+		}
+
+		fmt.Println("Policy is valid")
+
+		return nil
 	},
 }

cmd/headscale/cli/preauthkeys.go

@@ -1,17 +1,15 @@
 package cli
 
 import (
+	"context"
 	"fmt"
 	"strconv"
 	"strings"
-	"time"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
-	"github.com/prometheus/common/model"
+	"github.com/juanfont/headscale/hscontrol/util"
 	"github.com/pterm/pterm"
-	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
-	"google.golang.org/protobuf/types/known/timestamppb"
 )
 
 const (
@@ -20,20 +18,10 @@ const (
 
 func init() {
 	rootCmd.AddCommand(preauthkeysCmd)
-	preauthkeysCmd.PersistentFlags().Uint64P("user", "u", 0, "User identifier (ID)")
-
-	preauthkeysCmd.PersistentFlags().StringP("namespace", "n", "", "User")
-	pakNamespaceFlag := preauthkeysCmd.PersistentFlags().Lookup("namespace")
-	pakNamespaceFlag.Deprecated = deprecateNamespaceMessage
-	pakNamespaceFlag.Hidden = true
-
-	err := preauthkeysCmd.MarkPersistentFlagRequired("user")
-	if err != nil {
-		log.Fatal().Err(err).Msg("")
-	}
 	preauthkeysCmd.AddCommand(listPreAuthKeys)
 	preauthkeysCmd.AddCommand(createPreAuthKeyCmd)
 	preauthkeysCmd.AddCommand(expirePreAuthKeyCmd)
+	preauthkeysCmd.AddCommand(deletePreAuthKeyCmd)
 	createPreAuthKeyCmd.PersistentFlags().
 		Bool("reusable", false, "Make the preauthkey reusable")
 	createPreAuthKeyCmd.PersistentFlags().
@@ -42,6 +30,9 @@ func init() {
 		StringP("expiration", "e", DefaultPreAuthKeyExpiry, "Human-readable expiration of the key (e.g. 30m, 24h)")
 	createPreAuthKeyCmd.Flags().
 		StringSlice("tags", []string{}, "Tags to automatically assign to node")
+	createPreAuthKeyCmd.PersistentFlags().Uint64P("user", "u", 0, "User identifier (ID)")
+	expirePreAuthKeyCmd.PersistentFlags().Uint64P("id", "i", 0, "Authkey ID")
+	deletePreAuthKeyCmd.PersistentFlags().Uint64P("id", "i", 0, "Authkey ID")
 }
 
 var preauthkeysCmd = &cobra.Command{
@@ -52,183 +43,136 @@ var preauthkeysCmd = &cobra.Command{
 
 var listPreAuthKeys = &cobra.Command{
 	Use:     "list",
-	Short:   "List the preauthkeys for this user",
+	Short:   "List all preauthkeys",
 	Aliases: []string{"ls", "show"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		user, err := cmd.Flags().GetUint64("user")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		response, err := client.ListPreAuthKeys(ctx, &v1.ListPreAuthKeysRequest{})
 		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
+			return fmt.Errorf("listing preauthkeys: %w", err)
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		request := &v1.ListPreAuthKeysRequest{
-			User: user,
-		}
-
-		response, err := client.ListPreAuthKeys(ctx, request)
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Error getting the list of keys: %s", err),
-				output,
-			)
-
-			return
-		}
-
-		if output != "" {
-			SuccessOutput(response.GetPreAuthKeys(), "", output)
-		}
-
-		tableData := pterm.TableData{
-			{
-				"ID",
-				"Key",
-				"Reusable",
-				"Ephemeral",
-				"Used",
-				"Expiration",
-				"Created",
-				"Tags",
-			},
-		}
-		for _, key := range response.GetPreAuthKeys() {
-			expiration := "-"
-			if key.GetExpiration() != nil {
-				expiration = ColourTime(key.GetExpiration().AsTime())
+		return printListOutput(cmd, response.GetPreAuthKeys(), func() error {
+			tableData := pterm.TableData{
+				{
+					"ID",
+					"Key/Prefix",
+					"Reusable",
+					"Ephemeral",
+					"Used",
+					"Expiration",
+					"Created",
+					"Owner",
+				},
 			}
 
-			aclTags := ""
+			for _, key := range response.GetPreAuthKeys() {
+				expiration := "-"
+				if key.GetExpiration() != nil {
+					expiration = ColourTime(key.GetExpiration().AsTime())
+				}
 
-			for _, tag := range key.GetAclTags() {
-				aclTags += "," + tag
+				var owner string
+				if len(key.GetAclTags()) > 0 {
+					owner = strings.Join(key.GetAclTags(), "\n")
+				} else if key.GetUser() != nil {
+					owner = key.GetUser().GetName()
+				} else {
+					owner = "-"
+				}
+
+				tableData = append(tableData, []string{
+					strconv.FormatUint(key.GetId(), util.Base10),
+					key.GetKey(),
+					strconv.FormatBool(key.GetReusable()),
+					strconv.FormatBool(key.GetEphemeral()),
+					strconv.FormatBool(key.GetUsed()),
+					expiration,
+					key.GetCreatedAt().AsTime().Format(HeadscaleDateTimeFormat),
+					owner,
+				})
 			}
 
-			aclTags = strings.TrimLeft(aclTags, ",")
-
-			tableData = append(tableData, []string{
-				strconv.FormatUint(key.GetId(), 10),
-				key.GetKey(),
-				strconv.FormatBool(key.GetReusable()),
-				strconv.FormatBool(key.GetEphemeral()),
-				strconv.FormatBool(key.GetUsed()),
-				expiration,
-				key.GetCreatedAt().AsTime().Format("2006-01-02 15:04:05"),
-				aclTags,
-			})
-
-		}
-		err = pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to render pterm table: %s", err),
-				output,
-			)
-		}
-	},
+			return pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
+		})
+	}),
 }
 
 var createPreAuthKeyCmd = &cobra.Command{
 	Use:     "create",
-	Short:   "Creates a new preauthkey in the specified user",
+	Short:   "Creates a new preauthkey",
 	Aliases: []string{"c", "new"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		user, err := cmd.Flags().GetUint64("user")
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
-		}
-
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		user, _ := cmd.Flags().GetUint64("user")
 		reusable, _ := cmd.Flags().GetBool("reusable")
 		ephemeral, _ := cmd.Flags().GetBool("ephemeral")
 		tags, _ := cmd.Flags().GetStringSlice("tags")
 
-		request := &v1.CreatePreAuthKeyRequest{
-			User:      user,
-			Reusable:  reusable,
-			Ephemeral: ephemeral,
-			AclTags:   tags,
-		}
-
-		durationStr, _ := cmd.Flags().GetString("expiration")
-
-		duration, err := model.ParseDuration(durationStr)
+		expiration, err := expirationFromFlag(cmd)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Could not parse duration: %s\n", err),
-				output,
-			)
+			return err
 		}
 
-		expiration := time.Now().UTC().Add(time.Duration(duration))
-
-		log.Trace().
-			Dur("expiration", time.Duration(duration)).
-			Msg("expiration has been set")
-
-		request.Expiration = timestamppb.New(expiration)
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
+		request := &v1.CreatePreAuthKeyRequest{
+			User:       user,
+			Reusable:   reusable,
+			Ephemeral:  ephemeral,
+			AclTags:    tags,
+			Expiration: expiration,
+		}
 
 		response, err := client.CreatePreAuthKey(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Cannot create Pre Auth Key: %s\n", err),
-				output,
-			)
+			return fmt.Errorf("creating preauthkey: %w", err)
 		}
 
-		SuccessOutput(response.GetPreAuthKey(), response.GetPreAuthKey().GetKey(), output)
-	},
+		return printOutput(cmd, response.GetPreAuthKey(), response.GetPreAuthKey().GetKey())
+	}),
 }
 
 var expirePreAuthKeyCmd = &cobra.Command{
-	Use:     "expire KEY",
+	Use:     "expire",
 	Short:   "Expire a preauthkey",
 	Aliases: []string{"revoke", "exp", "e"},
-	Args: func(cmd *cobra.Command, args []string) error {
-		if len(args) < 1 {
-			return errMissingParameter
-		}
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, _ := cmd.Flags().GetUint64("id")
 
-		return nil
-	},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-		user, err := cmd.Flags().GetUint64("user")
-		if err != nil {
-			ErrorOutput(err, fmt.Sprintf("Error getting user: %s", err), output)
+		if id == 0 {
+			return fmt.Errorf("missing --id parameter: %w", errMissingParameter)
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
 		request := &v1.ExpirePreAuthKeyRequest{
-			User: user,
-			Key:  args[0],
+			Id: id,
 		}
 
 		response, err := client.ExpirePreAuthKey(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Cannot expire Pre Auth Key: %s\n", err),
-				output,
-			)
+			return fmt.Errorf("expiring preauthkey: %w", err)
 		}
 
-		SuccessOutput(response, "Key expired", output)
-	},
+		return printOutput(cmd, response, "Key expired")
+	}),
+}
+
+var deletePreAuthKeyCmd = &cobra.Command{
+	Use:     "delete",
+	Short:   "Delete a preauthkey",
+	Aliases: []string{"del", "rm", "d"},
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, _ := cmd.Flags().GetUint64("id")
+
+		if id == 0 {
+			return fmt.Errorf("missing --id parameter: %w", errMissingParameter)
+		}
+
+		request := &v1.DeletePreAuthKeyRequest{
+			Id: id,
+		}
+
+		response, err := client.DeletePreAuthKey(ctx, request)
+		if err != nil {
+			return fmt.Errorf("deleting preauthkey: %w", err)
+		}
+
+		return printOutput(cmd, response, "Key deleted")
+	}),
 }

cmd/headscale/cli/pterm_style.go

@@ -7,7 +7,7 @@ import (
 )
 
 func ColourTime(date time.Time) string {
-	dateStr := date.Format("2006-01-02 15:04:05")
+	dateStr := date.Format(HeadscaleDateTimeFormat)
 
 	if date.After(time.Now()) {
 		dateStr = pterm.LightGreen(dateStr)

cmd/headscale/cli/root.go

@@ -1,10 +1,8 @@
 package cli
 
 import (
-	"fmt"
 	"os"
 	"runtime"
-	"slices"
 	"strings"
 
 	"github.com/juanfont/headscale/hscontrol/types"
@@ -15,10 +13,6 @@ import (
 	"github.com/tcnksm/go-latest"
 )
 
-const (
-	deprecateNamespaceMessage = "use --user"
-)
-
 var cfgFile string = ""
 
 func init() {
@@ -27,11 +21,6 @@ func init() {
 		return
 	}
 
-	if slices.Contains(os.Args, "policy") && slices.Contains(os.Args, "check") {
-		zerolog.SetGlobalLevel(zerolog.Disabled)
-		return
-	}
-
 	cobra.OnInitialize(initConfig)
 	rootCmd.PersistentFlags().
 		StringVarP(&cfgFile, "config", "c", "", "config file (default is /etc/headscale/config.yaml)")
@@ -39,25 +28,34 @@ func init() {
 		StringP("output", "o", "", "Output format. Empty for human-readable, 'json', 'json-line' or 'yaml'")
 	rootCmd.PersistentFlags().
 		Bool("force", false, "Disable prompts and forces the execution")
+
+	// Re-enable usage output only for flag-parsing errors; runtime errors
+	// from RunE should never dump usage text.
+	rootCmd.SetFlagErrorFunc(func(cmd *cobra.Command, err error) error {
+		cmd.SilenceUsage = false
+
+		return err
+	})
 }
 
 func initConfig() {
 	if cfgFile == "" {
 		cfgFile = os.Getenv("HEADSCALE_CONFIG")
 	}
+
 	if cfgFile != "" {
 		err := types.LoadConfig(cfgFile, true)
 		if err != nil {
-			log.Fatal().Caller().Err(err).Msgf("Error loading config file %s", cfgFile)
+			log.Fatal().Caller().Err(err).Msgf("error loading config file %s", cfgFile)
 		}
 	} else {
 		err := types.LoadConfig("", false)
 		if err != nil {
-			log.Fatal().Caller().Err(err).Msgf("Error loading config")
+			log.Fatal().Caller().Err(err).Msgf("error loading config")
 		}
 	}
 
-	machineOutput := HasMachineOutputFlag()
+	machineOutput := hasMachineOutputFlag()
 
 	// If the user has requested a "node" readable format,
 	// then disable login so the output remains valid.
@@ -80,6 +78,7 @@ func initConfig() {
 				Repository:    "headscale",
 				TagFilterFunc: filterPreReleasesIfStable(func() string { return versionInfo.Version }),
 			}
+
 			res, err := latest.Check(githubTag, versionInfo.Version)
 			if err == nil && res.Outdated {
 				//nolint
@@ -101,6 +100,7 @@ func isPreReleaseVersion(version string) bool {
 			return true
 		}
 	}
+
 	return false
 }
 
@@ -137,11 +137,15 @@ var rootCmd = &cobra.Command{
 headscale is an open source implementation of the Tailscale control server
 
 https://github.com/juanfont/headscale`,
+	SilenceErrors: true,
+	SilenceUsage:  true,
 }
 
 func Execute() {
-	if err := rootCmd.Execute(); err != nil {
-		fmt.Fprintln(os.Stderr, err)
+	cmd, err := rootCmd.ExecuteC()
+	if err != nil {
+		outputFormat, _ := cmd.Flags().GetString("output")
+		printError(err, outputFormat)
 		os.Exit(1)
 	}
 }

cmd/headscale/cli/serve.go

@@ -5,7 +5,6 @@ import (
 	"fmt"
 	"net/http"
 
-	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
 	"github.com/tailscale/squibble"
 )
@@ -17,24 +16,22 @@ func init() {
 var serveCmd = &cobra.Command{
 	Use:   "serve",
 	Short: "Launches the headscale server",
-	Args: func(cmd *cobra.Command, args []string) error {
-		return nil
-	},
-	Run: func(cmd *cobra.Command, args []string) {
+	RunE: func(cmd *cobra.Command, args []string) error {
 		app, err := newHeadscaleServerWithConfig()
 		if err != nil {
-			var squibbleErr squibble.ValidationError
-			if errors.As(err, &squibbleErr) {
+			if squibbleErr, ok := errors.AsType[squibble.ValidationError](err); ok {
 				fmt.Printf("SQLite schema failed to validate:\n")
 				fmt.Println(squibbleErr.Diff)
 			}
 
-			log.Fatal().Caller().Err(err).Msg("Error initializing")
+			return fmt.Errorf("initializing: %w", err)
 		}
 
 		err = app.Serve()
 		if err != nil && !errors.Is(err, http.ErrServerClosed) {
-			log.Fatal().Caller().Err(err).Msg("Headscale ran into an error and had to shut down.")
+			return fmt.Errorf("headscale ran into an error and had to shut down: %w", err)
 		}
+
+		return nil
 	},
 }

cmd/headscale/cli/users.go

@@ -1,6 +1,7 @@
 package cli
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"net/url"
@@ -8,10 +9,16 @@ import (
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/juanfont/headscale/hscontrol/util"
+	"github.com/juanfont/headscale/hscontrol/util/zlog/zf"
 	"github.com/pterm/pterm"
 	"github.com/rs/zerolog/log"
 	"github.com/spf13/cobra"
-	"google.golang.org/grpc/status"
+)
+
+// CLI user errors.
+var (
+	errFlagRequired       = errors.New("--name or --identifier flag is required")
+	errMultipleUsersMatch = errors.New("multiple users match query, specify an ID")
 )
 
 func usernameAndIDFlag(cmd *cobra.Command) {
@@ -20,20 +27,21 @@ func usernameAndIDFlag(cmd *cobra.Command) {
 }
 
 // usernameAndIDFromFlag returns the username and ID from the flags of the command.
-// If both are empty, it will exit the program with an error.
-func usernameAndIDFromFlag(cmd *cobra.Command) (uint64, string) {
+func usernameAndIDFromFlag(cmd *cobra.Command) (uint64, string, error) {
 	username, _ := cmd.Flags().GetString("name")
+
 	identifier, _ := cmd.Flags().GetInt64("identifier")
 	if username == "" && identifier < 0 {
-		err := errors.New("--name or --identifier flag is required")
-		ErrorOutput(
-			err,
-			"Cannot rename user: "+status.Convert(err).Message(),
-			"",
-		)
+		return 0, "", errFlagRequired
 	}
 
-	return uint64(identifier), username
+	// Normalise unset/negative identifiers to 0 so the uint64
+	// conversion does not produce a bogus large value.
+	if identifier < 0 {
+		identifier = 0
+	}
+
+	return uint64(identifier), username, nil //nolint:gosec // identifier is clamped to >= 0 above
 }
 
 func init() {
@@ -50,15 +58,13 @@ func init() {
 	userCmd.AddCommand(renameUserCmd)
 	usernameAndIDFlag(renameUserCmd)
 	renameUserCmd.Flags().StringP("new-name", "r", "", "New username")
-	renameNodeCmd.MarkFlagRequired("new-name")
+	mustMarkRequired(renameUserCmd, "new-name")
 }
 
-var errMissingParameter = errors.New("missing parameters")
-
 var userCmd = &cobra.Command{
 	Use:     "users",
 	Short:   "Manage the users of Headscale",
-	Aliases: []string{"user", "namespace", "namespaces", "ns"},
+	Aliases: []string{"user"},
 }
 
 var createUserCmd = &cobra.Command{
@@ -72,16 +78,10 @@ var createUserCmd = &cobra.Command{
 
 		return nil
 	},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
 		userName := args[0]
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		log.Trace().Interface("client", client).Msg("Obtained gRPC client")
+		log.Trace().Interface(zf.Client, client).Msg("obtained gRPC client")
 
 		request := &v1.CreateUserRequest{Name: userName}
 
@@ -94,108 +94,73 @@ var createUserCmd = &cobra.Command{
 		}
 
 		if pictureURL, _ := cmd.Flags().GetString("picture-url"); pictureURL != "" {
-			if _, err := url.Parse(pictureURL); err != nil {
-				ErrorOutput(
-					err,
-					fmt.Sprintf(
-						"Invalid Picture URL: %s",
-						err,
-					),
-					output,
-				)
+			if _, err := url.Parse(pictureURL); err != nil { //nolint:noinlineerr
+				return fmt.Errorf("invalid picture URL: %w", err)
 			}
+
 			request.PictureUrl = pictureURL
 		}
 
-		log.Trace().Interface("request", request).Msg("Sending CreateUser request")
+		log.Trace().Interface(zf.Request, request).Msg("sending CreateUser request")
+
 		response, err := client.CreateUser(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot create user: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("creating user: %w", err)
 		}
 
-		SuccessOutput(response.GetUser(), "User created", output)
-	},
+		return printOutput(cmd, response.GetUser(), "User created")
+	}),
 }
 
 var destroyUserCmd = &cobra.Command{
 	Use:     "destroy --identifier ID or --name NAME",
 	Short:   "Destroys a user",
 	Aliases: []string{"delete"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, username, err := usernameAndIDFromFlag(cmd)
+		if err != nil {
+			return err
+		}
 
-		id, username := usernameAndIDFromFlag(cmd)
 		request := &v1.ListUsersRequest{
 			Name: username,
 			Id:   id,
 		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
 		users, err := client.ListUsers(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Error: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("listing users: %w", err)
 		}
 
 		if len(users.GetUsers()) != 1 {
-			err := errors.New("Unable to determine user to delete, query returned multiple users, use ID")
-			ErrorOutput(
-				err,
-				"Error: "+status.Convert(err).Message(),
-				output,
-			)
+			return errMultipleUsersMatch
 		}
 
 		user := users.GetUsers()[0]
 
-		confirm := false
-		force, _ := cmd.Flags().GetBool("force")
-		if !force {
-			confirm = util.YesNo(fmt.Sprintf(
-				"Do you want to remove the user %q (%d) and any associated preauthkeys?",
-				user.GetName(), user.GetId(),
-			))
+		if !confirmAction(cmd, fmt.Sprintf(
+			"Do you want to remove the user %q (%d) and any associated preauthkeys?",
+			user.GetName(), user.GetId(),
+		)) {
+			return printOutput(cmd, map[string]string{"Result": "User not destroyed"}, "User not destroyed")
 		}
 
-		if confirm || force {
-			request := &v1.DeleteUserRequest{Id: user.GetId()}
+		deleteRequest := &v1.DeleteUserRequest{Id: user.GetId()}
 
-			response, err := client.DeleteUser(ctx, request)
-			if err != nil {
-				ErrorOutput(
-					err,
-					"Cannot destroy user: "+status.Convert(err).Message(),
-					output,
-				)
-			}
-			SuccessOutput(response, "User destroyed", output)
-		} else {
-			SuccessOutput(map[string]string{"Result": "User not destroyed"}, "User not destroyed", output)
+		response, err := client.DeleteUser(ctx, deleteRequest)
+		if err != nil {
+			return fmt.Errorf("destroying user: %w", err)
 		}
-	},
+
+		return printOutput(cmd, response, "User destroyed")
+	}),
 }
 
 var listUsersCmd = &cobra.Command{
 	Use:     "list",
 	Short:   "List all the users",
 	Aliases: []string{"ls", "show"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
 		request := &v1.ListUsersRequest{}
 
 		id, _ := cmd.Flags().GetInt64("identifier")
@@ -214,53 +179,39 @@ var listUsersCmd = &cobra.Command{
 
 		response, err := client.ListUsers(ctx, request)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot get users: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("listing users: %w", err)
 		}
 
-		if output != "" {
-			SuccessOutput(response.GetUsers(), "", output)
-		}
+		return printListOutput(cmd, response.GetUsers(), func() error {
+			tableData := pterm.TableData{{"ID", "Name", "Username", "Email", "Created"}}
+			for _, user := range response.GetUsers() {
+				tableData = append(
+					tableData,
+					[]string{
+						strconv.FormatUint(user.GetId(), util.Base10),
+						user.GetDisplayName(),
+						user.GetName(),
+						user.GetEmail(),
+						user.GetCreatedAt().AsTime().Format(HeadscaleDateTimeFormat),
+					},
+				)
+			}
 
-		tableData := pterm.TableData{{"ID", "Name", "Username", "Email", "Created"}}
-		for _, user := range response.GetUsers() {
-			tableData = append(
-				tableData,
-				[]string{
-					strconv.FormatUint(user.GetId(), 10),
-					user.GetDisplayName(),
-					user.GetName(),
-					user.GetEmail(),
-					user.GetCreatedAt().AsTime().Format("2006-01-02 15:04:05"),
-				},
-			)
-		}
-		err = pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
-		if err != nil {
-			ErrorOutput(
-				err,
-				fmt.Sprintf("Failed to render pterm table: %s", err),
-				output,
-			)
-		}
-	},
+			return pterm.DefaultTable.WithHasHeader().WithData(tableData).Render()
+		})
+	}),
 }
 
 var renameUserCmd = &cobra.Command{
 	Use:     "rename",
 	Short:   "Renames a user",
 	Aliases: []string{"mv"},
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
+	RunE: grpcRunE(func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error {
+		id, username, err := usernameAndIDFromFlag(cmd)
+		if err != nil {
+			return err
+		}
 
-		ctx, client, conn, cancel := newHeadscaleCLIWithConfig()
-		defer cancel()
-		defer conn.Close()
-
-		id, username := usernameAndIDFromFlag(cmd)
 		listReq := &v1.ListUsersRequest{
 			Name: username,
 			Id:   id,
@@ -268,20 +219,11 @@ var renameUserCmd = &cobra.Command{
 
 		users, err := client.ListUsers(ctx, listReq)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Error: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("listing users: %w", err)
 		}
 
 		if len(users.GetUsers()) != 1 {
-			err := errors.New("Unable to determine user to delete, query returned multiple users, use ID")
-			ErrorOutput(
-				err,
-				"Error: "+status.Convert(err).Message(),
-				output,
-			)
+			return errMultipleUsersMatch
 		}
 
 		newName, _ := cmd.Flags().GetString("new-name")
@@ -293,13 +235,9 @@ var renameUserCmd = &cobra.Command{
 
 		response, err := client.RenameUser(ctx, renameReq)
 		if err != nil {
-			ErrorOutput(
-				err,
-				"Cannot rename user: "+status.Convert(err).Message(),
-				output,
-			)
+			return fmt.Errorf("renaming user: %w", err)
 		}
 
-		SuccessOutput(response.GetUser(), "User renamed", output)
-	},
+		return printOutput(cmd, response.GetUser(), "User renamed")
+	}),
 }

cmd/headscale/cli/utils.go

@@ -4,25 +4,52 @@ import (
 	"context"
 	"crypto/tls"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"os"
+	"time"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
 	"github.com/juanfont/headscale/hscontrol"
 	"github.com/juanfont/headscale/hscontrol/types"
 	"github.com/juanfont/headscale/hscontrol/util"
+	"github.com/juanfont/headscale/hscontrol/util/zlog/zf"
+	"github.com/prometheus/common/model"
 	"github.com/rs/zerolog/log"
+	"github.com/spf13/cobra"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials"
 	"google.golang.org/grpc/credentials/insecure"
+	"google.golang.org/protobuf/types/known/timestamppb"
 	"gopkg.in/yaml.v3"
 )
 
 const (
 	HeadscaleDateTimeFormat = "2006-01-02 15:04:05"
 	SocketWritePermissions  = 0o666
+
+	outputFormatJSON     = "json"
+	outputFormatJSONLine = "json-line"
+	outputFormatYAML     = "yaml"
 )
 
+var (
+	errAPIKeyNotSet     = errors.New("HEADSCALE_CLI_API_KEY environment variable needs to be set")
+	errMissingParameter = errors.New("missing parameters")
+)
+
+// mustMarkRequired marks the named flags as required on cmd, panicking
+// if any name does not match a registered flag.  This is only called
+// from init() where a failure indicates a programming error.
+func mustMarkRequired(cmd *cobra.Command, names ...string) {
+	for _, n := range names {
+		err := cmd.MarkFlagRequired(n)
+		if err != nil {
+			panic(fmt.Sprintf("marking flag %q required on %q: %v", n, cmd.Name(), err))
+		}
+	}
+}
+
 func newHeadscaleServerWithConfig() (*hscontrol.Headscale, error) {
 	cfg, err := types.LoadServerConfig()
 	if err != nil {
@@ -40,14 +67,28 @@ func newHeadscaleServerWithConfig() (*hscontrol.Headscale, error) {
 	return app, nil
 }
 
-func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *grpc.ClientConn, context.CancelFunc) {
+// grpcRunE wraps a cobra RunE func, injecting a ready gRPC client and
+// context. Connection lifecycle is managed by the wrapper — callers
+// never see the underlying conn or cancel func.
+func grpcRunE(
+	fn func(ctx context.Context, client v1.HeadscaleServiceClient, cmd *cobra.Command, args []string) error,
+) func(*cobra.Command, []string) error {
+	return func(cmd *cobra.Command, args []string) error {
+		ctx, client, conn, cancel, err := newHeadscaleCLIWithConfig()
+		if err != nil {
+			return fmt.Errorf("connecting to headscale: %w", err)
+		}
+		defer cancel()
+		defer conn.Close()
+
+		return fn(ctx, client, cmd, args)
+	}
+}
+
+func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *grpc.ClientConn, context.CancelFunc, error) {
 	cfg, err := types.LoadCLIConfig()
 	if err != nil {
-		log.Fatal().
-			Err(err).
-			Caller().
-			Msgf("Failed to load configuration")
-		os.Exit(-1) // we get here if logging is suppressed (i.e., json output)
+		return nil, nil, nil, nil, fmt.Errorf("loading configuration: %w", err)
 	}
 
 	log.Debug().
@@ -57,7 +98,7 @@ func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *g
 	ctx, cancel := context.WithTimeout(context.Background(), cfg.CLI.Timeout)
 
 	grpcOptions := []grpc.DialOption{
-		grpc.WithBlock(),
+		grpc.WithBlock(), //nolint:staticcheck // SA1019: deprecated but supported in 1.x
 	}
 
 	address := cfg.CLI.Address
@@ -71,17 +112,23 @@ func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *g
 		address = cfg.UnixSocket
 
 		// Try to give the user better feedback if we cannot write to the headscale
-		// socket.
-		socket, err := os.OpenFile(cfg.UnixSocket, os.O_WRONLY, SocketWritePermissions) // nolint
+		// socket.  Note: os.OpenFile on a Unix domain socket returns ENXIO on
+		// Linux which is expected — only permission errors are actionable here.
+		// The actual gRPC connection uses net.Dial which handles sockets properly.
+		socket, err := os.OpenFile(cfg.UnixSocket, os.O_WRONLY, SocketWritePermissions) //nolint
 		if err != nil {
 			if os.IsPermission(err) {
-				log.Fatal().
-					Err(err).
-					Str("socket", cfg.UnixSocket).
-					Msgf("Unable to read/write to headscale socket, do you have the correct permissions?")
+				cancel()
+
+				return nil, nil, nil, nil, fmt.Errorf(
+					"unable to read/write to headscale socket %q, do you have the correct permissions? %w",
+					cfg.UnixSocket,
+					err,
+				)
 			}
+		} else {
+			socket.Close()
 		}
-		socket.Close()
 
 		grpcOptions = append(
 			grpcOptions,
@@ -92,8 +139,11 @@ func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *g
 		// If we are not connecting to a local server, require an API key for authentication
 		apiKey := cfg.CLI.APIKey
 		if apiKey == "" {
-			log.Fatal().Caller().Msgf("HEADSCALE_CLI_API_KEY environment variable needs to be set.")
+			cancel()
+
+			return nil, nil, nil, nil, errAPIKeyNotSet
 		}
+
 		grpcOptions = append(grpcOptions,
 			grpc.WithPerRPCCredentials(tokenAuth{
 				token: apiKey,
@@ -118,71 +168,136 @@ func newHeadscaleCLIWithConfig() (context.Context, v1.HeadscaleServiceClient, *g
 		}
 	}
 
-	log.Trace().Caller().Str("address", address).Msg("Connecting via gRPC")
-	conn, err := grpc.DialContext(ctx, address, grpcOptions...)
+	log.Trace().Caller().Str(zf.Address, address).Msg("connecting via gRPC")
+
+	conn, err := grpc.DialContext(ctx, address, grpcOptions...) //nolint:staticcheck // SA1019: deprecated but supported in 1.x
 	if err != nil {
-		log.Fatal().Caller().Err(err).Msgf("Could not connect: %v", err)
-		os.Exit(-1) // we get here if logging is suppressed (i.e., json output)
+		cancel()
+
+		return nil, nil, nil, nil, fmt.Errorf("connecting to %s: %w", address, err)
 	}
 
 	client := v1.NewHeadscaleServiceClient(conn)
 
-	return ctx, client, conn, cancel
+	return ctx, client, conn, cancel, nil
 }
 
-func output(result interface{}, override string, outputFormat string) string {
-	var jsonBytes []byte
-	var err error
+// formatOutput serialises result into the requested format. For the
+// default (empty) format the human-readable override string is returned.
+func formatOutput(result any, override string, outputFormat string) (string, error) {
 	switch outputFormat {
-	case "json":
-		jsonBytes, err = json.MarshalIndent(result, "", "\t")
+	case outputFormatJSON:
+		b, err := json.MarshalIndent(result, "", "\t")
 		if err != nil {
-			log.Fatal().Err(err).Msg("failed to unmarshal output")
+			return "", fmt.Errorf("marshalling JSON output: %w", err)
 		}
-	case "json-line":
-		jsonBytes, err = json.Marshal(result)
+
+		return string(b), nil
+	case outputFormatJSONLine:
+		b, err := json.Marshal(result)
 		if err != nil {
-			log.Fatal().Err(err).Msg("failed to unmarshal output")
+			return "", fmt.Errorf("marshalling JSON-line output: %w", err)
 		}
-	case "yaml":
-		jsonBytes, err = yaml.Marshal(result)
+
+		return string(b), nil
+	case outputFormatYAML:
+		b, err := yaml.Marshal(result)
 		if err != nil {
-			log.Fatal().Err(err).Msg("failed to unmarshal output")
+			return "", fmt.Errorf("marshalling YAML output: %w", err)
 		}
+
+		return string(b), nil
 	default:
-		// nolint
-		return override
+		return override, nil
+	}
+}
+
+// printOutput formats result and writes it to stdout. It reads the --output
+// flag from cmd to decide the serialisation format.
+func printOutput(cmd *cobra.Command, result any, override string) error {
+	format, _ := cmd.Flags().GetString("output")
+
+	out, err := formatOutput(result, override, format)
+	if err != nil {
+		return err
 	}
 
-	return string(jsonBytes)
+	fmt.Println(out)
+
+	return nil
 }
 
-// SuccessOutput prints the result to stdout and exits with status code 0.
-func SuccessOutput(result interface{}, override string, outputFormat string) {
-	fmt.Println(output(result, override, outputFormat))
-	os.Exit(0)
+// expirationFromFlag parses the --expiration flag as a Prometheus-style
+// duration (e.g. "90d", "1h") and returns an absolute timestamp.
+func expirationFromFlag(cmd *cobra.Command) (*timestamppb.Timestamp, error) {
+	durationStr, _ := cmd.Flags().GetString("expiration")
+
+	duration, err := model.ParseDuration(durationStr)
+	if err != nil {
+		return nil, fmt.Errorf("parsing duration: %w", err)
+	}
+
+	return timestamppb.New(time.Now().UTC().Add(time.Duration(duration))), nil
 }
 
-// ErrorOutput prints an error message to stderr and exits with status code 1.
-func ErrorOutput(errResult error, override string, outputFormat string) {
+// confirmAction returns true when the user confirms a prompt, or when
+// --force is set.  Callers decide what to do when it returns false.
+func confirmAction(cmd *cobra.Command, prompt string) bool {
+	force, _ := cmd.Flags().GetBool("force")
+	if force {
+		return true
+	}
+
+	return util.YesNo(prompt)
+}
+
+// printListOutput checks the --output flag: when a machine-readable format is
+// requested it serialises data as JSON/YAML; otherwise it calls renderTable
+// to produce the human-readable pterm table.
+func printListOutput(
+	cmd *cobra.Command,
+	data any,
+	renderTable func() error,
+) error {
+	format, _ := cmd.Flags().GetString("output")
+	if format != "" {
+		return printOutput(cmd, data, "")
+	}
+
+	return renderTable()
+}
+
+// printError writes err to stderr, formatting it as JSON/YAML when the
+// --output flag requests machine-readable output.  Used exclusively by
+// Execute() so that every error surfaces in the format the caller asked for.
+func printError(err error, outputFormat string) {
 	type errOutput struct {
 		Error string `json:"error"`
 	}
 
-	var errorMessage string
-	if errResult != nil {
-		errorMessage = errResult.Error()
-	} else {
-		errorMessage = override
+	e := errOutput{Error: err.Error()}
+
+	var formatted []byte
+
+	switch outputFormat {
+	case outputFormatJSON:
+		formatted, _ = json.MarshalIndent(e, "", "\t") //nolint:errchkjson // errOutput contains only a string field
+	case outputFormatJSONLine:
+		formatted, _ = json.Marshal(e) //nolint:errchkjson // errOutput contains only a string field
+	case outputFormatYAML:
+		formatted, _ = yaml.Marshal(e)
+	default:
+		fmt.Fprintf(os.Stderr, "Error: %s\n", err)
+
+		return
 	}
 
-	fmt.Fprintf(os.Stderr, "%s\n", output(errOutput{errorMessage}, override, outputFormat))
-	os.Exit(1)
+	fmt.Fprintf(os.Stderr, "%s\n", formatted)
 }
 
-func HasMachineOutputFlag() bool {
+func hasMachineOutputFlag() bool {
 	for _, arg := range os.Args {
-		if arg == "json" || arg == "json-line" || arg == "yaml" {
+		if arg == outputFormatJSON || arg == outputFormatJSONLine || arg == outputFormatYAML {
 			return true
 		}
 	}

cmd/headscale/cli/version.go

@@ -14,11 +14,9 @@ var versionCmd = &cobra.Command{
 	Use:   "version",
 	Short: "Print the version.",
 	Long:  "The version of headscale.",
-	Run: func(cmd *cobra.Command, args []string) {
-		output, _ := cmd.Flags().GetString("output")
-
+	RunE: func(cmd *cobra.Command, args []string) error {
 		info := types.GetVersionInfo()
 
-		SuccessOutput(info, info.String(), output)
+		return printOutput(cmd, info, info.String())
 	},
 }

cmd/headscale/headscale.go

@@ -12,6 +12,7 @@ import (
 
 func main() {
 	var colors bool
+
 	switch l := termcolor.SupportLevel(os.Stderr); l {
 	case termcolor.Level16M:
 		colors = true

cmd/headscale/headscale_test.go

@@ -9,34 +9,15 @@ import (
 	"github.com/juanfont/headscale/hscontrol/types"
 	"github.com/juanfont/headscale/hscontrol/util"
 	"github.com/spf13/viper"
-	"gopkg.in/check.v1"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
-func Test(t *testing.T) {
-	check.TestingT(t)
-}
-
-var _ = check.Suite(&Suite{})
-
-type Suite struct{}
-
-func (s *Suite) SetUpSuite(c *check.C) {
-}
-
-func (s *Suite) TearDownSuite(c *check.C) {
-}
-
-func (*Suite) TestConfigFileLoading(c *check.C) {
-	tmpDir, err := os.MkdirTemp("", "headscale")
-	if err != nil {
-		c.Fatal(err)
-	}
-	defer os.RemoveAll(tmpDir)
+func TestConfigFileLoading(t *testing.T) {
+	tmpDir := t.TempDir()
 
 	path, err := os.Getwd()
-	if err != nil {
-		c.Fatal(err)
-	}
+	require.NoError(t, err)
 
 	cfgFile := filepath.Join(tmpDir, "config.yaml")
 
@@ -45,70 +26,52 @@ func (*Suite) TestConfigFileLoading(c *check.C) {
 		filepath.Clean(path+"/../../config-example.yaml"),
 		cfgFile,
 	)
-	if err != nil {
-		c.Fatal(err)
-	}
+	require.NoError(t, err)
 
 	// Load example config, it should load without validation errors
 	err = types.LoadConfig(cfgFile, true)
-	c.Assert(err, check.IsNil)
+	require.NoError(t, err)
 
 	// Test that config file was interpreted correctly
-	c.Assert(viper.GetString("server_url"), check.Equals, "http://127.0.0.1:8080")
-	c.Assert(viper.GetString("listen_addr"), check.Equals, "127.0.0.1:8080")
-	c.Assert(viper.GetString("metrics_listen_addr"), check.Equals, "127.0.0.1:9090")
-	c.Assert(viper.GetString("database.type"), check.Equals, "sqlite")
-	c.Assert(viper.GetString("database.sqlite.path"), check.Equals, "/var/lib/headscale/db.sqlite")
-	c.Assert(viper.GetString("tls_letsencrypt_hostname"), check.Equals, "")
-	c.Assert(viper.GetString("tls_letsencrypt_listen"), check.Equals, ":http")
-	c.Assert(viper.GetString("tls_letsencrypt_challenge_type"), check.Equals, "HTTP-01")
-	c.Assert(
-		util.GetFileMode("unix_socket_permission"),
-		check.Equals,
-		fs.FileMode(0o770),
-	)
-	c.Assert(viper.GetBool("logtail.enabled"), check.Equals, false)
+	assert.Equal(t, "http://127.0.0.1:8080", viper.GetString("server_url"))
+	assert.Equal(t, "127.0.0.1:8080", viper.GetString("listen_addr"))
+	assert.Equal(t, "127.0.0.1:9090", viper.GetString("metrics_listen_addr"))
+	assert.Equal(t, "sqlite", viper.GetString("database.type"))
+	assert.Equal(t, "/var/lib/headscale/db.sqlite", viper.GetString("database.sqlite.path"))
+	assert.Empty(t, viper.GetString("tls_letsencrypt_hostname"))
+	assert.Equal(t, ":http", viper.GetString("tls_letsencrypt_listen"))
+	assert.Equal(t, "HTTP-01", viper.GetString("tls_letsencrypt_challenge_type"))
+	assert.Equal(t, fs.FileMode(0o770), util.GetFileMode("unix_socket_permission"))
+	assert.False(t, viper.GetBool("logtail.enabled"))
 }
 
-func (*Suite) TestConfigLoading(c *check.C) {
-	tmpDir, err := os.MkdirTemp("", "headscale")
-	if err != nil {
-		c.Fatal(err)
-	}
-	defer os.RemoveAll(tmpDir)
+func TestConfigLoading(t *testing.T) {
+	tmpDir := t.TempDir()
 
 	path, err := os.Getwd()
-	if err != nil {
-		c.Fatal(err)
-	}
+	require.NoError(t, err)
 
 	// Symlink the example config file
 	err = os.Symlink(
 		filepath.Clean(path+"/../../config-example.yaml"),
 		filepath.Join(tmpDir, "config.yaml"),
 	)
-	if err != nil {
-		c.Fatal(err)
-	}
+	require.NoError(t, err)
 
 	// Load example config, it should load without validation errors
 	err = types.LoadConfig(tmpDir, false)
-	c.Assert(err, check.IsNil)
+	require.NoError(t, err)
 
 	// Test that config file was interpreted correctly
-	c.Assert(viper.GetString("server_url"), check.Equals, "http://127.0.0.1:8080")
-	c.Assert(viper.GetString("listen_addr"), check.Equals, "127.0.0.1:8080")
-	c.Assert(viper.GetString("metrics_listen_addr"), check.Equals, "127.0.0.1:9090")
-	c.Assert(viper.GetString("database.type"), check.Equals, "sqlite")
-	c.Assert(viper.GetString("database.sqlite.path"), check.Equals, "/var/lib/headscale/db.sqlite")
-	c.Assert(viper.GetString("tls_letsencrypt_hostname"), check.Equals, "")
-	c.Assert(viper.GetString("tls_letsencrypt_listen"), check.Equals, ":http")
-	c.Assert(viper.GetString("tls_letsencrypt_challenge_type"), check.Equals, "HTTP-01")
-	c.Assert(
-		util.GetFileMode("unix_socket_permission"),
-		check.Equals,
-		fs.FileMode(0o770),
-	)
-	c.Assert(viper.GetBool("logtail.enabled"), check.Equals, false)
-	c.Assert(viper.GetBool("randomize_client_port"), check.Equals, false)
+	assert.Equal(t, "http://127.0.0.1:8080", viper.GetString("server_url"))
+	assert.Equal(t, "127.0.0.1:8080", viper.GetString("listen_addr"))
+	assert.Equal(t, "127.0.0.1:9090", viper.GetString("metrics_listen_addr"))
+	assert.Equal(t, "sqlite", viper.GetString("database.type"))
+	assert.Equal(t, "/var/lib/headscale/db.sqlite", viper.GetString("database.sqlite.path"))
+	assert.Empty(t, viper.GetString("tls_letsencrypt_hostname"))
+	assert.Equal(t, ":http", viper.GetString("tls_letsencrypt_listen"))
+	assert.Equal(t, "HTTP-01", viper.GetString("tls_letsencrypt_challenge_type"))
+	assert.Equal(t, fs.FileMode(0o770), util.GetFileMode("unix_socket_permission"))
+	assert.False(t, viper.GetBool("logtail.enabled"))
+	assert.False(t, viper.GetBool("randomize_client_port"))
 }

cmd/hi/README.md

@@ -0,0 +1,262 @@
+# hi — Headscale Integration test runner
+
+`hi` wraps Docker container orchestration around the tests in
+[`../../integration`](../../integration) and extracts debugging artefacts
+(logs, database snapshots, MapResponse protocol captures) for post-mortem
+analysis.
+
+**Read this file in full before running any `hi` command.** The test
+runner has sharp edges — wrong flags produce stale containers, lost
+artefacts, or hung CI.
+
+For test-authoring patterns (scenario setup, `EventuallyWithT`,
+`IntegrationSkip`, helper variants), read
+[`../../integration/README.md`](../../integration/README.md).
+
+## Quick Start
+
+```bash
+# Verify system requirements (Docker, Go, disk space, images)
+go run ./cmd/hi doctor
+
+# Run a single test (the default flags are tuned for development)
+go run ./cmd/hi run "TestPingAllByIP"
+
+# Run a database-heavy test against PostgreSQL
+go run ./cmd/hi run "TestExpireNode" --postgres
+
+# Pattern matching
+go run ./cmd/hi run "TestSubnet*"
+```
+
+Run `doctor` before the first `run` in any new environment. Tests
+generate ~100 MB of logs per run in `control_logs/`; `doctor` verifies
+there is enough space and that the required Docker images are available.
+
+## Commands
+
+| Command            | Purpose                                              |
+| ------------------ | ---------------------------------------------------- |
+| `run [pattern]`    | Execute the test(s) matching `pattern`               |
+| `doctor`           | Verify system requirements                           |
+| `clean networks`   | Prune unused Docker networks                         |
+| `clean images`     | Clean old test images                                |
+| `clean containers` | Kill **all** test containers (dangerous — see below) |
+| `clean cache`      | Clean Go module cache volume                         |
+| `clean all`        | Run all cleanup operations                           |
+
+## Flags
+
+Defaults are tuned for single-test development runs. Review before
+changing.
+
+| Flag                | Default        | Purpose                                                                     |
+| ------------------- | -------------- | --------------------------------------------------------------------------- |
+| `--timeout`         | `120m`         | Total test timeout. Use the built-in flag — never wrap with bash `timeout`. |
+| `--postgres`        | `false`        | Use PostgreSQL instead of SQLite                                            |
+| `--failfast`        | `true`         | Stop on first test failure                                                  |
+| `--go-version`      | auto           | Detected from `go.mod` (currently 1.26.1)                                   |
+| `--clean-before`    | `true`         | Clean stale (stopped/exited) containers before starting                     |
+| `--clean-after`     | `true`         | Clean this run's containers after completion                                |
+| `--keep-on-failure` | `false`        | Preserve containers for manual inspection on failure                        |
+| `--logs-dir`        | `control_logs` | Where to save run artefacts                                                 |
+| `--verbose`         | `false`        | Verbose output                                                              |
+| `--stats`           | `false`        | Collect container resource-usage stats                                      |
+| `--hs-memory-limit` | `0`            | Fail if any headscale container exceeds N MB (0 = disabled)                 |
+| `--ts-memory-limit` | `0`            | Fail if any tailscale container exceeds N MB                                |
+
+### Timeout guidance
+
+The default `120m` is generous for a single test. If you must tune it,
+these are realistic floors by category:
+
+| Test type                 | Minimum     | Examples                              |
+| ------------------------- | ----------- | ------------------------------------- |
+| Basic functionality / CLI | 900s (15m)  | `TestPingAllByIP`, `TestCLI*`         |
+| Route / ACL               | 1200s (20m) | `TestSubnet*`, `TestACL*`             |
+| HA / failover             | 1800s (30m) | `TestHASubnetRouter*`                 |
+| Long-running              | 2100s (35m) | `TestNodeOnlineStatus` (~12 min body) |
+| Full suite                | 45m         | `go test ./integration -timeout 45m`  |
+
+**Never** use the shell `timeout` command around `hi`. It kills the
+process mid-cleanup and leaves stale containers:
+
+```bash
+timeout 300 go run ./cmd/hi run "TestName"   # WRONG — orphaned containers
+go run ./cmd/hi run "TestName" --timeout=900s  # correct
+```
+
+## Concurrent Execution
+
+Multiple `hi run` invocations can run simultaneously on the same Docker
+daemon. Each invocation gets a unique **Run ID** (format
+`YYYYMMDD-HHMMSS-6charhash`, e.g. `20260409-104215-mdjtzx`).
+
+- **Container names** include the short run ID: `ts-mdjtzx-1-74-fgdyls`
+- **Docker labels**: `hi.run-id={runID}` on every container
+- **Port allocation**: dynamic — kernel assigns free ports, no conflicts
+- **Cleanup isolation**: each run cleans only its own containers
+- **Log directories**: `control_logs/{runID}/`
+
+```bash
+# Start three tests in parallel — each gets its own run ID
+go run ./cmd/hi run "TestPingAllByIP" &
+go run ./cmd/hi run "TestACLAllowUserDst" &
+go run ./cmd/hi run "TestOIDCAuthenticationPingAll" &
+```
+
+### Safety rules for concurrent runs
+
+- ✅ Your run cleans only containers labelled with its own `hi.run-id`
+- ✅ `--clean-before` removes only stopped/exited containers
+- ❌ **Never** run `docker rm -f $(docker ps -q --filter name=hs-)` —
+  this destroys other agents' live test sessions
+- ❌ **Never** run `docker system prune -f` while any tests are running
+- ❌ **Never** run `hi clean containers` / `hi clean all` while other
+  tests are running — both kill all test containers on the daemon
+
+To identify your own containers:
+
+```bash
+docker ps --filter "label=hi.run-id=20260409-104215-mdjtzx"
+```
+
+The run ID appears at the top of the `hi run` output — copy it from
+there rather than trying to reconstruct it.
+
+## Artefacts
+
+Every run saves debugging artefacts under `control_logs/{runID}/`:
+
+```
+control_logs/20260409-104215-mdjtzx/
+├── hs-<test>-<hash>.stderr.log        # headscale server errors
+├── hs-<test>-<hash>.stdout.log        # headscale server output
+├── hs-<test>-<hash>.db                # database snapshot (SQLite)
+├── hs-<test>-<hash>_metrics.txt       # Prometheus metrics dump
+├── hs-<test>-<hash>-mapresponses/     # MapResponse protocol captures
+├── ts-<client>-<hash>.stderr.log      # tailscale client errors
+├── ts-<client>-<hash>.stdout.log      # tailscale client output
+└── ts-<client>-<hash>_status.json     # client network-status dump
+```
+
+Artefacts persist after cleanup. Old runs accumulate fast — delete
+unwanted directories to reclaim disk.
+
+## Debugging workflow
+
+When a test fails, read the artefacts **in this order**:
+
+1. **`hs-*.stderr.log`** — headscale server errors, panics, policy
+   evaluation failures. Most issues originate server-side.
+
+   ```bash
+   grep -E "ERROR|panic|FATAL" control_logs/*/hs-*.stderr.log
+   ```
+
+2. **`ts-*.stderr.log`** — authentication failures, connectivity issues,
+   DNS resolution problems on the client side.
+
+3. **MapResponse JSON** in `hs-*-mapresponses/` — protocol-level
+   debugging for network map generation, peer visibility, route
+   distribution, policy evaluation results.
+
+   ```bash
+   ls control_logs/*/hs-*-mapresponses/
+   jq '.Peers[] | {Name, Tags, PrimaryRoutes}' \
+       control_logs/*/hs-*-mapresponses/001.json
+   ```
+
+4. **`*_status.json`** — client peer-connectivity state.
+
+5. **`hs-*.db`** — SQLite snapshot for post-mortem consistency checks.
+
+   ```bash
+   sqlite3 control_logs/<runID>/hs-*.db
+   sqlite> .tables
+   sqlite> .schema nodes
+   sqlite> SELECT id, hostname, user_id, tags FROM nodes WHERE hostname LIKE '%problematic%';
+   ```
+
+6. **`*_metrics.txt`** — Prometheus dumps for latency, NodeStore
+   operation timing, database query performance, memory usage.
+
+## Heuristic: infrastructure vs code
+
+**Before blaming Docker, disk, or network: read `hs-*.stderr.log` in
+full.** In practice, well over 99% of failures are code bugs (policy
+evaluation, NodeStore sync, route approval) rather than infrastructure.
+
+Actual infrastructure failures have signature error messages:
+
+| Signature                                                       | Cause                     | Fix                                                           |
+| --------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------- |
+| `failed to resolve "hs-...": no DNS fallback candidates remain` | Docker DNS                | Reset Docker networking                                       |
+| `container creation timeout`, no progress >2 min                | Resource exhaustion       | `docker system prune -f` (when no other tests running), retry |
+| OOM kills, slow Docker daemon                                   | Too many concurrent tests | Reduce concurrency, wait for completion                       |
+| `no space left on device`                                       | Disk full                 | Delete old `control_logs/`                                    |
+
+If you don't see a signature error, **assume it's a code regression** —
+do not retry hoping the flake goes away.
+
+## Common failure patterns (code bugs)
+
+### Route advertisement timing
+
+Test asserts route state before the client has finished propagating its
+Hostinfo update. Symptom: `nodes[0].GetAvailableRoutes()` empty when
+the test expects a route.
+
+- **Wrong fix**: `time.Sleep(5 * time.Second)` — fragile and slow.
+- **Right fix**: wrap the assertion in `EventuallyWithT`. See
+  [`../../integration/README.md`](../../integration/README.md).
+
+### NodeStore sync issues
+
+Route changes not reflected in the NodeStore snapshot. Symptom: route
+advertisements in logs but no tracking updates in subsequent reads.
+
+The sync point is `State.UpdateNodeFromMapRequest()` in
+`hscontrol/state/state.go`. If you added a new kind of client state
+update, make sure it lands here.
+
+### HA failover: routes disappearing on disconnect
+
+`TestHASubnetRouterFailover` fails because approved routes vanish when
+a subnet router goes offline. **This is a bug, not expected behaviour.**
+Route approval must not be coupled to client connectivity — routes
+stay approved; only the primary-route selection is affected by
+connectivity.
+
+### Policy evaluation race
+
+Symptom: tests that change policy and immediately assert peer visibility
+fail intermittently. Policy changes trigger async recomputation.
+
+- See recent fixes in `git log -- hscontrol/state/` for examples (e.g.
+  the `PolicyChange` trigger on every Connect/Disconnect).
+
+### SQLite vs PostgreSQL timing differences
+
+Some race conditions only surface on one backend. If a test is flaky,
+try the other backend with `--postgres`:
+
+```bash
+go run ./cmd/hi run "TestName" --postgres --verbose
+```
+
+PostgreSQL generally has more consistent timing; SQLite can expose
+races during rapid writes.
+
+## Keeping containers for inspection
+
+If you need to inspect a failed test's state manually:
+
+```bash
+go run ./cmd/hi run "TestName" --keep-on-failure
+# containers survive — inspect them
+docker exec -it ts-<runID>-<...> /bin/sh
+docker logs hs-<runID>-<...>
+# clean up manually when done
+go run ./cmd/hi clean all   # only when no other tests are running
+```

cmd/hi/cleanup.go

@@ -3,9 +3,13 @@ package main
 import (
 	"context"
 	"fmt"
+	"log"
+	"os"
+	"path/filepath"
 	"strings"
 	"time"
 
+	"github.com/cenkalti/backoff/v5"
 	"github.com/docker/docker/api/types/container"
 	"github.com/docker/docker/api/types/filters"
 	"github.com/docker/docker/api/types/image"
@@ -14,30 +18,46 @@ import (
 )
 
 // cleanupBeforeTest performs cleanup operations before running tests.
+// Only removes stale (stopped/exited) test containers to avoid interfering with concurrent test runs.
 func cleanupBeforeTest(ctx context.Context) error {
-	if err := killTestContainers(ctx); err != nil {
-		return fmt.Errorf("failed to kill test containers: %w", err)
+	err := cleanupStaleTestContainers(ctx)
+	if err != nil {
+		return fmt.Errorf("cleaning stale test containers: %w", err)
 	}
 
-	if err := pruneDockerNetworks(ctx); err != nil {
-		return fmt.Errorf("failed to prune networks: %w", err)
+	if err := pruneDockerNetworks(ctx); err != nil { //nolint:noinlineerr
+		return fmt.Errorf("pruning networks: %w", err)
 	}
 
 	return nil
 }
 
-// cleanupAfterTest removes the test container after completion.
-func cleanupAfterTest(ctx context.Context, cli *client.Client, containerID string) error {
-	return cli.ContainerRemove(ctx, containerID, container.RemoveOptions{
+// cleanupAfterTest removes the test container and all associated integration test containers for the run.
+func cleanupAfterTest(ctx context.Context, cli *client.Client, containerID, runID string) error {
+	// Remove the main test container
+	err := cli.ContainerRemove(ctx, containerID, container.RemoveOptions{
 		Force: true,
 	})
+	if err != nil {
+		return fmt.Errorf("removing test container: %w", err)
+	}
+
+	// Clean up integration test containers for this run only
+	if runID != "" {
+		err := killTestContainersByRunID(ctx, runID)
+		if err != nil {
+			return fmt.Errorf("cleaning up containers for run %s: %w", runID, err)
+		}
+	}
+
+	return nil
 }
 
 // killTestContainers terminates and removes all test containers.
 func killTestContainers(ctx context.Context) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
@@ -45,12 +65,14 @@ func killTestContainers(ctx context.Context) error {
 		All: true,
 	})
 	if err != nil {
-		return fmt.Errorf("failed to list containers: %w", err)
+		return fmt.Errorf("listing containers: %w", err)
 	}
 
 	removed := 0
+
 	for _, cont := range containers {
 		shouldRemove := false
+
 		for _, name := range cont.Names {
 			if strings.Contains(name, "headscale-test-suite") ||
 				strings.Contains(name, "hs-") ||
@@ -83,43 +105,135 @@ func killTestContainers(ctx context.Context) error {
 	return nil
 }
 
+// killTestContainersByRunID terminates and removes all test containers for a specific run ID.
+// This function filters containers by the hi.run-id label to only affect containers
+// belonging to the specified test run, leaving other concurrent test runs untouched.
+func killTestContainersByRunID(ctx context.Context, runID string) error {
+	cli, err := createDockerClient(ctx)
+	if err != nil {
+		return fmt.Errorf("creating Docker client: %w", err)
+	}
+	defer cli.Close()
+
+	// Filter containers by hi.run-id label
+	containers, err := cli.ContainerList(ctx, container.ListOptions{
+		All: true,
+		Filters: filters.NewArgs(
+			filters.Arg("label", "hi.run-id="+runID),
+		),
+	})
+	if err != nil {
+		return fmt.Errorf("listing containers for run %s: %w", runID, err)
+	}
+
+	removed := 0
+
+	for _, cont := range containers {
+		// Kill the container if it's running
+		if cont.State == "running" {
+			_ = cli.ContainerKill(ctx, cont.ID, "KILL")
+		}
+
+		// Remove the container with retry logic
+		if removeContainerWithRetry(ctx, cli, cont.ID) {
+			removed++
+		}
+	}
+
+	if removed > 0 {
+		fmt.Printf("Removed %d containers for run ID %s\n", removed, runID)
+	}
+
+	return nil
+}
+
+// cleanupStaleTestContainers removes stopped/exited test containers without affecting running tests.
+// This is useful for cleaning up leftover containers from previous crashed or interrupted test runs
+// without interfering with currently running concurrent tests.
+func cleanupStaleTestContainers(ctx context.Context) error {
+	cli, err := createDockerClient(ctx)
+	if err != nil {
+		return fmt.Errorf("creating Docker client: %w", err)
+	}
+	defer cli.Close()
+
+	// Only get stopped/exited containers
+	containers, err := cli.ContainerList(ctx, container.ListOptions{
+		All: true,
+		Filters: filters.NewArgs(
+			filters.Arg("status", "exited"),
+			filters.Arg("status", "dead"),
+		),
+	})
+	if err != nil {
+		return fmt.Errorf("listing stopped containers: %w", err)
+	}
+
+	removed := 0
+
+	for _, cont := range containers {
+		// Only remove containers that look like test containers
+		shouldRemove := false
+
+		for _, name := range cont.Names {
+			if strings.Contains(name, "headscale-test-suite") ||
+				strings.Contains(name, "hs-") ||
+				strings.Contains(name, "ts-") ||
+				strings.Contains(name, "derp-") {
+				shouldRemove = true
+				break
+			}
+		}
+
+		if shouldRemove {
+			if removeContainerWithRetry(ctx, cli, cont.ID) {
+				removed++
+			}
+		}
+	}
+
+	if removed > 0 {
+		fmt.Printf("Removed %d stale test containers\n", removed)
+	}
+
+	return nil
+}
+
+const (
+	containerRemoveInitialInterval = 100 * time.Millisecond
+	containerRemoveMaxElapsedTime  = 2 * time.Second
+)
+
 // removeContainerWithRetry attempts to remove a container with exponential backoff retry logic.
 func removeContainerWithRetry(ctx context.Context, cli *client.Client, containerID string) bool {
-	maxRetries := 3
-	baseDelay := 100 * time.Millisecond
+	expBackoff := backoff.NewExponentialBackOff()
+	expBackoff.InitialInterval = containerRemoveInitialInterval
 
-	for attempt := range maxRetries {
+	_, err := backoff.Retry(ctx, func() (struct{}, error) {
 		err := cli.ContainerRemove(ctx, containerID, container.RemoveOptions{
 			Force: true,
 		})
-		if err == nil {
-			return true
+		if err != nil {
+			return struct{}{}, err
 		}
 
-		// If this is the last attempt, don't wait
-		if attempt == maxRetries-1 {
-			break
-		}
+		return struct{}{}, nil
+	}, backoff.WithBackOff(expBackoff), backoff.WithMaxElapsedTime(containerRemoveMaxElapsedTime))
 
-		// Wait with exponential backoff
-		delay := baseDelay * time.Duration(1<<attempt)
-		time.Sleep(delay)
-	}
-
-	return false
+	return err == nil
 }
 
 // pruneDockerNetworks removes unused Docker networks.
 func pruneDockerNetworks(ctx context.Context) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
 	report, err := cli.NetworksPrune(ctx, filters.Args{})
 	if err != nil {
-		return fmt.Errorf("failed to prune networks: %w", err)
+		return fmt.Errorf("pruning networks: %w", err)
 	}
 
 	if len(report.NetworksDeleted) > 0 {
@@ -133,9 +247,9 @@ func pruneDockerNetworks(ctx context.Context) error {
 
 // cleanOldImages removes test-related and old dangling Docker images.
 func cleanOldImages(ctx context.Context) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
@@ -143,12 +257,14 @@ func cleanOldImages(ctx context.Context) error {
 		All: true,
 	})
 	if err != nil {
-		return fmt.Errorf("failed to list images: %w", err)
+		return fmt.Errorf("listing images: %w", err)
 	}
 
 	removed := 0
+
 	for _, img := range images {
 		shouldRemove := false
+
 		for _, tag := range img.RepoTags {
 			if strings.Contains(tag, "hs-") ||
 				strings.Contains(tag, "headscale-integration") ||
@@ -183,18 +299,19 @@ func cleanOldImages(ctx context.Context) error {
 
 // cleanCacheVolume removes the Docker volume used for Go module cache.
 func cleanCacheVolume(ctx context.Context) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
 	volumeName := "hs-integration-go-cache"
+
 	err = cli.VolumeRemove(ctx, volumeName, true)
 	if err != nil {
-		if errdefs.IsNotFound(err) {
+		if errdefs.IsNotFound(err) { //nolint:staticcheck // SA1019: deprecated but functional
 			fmt.Printf("Go module cache volume not found: %s\n", volumeName)
-		} else if errdefs.IsConflict(err) {
+		} else if errdefs.IsConflict(err) { //nolint:staticcheck // SA1019: deprecated but functional
 			fmt.Printf("Go module cache volume is in use and cannot be removed: %s\n", volumeName)
 		} else {
 			fmt.Printf("Failed to remove Go module cache volume %s: %v\n", volumeName, err)
@@ -205,3 +322,110 @@ func cleanCacheVolume(ctx context.Context) error {
 
 	return nil
 }
+
+// cleanupSuccessfulTestArtifacts removes artifacts from successful test runs to save disk space.
+// This function removes large artifacts that are mainly useful for debugging failures:
+// - Database dumps (.db files)
+// - Profile data (pprof directories)
+// - MapResponse data (mapresponses directories)
+// - Prometheus metrics files
+//
+// It preserves:
+// - Log files (.log) which are small and useful for verification.
+func cleanupSuccessfulTestArtifacts(logsDir string, verbose bool) error {
+	entries, err := os.ReadDir(logsDir)
+	if err != nil {
+		return fmt.Errorf("reading logs directory: %w", err)
+	}
+
+	var (
+		removedFiles, removedDirs int
+		totalSize                 int64
+	)
+
+	for _, entry := range entries {
+		name := entry.Name()
+		fullPath := filepath.Join(logsDir, name)
+
+		if entry.IsDir() {
+			// Remove pprof and mapresponses directories (typically large)
+			// These directories contain artifacts from all containers in the test run
+			if name == "pprof" || name == "mapresponses" {
+				size, sizeErr := getDirSize(fullPath)
+				if sizeErr == nil {
+					totalSize += size
+				}
+
+				err := os.RemoveAll(fullPath)
+				if err != nil {
+					if verbose {
+						log.Printf("Warning: failed to remove directory %s: %v", name, err)
+					}
+				} else {
+					removedDirs++
+
+					if verbose {
+						log.Printf("Removed directory: %s/", name)
+					}
+				}
+			}
+		} else {
+			// Only process test-related files (headscale and tailscale)
+			if !strings.HasPrefix(name, "hs-") && !strings.HasPrefix(name, "ts-") {
+				continue
+			}
+
+			// Remove database, metrics, and status files, but keep logs
+			shouldRemove := strings.HasSuffix(name, ".db") ||
+				strings.HasSuffix(name, "_metrics.txt") ||
+				strings.HasSuffix(name, "_status.json")
+
+			if shouldRemove {
+				info, infoErr := entry.Info()
+				if infoErr == nil {
+					totalSize += info.Size()
+				}
+
+				err := os.Remove(fullPath)
+				if err != nil {
+					if verbose {
+						log.Printf("Warning: failed to remove file %s: %v", name, err)
+					}
+				} else {
+					removedFiles++
+
+					if verbose {
+						log.Printf("Removed file: %s", name)
+					}
+				}
+			}
+		}
+	}
+
+	if removedFiles > 0 || removedDirs > 0 {
+		const bytesPerMB = 1024 * 1024
+		log.Printf("Cleaned up %d files and %d directories (freed ~%.2f MB)",
+			removedFiles, removedDirs, float64(totalSize)/bytesPerMB)
+	}
+
+	return nil
+}
+
+// getDirSize calculates the total size of a directory.
+func getDirSize(path string) (int64, error) {
+	var size int64
+
+	err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+
+		if !info.IsDir() {
+			size += info.Size()
+		}
+
+		return nil
+	})
+
+	return size, err
+}

cmd/hi/docker.go

@@ -22,17 +22,22 @@ import (
 	"github.com/juanfont/headscale/integration/dockertestutil"
 )
 
+const defaultDirPerm = 0o755
+
 var (
 	ErrTestFailed              = errors.New("test failed")
 	ErrUnexpectedContainerWait = errors.New("unexpected end of container wait")
 	ErrNoDockerContext         = errors.New("no docker context found")
+	ErrMemoryLimitViolations   = errors.New("container(s) exceeded memory limits")
 )
 
 // runTestContainer executes integration tests in a Docker container.
+//
+//nolint:gocyclo // complex test orchestration function
 func runTestContainer(ctx context.Context, config *RunConfig) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
@@ -48,19 +53,21 @@ func runTestContainer(ctx context.Context, config *RunConfig) error {
 
 	absLogsDir, err := filepath.Abs(logsDir)
 	if err != nil {
-		return fmt.Errorf("failed to get absolute path for logs directory: %w", err)
+		return fmt.Errorf("getting absolute path for logs directory: %w", err)
 	}
 
 	const dirPerm = 0o755
-	if err := os.MkdirAll(absLogsDir, dirPerm); err != nil {
-		return fmt.Errorf("failed to create logs directory: %w", err)
+	if err := os.MkdirAll(absLogsDir, dirPerm); err != nil { //nolint:noinlineerr
+		return fmt.Errorf("creating logs directory: %w", err)
 	}
 
 	if config.CleanBefore {
 		if config.Verbose {
 			log.Printf("Running pre-test cleanup...")
 		}
-		if err := cleanupBeforeTest(ctx); err != nil && config.Verbose {
+
+		err := cleanupBeforeTest(ctx)
+		if err != nil && config.Verbose {
 			log.Printf("Warning: pre-test cleanup failed: %v", err)
 		}
 	}
@@ -71,34 +78,40 @@ func runTestContainer(ctx context.Context, config *RunConfig) error {
 	}
 
 	imageName := "golang:" + config.GoVersion
-	if err := ensureImageAvailable(ctx, cli, imageName, config.Verbose); err != nil {
-		return fmt.Errorf("failed to ensure image availability: %w", err)
+	if err := ensureImageAvailable(ctx, cli, imageName, config.Verbose); err != nil { //nolint:noinlineerr
+		return fmt.Errorf("ensuring image availability: %w", err)
 	}
 
 	resp, err := createGoTestContainer(ctx, cli, config, containerName, absLogsDir, goTestCmd)
 	if err != nil {
-		return fmt.Errorf("failed to create container: %w", err)
+		return fmt.Errorf("creating container: %w", err)
 	}
 
 	if config.Verbose {
 		log.Printf("Created container: %s", resp.ID)
 	}
 
-	if err := cli.ContainerStart(ctx, resp.ID, container.StartOptions{}); err != nil {
-		return fmt.Errorf("failed to start container: %w", err)
+	if err := cli.ContainerStart(ctx, resp.ID, container.StartOptions{}); err != nil { //nolint:noinlineerr
+		return fmt.Errorf("starting container: %w", err)
 	}
 
 	log.Printf("Starting test: %s", config.TestPattern)
+	log.Printf("Run ID: %s", runID)
+	log.Printf("Monitor with: docker logs -f %s", containerName)
+	log.Printf("Logs directory: %s", logsDir)
 
 	// Start stats collection for container resource monitoring (if enabled)
 	var statsCollector *StatsCollector
+
 	if config.Stats {
 		var err error
-		statsCollector, err = NewStatsCollector()
+
+		statsCollector, err = NewStatsCollector(ctx)
 		if err != nil {
 			if config.Verbose {
 				log.Printf("Warning: failed to create stats collector: %v", err)
 			}
+
 			statsCollector = nil
 		}
 
@@ -107,7 +120,8 @@ func runTestContainer(ctx context.Context, config *RunConfig) error {
 
 			// Start stats collection immediately - no need for complex retry logic
 			// The new implementation monitors Docker events and will catch containers as they start
-			if err := statsCollector.StartCollection(ctx, runID, config.Verbose); err != nil {
+			err := statsCollector.StartCollection(ctx, runID, config.Verbose)
+			if err != nil {
 				if config.Verbose {
 					log.Printf("Warning: failed to start stats collection: %v", err)
 				}
@@ -119,12 +133,13 @@ func runTestContainer(ctx context.Context, config *RunConfig) error {
 	exitCode, err := streamAndWait(ctx, cli, resp.ID)
 
 	// Ensure all containers have finished and logs are flushed before extracting artifacts
-	if waitErr := waitForContainerFinalization(ctx, cli, resp.ID, config.Verbose); waitErr != nil && config.Verbose {
+	waitErr := waitForContainerFinalization(ctx, cli, resp.ID, config.Verbose)
+	if waitErr != nil && config.Verbose {
 		log.Printf("Warning: failed to wait for container finalization: %v", waitErr)
 	}
 
 	// Extract artifacts from test containers before cleanup
-	if err := extractArtifactsFromContainers(ctx, resp.ID, logsDir, config.Verbose); err != nil && config.Verbose {
+	if err := extractArtifactsFromContainers(ctx, resp.ID, logsDir, config.Verbose); err != nil && config.Verbose { //nolint:noinlineerr
 		log.Printf("Warning: failed to extract artifacts from containers: %v", err)
 	}
 
@@ -137,27 +152,44 @@ func runTestContainer(ctx context.Context, config *RunConfig) error {
 		if len(violations) > 0 {
 			log.Printf("MEMORY LIMIT VIOLATIONS DETECTED:")
 			log.Printf("=================================")
+
 			for _, violation := range violations {
 				log.Printf("Container %s exceeded memory limit: %.1f MB > %.1f MB",
 					violation.ContainerName, violation.MaxMemoryMB, violation.LimitMB)
 			}
 
-			return fmt.Errorf("test failed: %d container(s) exceeded memory limits", len(violations))
+			return fmt.Errorf("test failed: %d %w", len(violations), ErrMemoryLimitViolations)
 		}
 	}
 
 	shouldCleanup := config.CleanAfter && (!config.KeepOnFailure || exitCode == 0)
 	if shouldCleanup {
 		if config.Verbose {
-			log.Printf("Running post-test cleanup...")
+			log.Printf("Running post-test cleanup for run %s...", runID)
 		}
-		if cleanErr := cleanupAfterTest(ctx, cli, resp.ID); cleanErr != nil && config.Verbose {
+
+		cleanErr := cleanupAfterTest(ctx, cli, resp.ID, runID)
+
+		if cleanErr != nil && config.Verbose {
 			log.Printf("Warning: post-test cleanup failed: %v", cleanErr)
 		}
+
+		// Clean up artifacts from successful tests to save disk space in CI
+		if exitCode == 0 {
+			if config.Verbose {
+				log.Printf("Test succeeded, cleaning up artifacts to save disk space...")
+			}
+
+			cleanErr := cleanupSuccessfulTestArtifacts(logsDir, config.Verbose)
+
+			if cleanErr != nil && config.Verbose {
+				log.Printf("Warning: artifact cleanup failed: %v", cleanErr)
+			}
+		}
 	}
 
 	if err != nil {
-		return fmt.Errorf("test execution failed: %w", err)
+		return fmt.Errorf("executing test: %w", err)
 	}
 
 	if exitCode != 0 {
@@ -191,7 +223,7 @@ func buildGoTestCommand(config *RunConfig) []string {
 func createGoTestContainer(ctx context.Context, cli *client.Client, config *RunConfig, containerName, logsDir string, goTestCmd []string) (container.CreateResponse, error) {
 	pwd, err := os.Getwd()
 	if err != nil {
-		return container.CreateResponse{}, fmt.Errorf("failed to get working directory: %w", err)
+		return container.CreateResponse{}, fmt.Errorf("getting working directory: %w", err)
 	}
 
 	projectRoot := findProjectRoot(pwd)
@@ -202,6 +234,28 @@ func createGoTestContainer(ctx context.Context, cli *client.Client, config *RunC
 		fmt.Sprintf("HEADSCALE_INTEGRATION_POSTGRES=%d", boolToInt(config.UsePostgres)),
 		"HEADSCALE_INTEGRATION_RUN_ID=" + runID,
 	}
+
+	// Pass through CI environment variable for CI detection
+	if ci := os.Getenv("CI"); ci != "" {
+		env = append(env, "CI="+ci)
+	}
+
+	// Pass through all HEADSCALE_INTEGRATION_* environment variables
+	for _, e := range os.Environ() {
+		if strings.HasPrefix(e, "HEADSCALE_INTEGRATION_") {
+			// Skip the ones we already set explicitly
+			if strings.HasPrefix(e, "HEADSCALE_INTEGRATION_POSTGRES=") ||
+				strings.HasPrefix(e, "HEADSCALE_INTEGRATION_RUN_ID=") {
+				continue
+			}
+
+			env = append(env, e)
+		}
+	}
+
+	// Set GOCACHE to a known location (used by both bind mount and volume cases)
+	env = append(env, "GOCACHE=/cache/go-build")
+
 	containerConfig := &container.Config{
 		Image:      "golang:" + config.GoVersion,
 		Cmd:        goTestCmd,
@@ -221,20 +275,43 @@ func createGoTestContainer(ctx context.Context, cli *client.Client, config *RunC
 		log.Printf("Using Docker socket: %s", dockerSocketPath)
 	}
 
+	binds := []string{
+		fmt.Sprintf("%s:%s", projectRoot, projectRoot),
+		dockerSocketPath + ":/var/run/docker.sock",
+		logsDir + ":/tmp/control",
+	}
+
+	// Use bind mounts for Go cache if provided via environment variables,
+	// otherwise fall back to Docker volumes for local development
+	var mounts []mount.Mount
+
+	goCache := os.Getenv("HEADSCALE_INTEGRATION_GO_CACHE")
+	goBuildCache := os.Getenv("HEADSCALE_INTEGRATION_GO_BUILD_CACHE")
+
+	if goCache != "" {
+		binds = append(binds, goCache+":/go")
+	} else {
+		mounts = append(mounts, mount.Mount{
+			Type:   mount.TypeVolume,
+			Source: "hs-integration-go-cache",
+			Target: "/go",
+		})
+	}
+
+	if goBuildCache != "" {
+		binds = append(binds, goBuildCache+":/cache/go-build")
+	} else {
+		mounts = append(mounts, mount.Mount{
+			Type:   mount.TypeVolume,
+			Source: "hs-integration-go-build-cache",
+			Target: "/cache/go-build",
+		})
+	}
+
 	hostConfig := &container.HostConfig{
 		AutoRemove: false, // We'll remove manually for better control
-		Binds: []string{
-			fmt.Sprintf("%s:%s", projectRoot, projectRoot),
-			dockerSocketPath + ":/var/run/docker.sock",
-			logsDir + ":/tmp/control",
-		},
-		Mounts: []mount.Mount{
-			{
-				Type:   mount.TypeVolume,
-				Source: "hs-integration-go-cache",
-				Target: "/go",
-			},
-		},
+		Binds:      binds,
+		Mounts:     mounts,
 	}
 
 	return cli.ContainerCreate(ctx, containerConfig, hostConfig, nil, nil, containerName)
@@ -248,7 +325,7 @@ func streamAndWait(ctx context.Context, cli *client.Client, containerID string)
 		Follow:     true,
 	})
 	if err != nil {
-		return -1, fmt.Errorf("failed to get container logs: %w", err)
+		return -1, fmt.Errorf("getting container logs: %w", err)
 	}
 	defer out.Close()
 
@@ -260,7 +337,7 @@ func streamAndWait(ctx context.Context, cli *client.Client, containerID string)
 	select {
 	case err := <-errCh:
 		if err != nil {
-			return -1, fmt.Errorf("error waiting for container: %w", err)
+			return -1, fmt.Errorf("waiting for container: %w", err)
 		}
 	case status := <-statusCh:
 		return int(status.StatusCode), nil
@@ -274,7 +351,7 @@ func waitForContainerFinalization(ctx context.Context, cli *client.Client, testC
 	// First, get all related test containers
 	containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})
 	if err != nil {
-		return fmt.Errorf("failed to list containers: %w", err)
+		return fmt.Errorf("listing containers: %w", err)
 	}
 
 	testContainers := getCurrentTestContainers(containers, testContainerID, verbose)
@@ -283,6 +360,7 @@ func waitForContainerFinalization(ctx context.Context, cli *client.Client, testC
 	maxWaitTime := 10 * time.Second
 	checkInterval := 500 * time.Millisecond
 	timeout := time.After(maxWaitTime)
+
 	ticker := time.NewTicker(checkInterval)
 	defer ticker.Stop()
 
@@ -292,6 +370,7 @@ func waitForContainerFinalization(ctx context.Context, cli *client.Client, testC
 			if verbose {
 				log.Printf("Timeout waiting for container finalization, proceeding with artifact extraction")
 			}
+
 			return nil
 		case <-ticker.C:
 			allFinalized := true
@@ -302,12 +381,14 @@ func waitForContainerFinalization(ctx context.Context, cli *client.Client, testC
 					if verbose {
 						log.Printf("Warning: failed to inspect container %s: %v", testCont.name, err)
 					}
+
 					continue
 				}
 
 				// Check if container is in a final state
 				if !isContainerFinalized(inspect.State) {
 					allFinalized = false
+
 					if verbose {
 						log.Printf("Container %s still finalizing (state: %s)", testCont.name, inspect.State.Status)
 					}
@@ -320,6 +401,7 @@ func waitForContainerFinalization(ctx context.Context, cli *client.Client, testC
 				if verbose {
 					log.Printf("All test containers finalized, ready for artifact extraction")
 				}
+
 				return nil
 			}
 		}
@@ -336,13 +418,15 @@ func isContainerFinalized(state *container.State) bool {
 func findProjectRoot(startPath string) string {
 	current := startPath
 	for {
-		if _, err := os.Stat(filepath.Join(current, "go.mod")); err == nil {
+		if _, err := os.Stat(filepath.Join(current, "go.mod")); err == nil { //nolint:noinlineerr
 			return current
 		}
+
 		parent := filepath.Dir(current)
 		if parent == current {
 			return startPath
 		}
+
 		current = parent
 	}
 }
@@ -352,34 +436,37 @@ func boolToInt(b bool) int {
 	if b {
 		return 1
 	}
+
 	return 0
 }
 
 // DockerContext represents Docker context information.
 type DockerContext struct {
-	Name      string                 `json:"Name"`
-	Metadata  map[string]interface{} `json:"Metadata"`
-	Endpoints map[string]interface{} `json:"Endpoints"`
-	Current   bool                   `json:"Current"`
+	Name      string         `json:"Name"`
+	Metadata  map[string]any `json:"Metadata"`
+	Endpoints map[string]any `json:"Endpoints"`
+	Current   bool           `json:"Current"`
 }
 
 // createDockerClient creates a Docker client with context detection.
-func createDockerClient() (*client.Client, error) {
-	contextInfo, err := getCurrentDockerContext()
+func createDockerClient(ctx context.Context) (*client.Client, error) {
+	contextInfo, err := getCurrentDockerContext(ctx)
 	if err != nil {
 		return client.NewClientWithOpts(client.FromEnv, client.WithAPIVersionNegotiation())
 	}
 
 	var clientOpts []client.Opt
+
 	clientOpts = append(clientOpts, client.WithAPIVersionNegotiation())
 
 	if contextInfo != nil {
 		if endpoints, ok := contextInfo.Endpoints["docker"]; ok {
-			if endpointMap, ok := endpoints.(map[string]interface{}); ok {
+			if endpointMap, ok := endpoints.(map[string]any); ok {
 				if host, ok := endpointMap["Host"].(string); ok {
 					if runConfig.Verbose {
 						log.Printf("Using Docker host from context '%s': %s", contextInfo.Name, host)
 					}
+
 					clientOpts = append(clientOpts, client.WithHost(host))
 				}
 			}
@@ -394,16 +481,17 @@ func createDockerClient() (*client.Client, error) {
 }
 
 // getCurrentDockerContext retrieves the current Docker context information.
-func getCurrentDockerContext() (*DockerContext, error) {
-	cmd := exec.Command("docker", "context", "inspect")
+func getCurrentDockerContext(ctx context.Context) (*DockerContext, error) {
+	cmd := exec.CommandContext(ctx, "docker", "context", "inspect")
+
 	output, err := cmd.Output()
 	if err != nil {
-		return nil, fmt.Errorf("failed to get docker context: %w", err)
+		return nil, fmt.Errorf("getting docker context: %w", err)
 	}
 
 	var contexts []DockerContext
-	if err := json.Unmarshal(output, &contexts); err != nil {
-		return nil, fmt.Errorf("failed to parse docker context: %w", err)
+	if err := json.Unmarshal(output, &contexts); err != nil { //nolint:noinlineerr
+		return nil, fmt.Errorf("parsing docker context: %w", err)
 	}
 
 	if len(contexts) > 0 {
@@ -422,12 +510,13 @@ func getDockerSocketPath() string {
 
 // checkImageAvailableLocally checks if the specified Docker image is available locally.
 func checkImageAvailableLocally(ctx context.Context, cli *client.Client, imageName string) (bool, error) {
-	_, _, err := cli.ImageInspectWithRaw(ctx, imageName)
+	_, _, err := cli.ImageInspectWithRaw(ctx, imageName) //nolint:staticcheck // SA1019: deprecated but functional
 	if err != nil {
-		if client.IsErrNotFound(err) {
+		if client.IsErrNotFound(err) { //nolint:staticcheck // SA1019: deprecated but functional
 			return false, nil
 		}
-		return false, fmt.Errorf("failed to inspect image %s: %w", imageName, err)
+
+		return false, fmt.Errorf("inspecting image %s: %w", imageName, err)
 	}
 
 	return true, nil
@@ -438,13 +527,14 @@ func ensureImageAvailable(ctx context.Context, cli *client.Client, imageName str
 	// First check if image is available locally
 	available, err := checkImageAvailableLocally(ctx, cli, imageName)
 	if err != nil {
-		return fmt.Errorf("failed to check local image availability: %w", err)
+		return fmt.Errorf("checking local image availability: %w", err)
 	}
 
 	if available {
 		if verbose {
 			log.Printf("Image %s is available locally", imageName)
 		}
+
 		return nil
 	}
 
@@ -455,20 +545,21 @@ func ensureImageAvailable(ctx context.Context, cli *client.Client, imageName str
 
 	reader, err := cli.ImagePull(ctx, imageName, image.PullOptions{})
 	if err != nil {
-		return fmt.Errorf("failed to pull image %s: %w", imageName, err)
+		return fmt.Errorf("pulling image %s: %w", imageName, err)
 	}
 	defer reader.Close()
 
 	if verbose {
 		_, err = io.Copy(os.Stdout, reader)
 		if err != nil {
-			return fmt.Errorf("failed to read pull output: %w", err)
+			return fmt.Errorf("reading pull output: %w", err)
 		}
 	} else {
 		_, err = io.Copy(io.Discard, reader)
 		if err != nil {
-			return fmt.Errorf("failed to read pull output: %w", err)
+			return fmt.Errorf("reading pull output: %w", err)
 		}
+
 		log.Printf("Image %s pulled successfully", imageName)
 	}
 
@@ -483,9 +574,11 @@ func listControlFiles(logsDir string) {
 		return
 	}
 
-	var logFiles []string
-	var dataFiles []string
-	var dataDirs []string
+	var (
+		logFiles  []string
+		dataFiles []string
+		dataDirs  []string
+	)
 
 	for _, entry := range entries {
 		name := entry.Name()
@@ -514,6 +607,7 @@ func listControlFiles(logsDir string) {
 
 	if len(logFiles) > 0 {
 		log.Printf("Headscale logs:")
+
 		for _, file := range logFiles {
 			log.Printf("  %s", file)
 		}
@@ -521,9 +615,11 @@ func listControlFiles(logsDir string) {
 
 	if len(dataFiles) > 0 || len(dataDirs) > 0 {
 		log.Printf("Headscale data:")
+
 		for _, file := range dataFiles {
 			log.Printf("  %s", file)
 		}
+
 		for _, dir := range dataDirs {
 			log.Printf("  %s/", dir)
 		}
@@ -532,25 +628,27 @@ func listControlFiles(logsDir string) {
 
 // extractArtifactsFromContainers collects container logs and files from the specific test run.
 func extractArtifactsFromContainers(ctx context.Context, testContainerID, logsDir string, verbose bool) error {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return fmt.Errorf("failed to create Docker client: %w", err)
+		return fmt.Errorf("creating Docker client: %w", err)
 	}
 	defer cli.Close()
 
 	// List all containers
 	containers, err := cli.ContainerList(ctx, container.ListOptions{All: true})
 	if err != nil {
-		return fmt.Errorf("failed to list containers: %w", err)
+		return fmt.Errorf("listing containers: %w", err)
 	}
 
 	// Get containers from the specific test run
 	currentTestContainers := getCurrentTestContainers(containers, testContainerID, verbose)
 
 	extractedCount := 0
+
 	for _, cont := range currentTestContainers {
 		// Extract container logs and tar files
-		if err := extractContainerArtifacts(ctx, cli, cont.ID, cont.name, logsDir, verbose); err != nil {
+		err := extractContainerArtifacts(ctx, cli, cont.ID, cont.name, logsDir, verbose)
+		if err != nil {
 			if verbose {
 				log.Printf("Warning: failed to extract artifacts from container %s (%s): %v", cont.name, cont.ID[:12], err)
 			}
@@ -558,6 +656,7 @@ func extractArtifactsFromContainers(ctx context.Context, testContainerID, logsDi
 			if verbose {
 				log.Printf("Extracted artifacts from container %s (%s)", cont.name, cont.ID[:12])
 			}
+
 			extractedCount++
 		}
 	}
@@ -581,11 +680,13 @@ func getCurrentTestContainers(containers []container.Summary, testContainerID st
 
 	// Find the test container to get its run ID label
 	var runID string
+
 	for _, cont := range containers {
 		if cont.ID == testContainerID {
 			if cont.Labels != nil {
 				runID = cont.Labels["hi.run-id"]
 			}
+
 			break
 		}
 	}
@@ -626,18 +727,21 @@ func getCurrentTestContainers(containers []container.Summary, testContainerID st
 // extractContainerArtifacts saves logs and tar files from a container.
 func extractContainerArtifacts(ctx context.Context, cli *client.Client, containerID, containerName, logsDir string, verbose bool) error {
 	// Ensure the logs directory exists
-	if err := os.MkdirAll(logsDir, 0o755); err != nil {
-		return fmt.Errorf("failed to create logs directory: %w", err)
+	err := os.MkdirAll(logsDir, defaultDirPerm)
+	if err != nil {
+		return fmt.Errorf("creating logs directory: %w", err)
 	}
 
 	// Extract container logs
-	if err := extractContainerLogs(ctx, cli, containerID, containerName, logsDir, verbose); err != nil {
-		return fmt.Errorf("failed to extract logs: %w", err)
+	err = extractContainerLogs(ctx, cli, containerID, containerName, logsDir, verbose)
+	if err != nil {
+		return fmt.Errorf("extracting logs: %w", err)
 	}
 
 	// Extract tar files for headscale containers only
 	if strings.HasPrefix(containerName, "hs-") {
-		if err := extractContainerFiles(ctx, cli, containerID, containerName, logsDir, verbose); err != nil {
+		err := extractContainerFiles(ctx, cli, containerID, containerName, logsDir, verbose)
+		if err != nil {
 			if verbose {
 				log.Printf("Warning: failed to extract files from %s: %v", containerName, err)
 			}
@@ -659,7 +763,7 @@ func extractContainerLogs(ctx context.Context, cli *client.Client, containerID,
 		Tail:       "all",
 	})
 	if err != nil {
-		return fmt.Errorf("failed to get container logs: %w", err)
+		return fmt.Errorf("getting container logs: %w", err)
 	}
 	defer logReader.Close()
 
@@ -673,17 +777,17 @@ func extractContainerLogs(ctx context.Context, cli *client.Client, containerID,
 	// Demultiplex the Docker logs stream to separate stdout and stderr
 	_, err = stdcopy.StdCopy(&stdoutBuf, &stderrBuf, logReader)
 	if err != nil {
-		return fmt.Errorf("failed to demultiplex container logs: %w", err)
+		return fmt.Errorf("demultiplexing container logs: %w", err)
 	}
 
 	// Write stdout logs
-	if err := os.WriteFile(stdoutPath, stdoutBuf.Bytes(), 0o644); err != nil {
-		return fmt.Errorf("failed to write stdout log: %w", err)
+	if err := os.WriteFile(stdoutPath, stdoutBuf.Bytes(), 0o644); err != nil { //nolint:gosec,noinlineerr // log files should be readable
+		return fmt.Errorf("writing stdout log: %w", err)
 	}
 
 	// Write stderr logs
-	if err := os.WriteFile(stderrPath, stderrBuf.Bytes(), 0o644); err != nil {
-		return fmt.Errorf("failed to write stderr log: %w", err)
+	if err := os.WriteFile(stderrPath, stderrBuf.Bytes(), 0o644); err != nil { //nolint:gosec,noinlineerr // log files should be readable
+		return fmt.Errorf("writing stderr log: %w", err)
 	}
 
 	if verbose {
@@ -701,63 +805,3 @@ func extractContainerFiles(ctx context.Context, cli *client.Client, containerID,
 	// This function is kept for potential future use or other file types
 	return nil
 }
-
-// logExtractionError logs extraction errors with appropriate level based on error type.
-func logExtractionError(artifactType, containerName string, err error, verbose bool) {
-	if errors.Is(err, ErrFileNotFoundInTar) {
-		// File not found is expected and only logged in verbose mode
-		if verbose {
-			log.Printf("No %s found in container %s", artifactType, containerName)
-		}
-	} else {
-		// Other errors are actual failures and should be logged as warnings
-		log.Printf("Warning: failed to extract %s from %s: %v", artifactType, containerName, err)
-	}
-}
-
-// extractSingleFile copies a single file from a container.
-func extractSingleFile(ctx context.Context, cli *client.Client, containerID, sourcePath, fileName, logsDir string, verbose bool) error {
-	tarReader, _, err := cli.CopyFromContainer(ctx, containerID, sourcePath)
-	if err != nil {
-		return fmt.Errorf("failed to copy %s from container: %w", sourcePath, err)
-	}
-	defer tarReader.Close()
-
-	// Extract the single file from the tar
-	filePath := filepath.Join(logsDir, fileName)
-	if err := extractFileFromTar(tarReader, filepath.Base(sourcePath), filePath); err != nil {
-		return fmt.Errorf("failed to extract file from tar: %w", err)
-	}
-
-	if verbose {
-		log.Printf("Extracted %s from %s", fileName, containerID[:12])
-	}
-
-	return nil
-}
-
-// extractDirectory copies a directory from a container and extracts its contents.
-func extractDirectory(ctx context.Context, cli *client.Client, containerID, sourcePath, dirName, logsDir string, verbose bool) error {
-	tarReader, _, err := cli.CopyFromContainer(ctx, containerID, sourcePath)
-	if err != nil {
-		return fmt.Errorf("failed to copy %s from container: %w", sourcePath, err)
-	}
-	defer tarReader.Close()
-
-	// Create target directory
-	targetDir := filepath.Join(logsDir, dirName)
-	if err := os.MkdirAll(targetDir, 0o755); err != nil {
-		return fmt.Errorf("failed to create directory %s: %w", targetDir, err)
-	}
-
-	// Extract the directory from the tar
-	if err := extractDirectoryFromTar(tarReader, targetDir); err != nil {
-		return fmt.Errorf("failed to extract directory from tar: %w", err)
-	}
-
-	if verbose {
-		log.Printf("Extracted %s/ from %s", dirName, containerID[:12])
-	}
-
-	return nil
-}

cmd/hi/doctor.go

@@ -38,13 +38,13 @@ func runDoctorCheck(ctx context.Context) error {
 	}
 
 	// Check 3: Go installation
-	results = append(results, checkGoInstallation())
+	results = append(results, checkGoInstallation(ctx))
 
 	// Check 4: Git repository
-	results = append(results, checkGitRepository())
+	results = append(results, checkGitRepository(ctx))
 
 	// Check 5: Required files
-	results = append(results, checkRequiredFiles())
+	results = append(results, checkRequiredFiles(ctx))
 
 	// Display results
 	displayDoctorResults(results)
@@ -86,7 +86,7 @@ func checkDockerBinary() DoctorResult {
 
 // checkDockerDaemon verifies Docker daemon is running and accessible.
 func checkDockerDaemon(ctx context.Context) DoctorResult {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
 		return DoctorResult{
 			Name:    "Docker Daemon",
@@ -124,8 +124,8 @@ func checkDockerDaemon(ctx context.Context) DoctorResult {
 }
 
 // checkDockerContext verifies Docker context configuration.
-func checkDockerContext(_ context.Context) DoctorResult {
-	contextInfo, err := getCurrentDockerContext()
+func checkDockerContext(ctx context.Context) DoctorResult {
+	contextInfo, err := getCurrentDockerContext(ctx)
 	if err != nil {
 		return DoctorResult{
 			Name:    "Docker Context",
@@ -155,7 +155,7 @@ func checkDockerContext(_ context.Context) DoctorResult {
 
 // checkDockerSocket verifies Docker socket accessibility.
 func checkDockerSocket(ctx context.Context) DoctorResult {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
 		return DoctorResult{
 			Name:    "Docker Socket",
@@ -192,7 +192,7 @@ func checkDockerSocket(ctx context.Context) DoctorResult {
 
 // checkGolangImage verifies the golang Docker image is available locally or can be pulled.
 func checkGolangImage(ctx context.Context) DoctorResult {
-	cli, err := createDockerClient()
+	cli, err := createDockerClient(ctx)
 	if err != nil {
 		return DoctorResult{
 			Name:    "Golang Image",
@@ -251,7 +251,7 @@ func checkGolangImage(ctx context.Context) DoctorResult {
 }
 
 // checkGoInstallation verifies Go is installed and working.
-func checkGoInstallation() DoctorResult {
+func checkGoInstallation(ctx context.Context) DoctorResult {
 	_, err := exec.LookPath("go")
 	if err != nil {
 		return DoctorResult{
@@ -265,7 +265,8 @@ func checkGoInstallation() DoctorResult {
 		}
 	}
 
-	cmd := exec.Command("go", "version")
+	cmd := exec.CommandContext(ctx, "go", "version")
+
 	output, err := cmd.Output()
 	if err != nil {
 		return DoctorResult{
@@ -285,8 +286,9 @@ func checkGoInstallation() DoctorResult {
 }
 
 // checkGitRepository verifies we're in a git repository.
-func checkGitRepository() DoctorResult {
-	cmd := exec.Command("git", "rev-parse", "--git-dir")
+func checkGitRepository(ctx context.Context) DoctorResult {
+	cmd := exec.CommandContext(ctx, "git", "rev-parse", "--git-dir")
+
 	err := cmd.Run()
 	if err != nil {
 		return DoctorResult{
@@ -308,7 +310,7 @@ func checkGitRepository() DoctorResult {
 }
 
 // checkRequiredFiles verifies required files exist.
-func checkRequiredFiles() DoctorResult {
+func checkRequiredFiles(ctx context.Context) DoctorResult {
 	requiredFiles := []string{
 		"go.mod",
 		"integration/",
@@ -316,9 +318,12 @@ func checkRequiredFiles() DoctorResult {
 	}
 
 	var missingFiles []string
+
 	for _, file := range requiredFiles {
-		cmd := exec.Command("test", "-e", file)
-		if err := cmd.Run(); err != nil {
+		cmd := exec.CommandContext(ctx, "test", "-e", file)
+
+		err := cmd.Run()
+		if err != nil {
 			missingFiles = append(missingFiles, file)
 		}
 	}
@@ -350,6 +355,7 @@ func displayDoctorResults(results []DoctorResult) {
 
 	for _, result := range results {
 		var icon string
+
 		switch result.Status {
 		case "PASS":
 			icon = "✅"

cmd/hi/main.go

@@ -79,13 +79,18 @@ func main() {
 }
 
 func cleanAll(ctx context.Context) error {
-	if err := killTestContainers(ctx); err != nil {
+	err := killTestContainers(ctx)
+	if err != nil {
 		return err
 	}
-	if err := pruneDockerNetworks(ctx); err != nil {
+
+	err = pruneDockerNetworks(ctx)
+	if err != nil {
 		return err
 	}
-	if err := cleanOldImages(ctx); err != nil {
+
+	err = cleanOldImages(ctx)
+	if err != nil {
 		return err
 	}
 

cmd/hi/run.go

@@ -19,7 +19,7 @@ type RunConfig struct {
 	FailFast      bool          `flag:"failfast,default=true,Stop on first test failure"`
 	UsePostgres   bool          `flag:"postgres,default=false,Use PostgreSQL instead of SQLite"`
 	GoVersion     string        `flag:"go-version,Go version to use (auto-detected from go.mod)"`
-	CleanBefore   bool          `flag:"clean-before,default=true,Clean resources before test"`
+	CleanBefore   bool          `flag:"clean-before,default=true,Clean stale resources before test"`
 	CleanAfter    bool          `flag:"clean-after,default=true,Clean resources after test"`
 	KeepOnFailure bool          `flag:"keep-on-failure,default=false,Keep containers on test failure"`
 	LogsDir       string        `flag:"logs-dir,default=control_logs,Control logs directory"`
@@ -48,7 +48,9 @@ func runIntegrationTest(env *command.Env) error {
 	if runConfig.Verbose {
 		log.Printf("Running pre-flight system checks...")
 	}
-	if err := runDoctorCheck(env.Context()); err != nil {
+
+	err := runDoctorCheck(env.Context())
+	if err != nil {
 		return fmt.Errorf("pre-flight checks failed: %w", err)
 	}
 
@@ -66,15 +68,15 @@ func runIntegrationTest(env *command.Env) error {
 func detectGoVersion() string {
 	goModPath := filepath.Join("..", "..", "go.mod")
 
-	if _, err := os.Stat("go.mod"); err == nil {
+	if _, err := os.Stat("go.mod"); err == nil { //nolint:noinlineerr
 		goModPath = "go.mod"
-	} else if _, err := os.Stat("../../go.mod"); err == nil {
+	} else if _, err := os.Stat("../../go.mod"); err == nil { //nolint:noinlineerr
 		goModPath = "../../go.mod"
 	}
 
 	content, err := os.ReadFile(goModPath)
 	if err != nil {
-		return "1.25"
+		return "1.26.1"
 	}
 
 	lines := splitLines(string(content))
@@ -89,13 +91,15 @@ func detectGoVersion() string {
 		}
 	}
 
-	return "1.25"
+	return "1.26.1"
 }
 
 // splitLines splits a string into lines without using strings.Split.
 func splitLines(s string) []string {
-	var lines []string
-	var current string
+	var (
+		lines   []string
+		current string
+	)
 
 	for _, char := range s {
 		if char == '\n' {

cmd/hi/stats.go

@@ -18,6 +18,9 @@ import (
 	"github.com/docker/docker/client"
 )
 
+// ErrStatsCollectionAlreadyStarted is returned when trying to start stats collection that is already running.
+var ErrStatsCollectionAlreadyStarted = errors.New("stats collection already started")
+
 // ContainerStats represents statistics for a single container.
 type ContainerStats struct {
 	ContainerID   string
@@ -44,10 +47,10 @@ type StatsCollector struct {
 }
 
 // NewStatsCollector creates a new stats collector instance.
-func NewStatsCollector() (*StatsCollector, error) {
-	cli, err := createDockerClient()
+func NewStatsCollector(ctx context.Context) (*StatsCollector, error) {
+	cli, err := createDockerClient(ctx)
 	if err != nil {
-		return nil, fmt.Errorf("failed to create Docker client: %w", err)
+		return nil, fmt.Errorf("creating Docker client: %w", err)
 	}
 
 	return &StatsCollector{
@@ -63,17 +66,19 @@ func (sc *StatsCollector) StartCollection(ctx context.Context, runID string, ver
 	defer sc.mutex.Unlock()
 
 	if sc.collectionStarted {
-		return errors.New("stats collection already started")
+		return ErrStatsCollectionAlreadyStarted
 	}
 
 	sc.collectionStarted = true
 
 	// Start monitoring existing containers
 	sc.wg.Add(1)
+
 	go sc.monitorExistingContainers(ctx, runID, verbose)
 
 	// Start Docker events monitoring for new containers
 	sc.wg.Add(1)
+
 	go sc.monitorDockerEvents(ctx, runID, verbose)
 
 	if verbose {
@@ -87,10 +92,12 @@ func (sc *StatsCollector) StartCollection(ctx context.Context, runID string, ver
 func (sc *StatsCollector) StopCollection() {
 	// Check if already stopped without holding lock
 	sc.mutex.RLock()
+
 	if !sc.collectionStarted {
 		sc.mutex.RUnlock()
 		return
 	}
+
 	sc.mutex.RUnlock()
 
 	// Signal stop to all goroutines
@@ -114,6 +121,7 @@ func (sc *StatsCollector) monitorExistingContainers(ctx context.Context, runID s
 		if verbose {
 			log.Printf("Failed to list existing containers: %v", err)
 		}
+
 		return
 	}
 
@@ -147,13 +155,13 @@ func (sc *StatsCollector) monitorDockerEvents(ctx context.Context, runID string,
 		case event := <-events:
 			if event.Type == "container" && event.Action == "start" {
 				// Get container details
-				containerInfo, err := sc.client.ContainerInspect(ctx, event.ID)
+				containerInfo, err := sc.client.ContainerInspect(ctx, event.ID) //nolint:staticcheck // SA1019: use Actor.ID
 				if err != nil {
 					continue
 				}
 
 				// Convert to types.Container format for consistency
-				cont := types.Container{
+				cont := types.Container{ //nolint:staticcheck // SA1019: use container.Summary
 					ID:     containerInfo.ID,
 					Names:  []string{containerInfo.Name},
 					Labels: containerInfo.Config.Labels,
@@ -167,13 +175,14 @@ func (sc *StatsCollector) monitorDockerEvents(ctx context.Context, runID string,
 			if verbose {
 				log.Printf("Error in Docker events stream: %v", err)
 			}
+
 			return
 		}
 	}
 }
 
 // shouldMonitorContainer determines if a container should be monitored.
-func (sc *StatsCollector) shouldMonitorContainer(cont types.Container, runID string) bool {
+func (sc *StatsCollector) shouldMonitorContainer(cont types.Container, runID string) bool { //nolint:staticcheck // SA1019: use container.Summary
 	// Check if it has the correct run ID label
 	if cont.Labels == nil || cont.Labels["hi.run-id"] != runID {
 		return false
@@ -213,6 +222,7 @@ func (sc *StatsCollector) startStatsForContainer(ctx context.Context, containerI
 	}
 
 	sc.wg.Add(1)
+
 	go sc.collectStatsForContainer(ctx, containerID, verbose)
 }
 
@@ -226,12 +236,14 @@ func (sc *StatsCollector) collectStatsForContainer(ctx context.Context, containe
 		if verbose {
 			log.Printf("Failed to get stats stream for container %s: %v", containerID[:12], err)
 		}
+
 		return
 	}
 	defer statsResponse.Body.Close()
 
 	decoder := json.NewDecoder(statsResponse.Body)
-	var prevStats *container.Stats
+
+	var prevStats *container.Stats //nolint:staticcheck // SA1019: use StatsResponse
 
 	for {
 		select {
@@ -240,12 +252,15 @@ func (sc *StatsCollector) collectStatsForContainer(ctx context.Context, containe
 		case <-ctx.Done():
 			return
 		default:
-			var stats container.Stats
-			if err := decoder.Decode(&stats); err != nil {
+			var stats container.Stats //nolint:staticcheck // SA1019: use StatsResponse
+
+			err := decoder.Decode(&stats)
+			if err != nil {
 				// EOF is expected when container stops or stream ends
 				if err.Error() != "EOF" && verbose {
 					log.Printf("Failed to decode stats for container %s: %v", containerID[:12], err)
 				}
+
 				return
 			}
 
@@ -261,8 +276,10 @@ func (sc *StatsCollector) collectStatsForContainer(ctx context.Context, containe
 			// Store the sample (skip first sample since CPU calculation needs previous stats)
 			if prevStats != nil {
 				// Get container stats reference without holding the main mutex
-				var containerStats *ContainerStats
-				var exists bool
+				var (
+					containerStats *ContainerStats
+					exists         bool
+				)
 
 				sc.mutex.RLock()
 				containerStats, exists = sc.containers[containerID]
@@ -286,7 +303,7 @@ func (sc *StatsCollector) collectStatsForContainer(ctx context.Context, containe
 }
 
 // calculateCPUPercent calculates CPU usage percentage from Docker stats.
-func calculateCPUPercent(prevStats, stats *container.Stats) float64 {
+func calculateCPUPercent(prevStats, stats *container.Stats) float64 { //nolint:staticcheck // SA1019: use StatsResponse
 	// CPU calculation based on Docker's implementation
 	cpuDelta := float64(stats.CPUStats.CPUUsage.TotalUsage) - float64(prevStats.CPUStats.CPUUsage.TotalUsage)
 	systemDelta := float64(stats.CPUStats.SystemUsage) - float64(prevStats.CPUStats.SystemUsage)
@@ -331,10 +348,12 @@ type StatsSummary struct {
 func (sc *StatsCollector) GetSummary() []ContainerStatsSummary {
 	// Take snapshot of container references without holding main lock long
 	sc.mutex.RLock()
+
 	containerRefs := make([]*ContainerStats, 0, len(sc.containers))
 	for _, containerStats := range sc.containers {
 		containerRefs = append(containerRefs, containerStats)
 	}
+
 	sc.mutex.RUnlock()
 
 	summaries := make([]ContainerStatsSummary, 0, len(containerRefs))
@@ -384,23 +403,25 @@ func calculateStatsSummary(values []float64) StatsSummary {
 		return StatsSummary{}
 	}
 
-	min := values[0]
-	max := values[0]
+	minVal := values[0]
+	maxVal := values[0]
 	sum := 0.0
 
 	for _, value := range values {
-		if value < min {
-			min = value
+		if value < minVal {
+			minVal = value
 		}
-		if value > max {
-			max = value
+
+		if value > maxVal {
+			maxVal = value
 		}
+
 		sum += value
 	}
 
 	return StatsSummary{
-		Min:     min,
-		Max:     max,
+		Min:     minVal,
+		Max:     maxVal,
 		Average: sum / float64(len(values)),
 	}
 }
@@ -434,6 +455,7 @@ func (sc *StatsCollector) CheckMemoryLimits(hsLimitMB, tsLimitMB float64) []Memo
 	}
 
 	summaries := sc.GetSummary()
+
 	var violations []MemoryViolation
 
 	for _, summary := range summaries {

cmd/hi/tar_utils.go

@@ -1,105 +0,0 @@
-package main
-
-import (
-	"archive/tar"
-	"errors"
-	"fmt"
-	"io"
-	"os"
-	"path/filepath"
-	"strings"
-)
-
-// ErrFileNotFoundInTar indicates a file was not found in the tar archive.
-var ErrFileNotFoundInTar = errors.New("file not found in tar")
-
-// extractFileFromTar extracts a single file from a tar reader.
-func extractFileFromTar(tarReader io.Reader, fileName, outputPath string) error {
-	tr := tar.NewReader(tarReader)
-
-	for {
-		header, err := tr.Next()
-		if err == io.EOF {
-			break
-		}
-		if err != nil {
-			return fmt.Errorf("failed to read tar header: %w", err)
-		}
-
-		// Check if this is the file we're looking for
-		if filepath.Base(header.Name) == fileName {
-			if header.Typeflag == tar.TypeReg {
-				// Create the output file
-				outFile, err := os.Create(outputPath)
-				if err != nil {
-					return fmt.Errorf("failed to create output file: %w", err)
-				}
-				defer outFile.Close()
-
-				// Copy file contents
-				if _, err := io.Copy(outFile, tr); err != nil {
-					return fmt.Errorf("failed to copy file contents: %w", err)
-				}
-
-				return nil
-			}
-		}
-	}
-
-	return fmt.Errorf("%w: %s", ErrFileNotFoundInTar, fileName)
-}
-
-// extractDirectoryFromTar extracts all files from a tar reader to a target directory.
-func extractDirectoryFromTar(tarReader io.Reader, targetDir string) error {
-	tr := tar.NewReader(tarReader)
-
-	for {
-		header, err := tr.Next()
-		if err == io.EOF {
-			break
-		}
-		if err != nil {
-			return fmt.Errorf("failed to read tar header: %w", err)
-		}
-
-		// Clean the path to prevent directory traversal
-		cleanName := filepath.Clean(header.Name)
-		if strings.Contains(cleanName, "..") {
-			continue // Skip potentially dangerous paths
-		}
-
-		targetPath := filepath.Join(targetDir, cleanName)
-
-		switch header.Typeflag {
-		case tar.TypeDir:
-			// Create directory
-			if err := os.MkdirAll(targetPath, os.FileMode(header.Mode)); err != nil {
-				return fmt.Errorf("failed to create directory %s: %w", targetPath, err)
-			}
-		case tar.TypeReg:
-			// Ensure parent directories exist
-			if err := os.MkdirAll(filepath.Dir(targetPath), 0o755); err != nil {
-				return fmt.Errorf("failed to create parent directories for %s: %w", targetPath, err)
-			}
-
-			// Create file
-			outFile, err := os.Create(targetPath)
-			if err != nil {
-				return fmt.Errorf("failed to create file %s: %w", targetPath, err)
-			}
-
-			if _, err := io.Copy(outFile, tr); err != nil {
-				outFile.Close()
-				return fmt.Errorf("failed to copy file contents: %w", err)
-			}
-			outFile.Close()
-
-			// Set file permissions
-			if err := os.Chmod(targetPath, os.FileMode(header.Mode)); err != nil {
-				return fmt.Errorf("failed to set file permissions: %w", err)
-			}
-		}
-	}
-
-	return nil
-}

cmd/mapresponses/main.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"encoding/json"
+	"errors"
 	"fmt"
 	"os"
 
@@ -15,7 +16,10 @@ type MapConfig struct {
 	Directory string `flag:"directory,Directory to read map responses from"`
 }
 
-var mapConfig MapConfig
+var (
+	mapConfig            MapConfig
+	errDirectoryRequired = errors.New("directory is required")
+)
 
 func main() {
 	root := command.C{
@@ -40,7 +44,7 @@ func main() {
 // runIntegrationTest executes the integration test workflow.
 func runOnline(env *command.Env) error {
 	if mapConfig.Directory == "" {
-		return fmt.Errorf("directory is required")
+		return errDirectoryRequired
 	}
 
 	resps, err := mapper.ReadMapResponsesFromDirectory(mapConfig.Directory)
@@ -57,5 +61,6 @@ func runOnline(env *command.Env) error {
 
 	os.Stderr.Write(out)
 	os.Stderr.Write([]byte("\n"))
+
 	return nil
 }

config-example.yaml

@@ -20,6 +20,7 @@ listen_addr: 127.0.0.1:8080
 
 # Address to listen to /metrics and /debug, you may want
 # to keep this endpoint private to your internal network
+# Use an empty value to disable the metrics listener.
 metrics_listen_addr: 127.0.0.1:9090
 
 # Address to listen for gRPC.
@@ -49,12 +50,21 @@ noise:
 # List of IP prefixes to allocate tailaddresses from.
 # Each prefix consists of either an IPv4 or IPv6 address,
 # and the associated prefix length, delimited by a slash.
-# It must be within IP ranges supported by the Tailscale
-# client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48.
-# See below:
-# IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
+#
+# WARNING: These prefixes MUST be subsets of the standard Tailscale ranges:
+#   - IPv4: 100.64.0.0/10 (CGNAT range)
+#   - IPv6: fd7a:115c:a1e0::/48 (Tailscale ULA range)
+#
+# Using a SUBSET of these ranges is supported and useful if you want to
+# limit IP allocation to a smaller block (e.g., 100.64.0.0/24).
+#
+# Using ranges OUTSIDE of CGNAT/ULA is NOT supported and will cause
+# undefined behaviour. The Tailscale client has hard-coded assumptions
+# about these ranges and will break in subtle, hard-to-debug ways.
+#
+# See:
 # IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33
-# Any other range is NOT supported, and it will cause unexpected issues.
+# IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
 prefixes:
   v4: 100.64.0.0/10
   v6: fd7a:115c:a1e0::/48
@@ -118,7 +128,7 @@ derp:
   #
   # This option is mostly interesting for people hosting
   # their own DERP servers:
-  # https://tailscale.com/kb/1118/custom-derp-servers/
+  # https://tailscale.com/docs/reference/derp-servers/custom-derp-servers
   #
   # paths:
   #   - /etc/headscale/derp-example.yaml
@@ -135,8 +145,45 @@ derp:
 # Disables the automatic check for headscale updates on startup
 disable_check_updates: false
 
-# Time before an inactive ephemeral node is deleted?
-ephemeral_node_inactivity_timeout: 30m
+# Node lifecycle configuration.
+node:
+  # Default key expiry for non-tagged nodes, regardless of registration method
+  # (auth key, CLI, web auth). Tagged nodes are exempt and never expire.
+  #
+  # This is the base default. OIDC can override this via oidc.expiry.
+  # If a client explicitly requests a specific expiry, the client value is used.
+  #
+  # Setting the value to "0" means no default expiry (nodes never expire unless
+  # explicitly expired via `headscale nodes expire`).
+  #
+  # Tailscale SaaS uses 180d; set to a positive duration to match that behaviour.
+  #
+  # Default: 0 (no default expiry)
+  expiry: 0
+
+  ephemeral:
+    # Time before an inactive ephemeral node is deleted.
+    inactivity_timeout: 30m
+
+  # HA subnet router health probing.
+  #
+  # When HA routes exist (2+ nodes advertising the same prefix), headscale
+  # pings each HA node every probe_interval via the Noise channel. If a node
+  # fails to respond within probe_timeout it is marked unhealthy and the
+  # primary role moves to the next healthy node. A node that later responds
+  # is marked healthy again but does NOT reclaim primary (avoids flapping).
+  #
+  # Worst-case detection time is probe_interval + probe_timeout (15s default).
+  # No-op when no HA routes exist. Set probe_interval to 0 to disable.
+  routes:
+    ha:
+      # How often to ping HA subnet routers. Set to 0 to disable probing.
+      # Must be >= 2s when enabled.
+      probe_interval: 10s
+
+      # How long to wait for a ping response before marking a node unhealthy.
+      # Must be >= 1s and less than probe_interval.
+      probe_timeout: 5s
 
 database:
   # Database type. Available options: sqlite, postgres
@@ -236,7 +283,7 @@ log:
 ## Policy
 # headscale supports Tailscale's ACL policies.
 # Please have a look to their KB to better
-# understand the concepts: https://tailscale.com/kb/1018/acls/
+# understand the concepts: https://tailscale.com/docs/features/access-control/acls
 policy:
   # The mode can be "file" or "database" that defines
   # where the ACL policies are stored and read from.
@@ -250,9 +297,9 @@ policy:
 # headscale supports Tailscale's DNS configuration and MagicDNS.
 # Please have a look to their KB to better understand the concepts:
 #
-# - https://tailscale.com/kb/1054/dns/
-# - https://tailscale.com/kb/1081/magicdns/
-# - https://tailscale.com/blog/2021-09-private-dns-with-magicdns/
+# - https://tailscale.com/docs/features/access-control/acls
+# - https://tailscale.com/docs/features/magicdns
+# - https://tailscale.com/blog/2021-09-private-dns-with-magicdns
 #
 # Please note that for the DNS configuration to have any effect,
 # clients must have the `--accept-dns=true` option enabled. This is the
@@ -262,12 +309,12 @@ policy:
 # Setting _any_ of the configuration and `--accept-dns=true` on the
 # clients will integrate with the DNS manager on the client or
 # overwrite /etc/resolv.conf.
-# https://tailscale.com/kb/1235/resolv-conf
+# https://tailscale.com/docs/reference/faq/dns-resolv-conf
 #
 # If you want stop Headscale from managing the DNS configuration
 # all the fields under `dns` should be set to empty values.
 dns:
-  # Whether to use [MagicDNS](https://tailscale.com/kb/1081/magicdns/).
+  # Whether to use [MagicDNS](https://tailscale.com/docs/features/magicdns).
   magic_dns: true
 
   # Defines the base domain to create the hostnames for MagicDNS.
@@ -289,11 +336,11 @@ dns:
       - 2606:4700:4700::1111
       - 2606:4700:4700::1001
 
-      # NextDNS (see https://tailscale.com/kb/1218/nextdns/).
+      # NextDNS (see https://tailscale.com/docs/integrations/nextdns).
       # "abc123" is example NextDNS ID, replace with yours.
       # - https://dns.nextdns.io/abc123
 
-    # Split DNS (see https://tailscale.com/kb/1054/dns/),
+    # Split DNS (see https://tailscale.com/docs/reference/dns-in-tailscale#restricted-nameservers),
     # a map of domains and which DNS server to use for each.
     split: {}
       # foo.bar.com:
@@ -345,15 +392,11 @@ unix_socket_permission: "0770"
 #   # `LoadCredential` straightforward:
 #   client_secret_path: "${CREDENTIALS_DIRECTORY}/oidc_client_secret"
 #
-#   # The amount of time a node is authenticated with OpenID until it expires
-#   # and needs to reauthenticate.
-#   # Setting the value to "0" will mean no expiry.
-#   expiry: 180d
-#
 #   # Use the expiry from the token received from OpenID when the user logged
 #   # in. This will typically lead to frequent need to reauthenticate and should
 #   # only be enabled if you know what you are doing.
-#   # Note: enabling this will cause `oidc.expiry` to be ignored.
+#   # Note: enabling this will cause `node.expiry` to be ignored for
+#   # OIDC-authenticated nodes.
 #   use_expiry_from_token: false
 #
 #   # The OIDC scopes to use, defaults to "openid", "profile" and "email".
@@ -361,6 +404,12 @@ unix_socket_permission: "0770"
 #   # required "openid" scope.
 #   scope: ["openid", "profile", "email"]
 #
+#   # Only verified email addresses are synchronized to the user profile by
+#   # default. Unverified emails may be allowed in case an identity provider
+#   # does not send the "email_verified: true" claim or email verification is
+#   # not required.
+#   email_verified_required: true
+#
 #   # Provide custom key/value pairs which get sent to the identity provider's
 #   # authorization endpoint.
 #   extra_params:
@@ -396,7 +445,7 @@ unix_socket_permission: "0770"
 # Logtail is Tailscales logging and auditing infrastructure, it allows the
 # control panel to instruct tailscale nodes to log their activity to a remote
 # server. To disable logging on the client side, please refer to:
-# https://tailscale.com/kb/1011/log-mesh-traffic#opting-out-of-client-logging
+# https://tailscale.com/docs/features/logging#opt-out-of-client-logging
 logtail:
   # Enable logtail for tailscale nodes of this Headscale instance.
   # As there is currently no support for overriding the log server in Headscale, this is
@@ -405,5 +454,30 @@ logtail:
 
 # Enabling this option makes devices prefer a random port for WireGuard traffic over the
 # default static port 41641. This option is intended as a workaround for some buggy
-# firewall devices. See https://tailscale.com/kb/1181/firewalls/ for more information.
+# firewall devices. See https://tailscale.com/docs/integrations/firewalls for more information.
 randomize_client_port: false
+
+# Taildrop configuration
+# Taildrop is the file sharing feature of Tailscale, allowing nodes to send files to each other.
+# https://tailscale.com/docs/features/taildrop
+taildrop:
+  # Enable or disable Taildrop for all nodes.
+  # When enabled, nodes can send files to other nodes owned by the same user.
+  # Tagged devices and cross-user transfers are not permitted by Tailscale clients.
+  enabled: true
+# Advanced performance tuning parameters.
+# The defaults are carefully chosen and should rarely need adjustment.
+# Only modify these if you have identified a specific performance issue.
+#
+# tuning:
+#   # Maximum number of pending registration entries in the auth cache.
+#   # Oldest entries are evicted when the cap is reached.
+#   #
+#   # register_cache_max_entries: 1024
+#
+#   # NodeStore write batching configuration.
+#   # The NodeStore batches write operations before rebuilding peer relationships,
+#   # which is computationally expensive. Batching reduces rebuild frequency.
+#   #
+#   # node_store_batch_size: 100
+#   # node_store_batch_timeout: 500ms

derp-example.yaml

@@ -1,6 +1,7 @@
-# If you plan to somehow use headscale, please deploy your own DERP infra: https://tailscale.com/kb/1118/custom-derp-servers/
+# If you plan to somehow use headscale, please deploy your own DERP infra.
+# See: https://tailscale.com/docs/reference/derp-servers/custom-derp-servers
 regions:
-  1: null   # Disable DERP region with ID 1
+  1: null # Disable DERP region with ID 1
   900:
     regionid: 900
     regioncode: custom

docs/about/contributing.md

@@ -1,3 +1,3 @@
 {%
-    include-markdown "../../CONTRIBUTING.md"
+include-markdown "../../CONTRIBUTING.md"
 %}

docs/about/faq.md

@@ -24,9 +24,12 @@ We are more than happy to exchange emails, or to have dedicated calls before a P
 
 ## When/Why is Feature X going to be implemented?
 
-We don't know. We might be working on it. If you're interested in contributing, please post a feature request about it.
+We use [GitHub Milestones to plan for upcoming Headscale releases](https://github.com/juanfont/headscale/milestones).
+Have a look at [our current plan](https://github.com/juanfont/headscale/milestones) to get an idea when a specific
+feature is about to be implemented. The release plan is subject to change at any time.
 
-Please be aware that there are a number of reasons why we might not accept specific contributions:
+If you're interested in contributing, please post a feature request about it. Please be aware that there are a number of
+reasons why we might not accept specific contributions:
 
 - It is not possible to implement the feature in a way that makes sense in a self-hosted environment.
 - Given that we are reverse-engineering Tailscale to satisfy our own curiosity, we might be interested in implementing the feature ourselves.
@@ -47,8 +50,8 @@ we have a "docker-issues" channel where you can ask for Docker-specific help to
 ## What is the recommended update path? Can I skip multiple versions while updating?
 
 Please follow the steps outlined in the [upgrade guide](../setup/upgrade.md) to update your existing Headscale
-installation. Its best to update from one stable version to the next (e.g. 0.24.0 → 0.25.1 → 0.26.1) in case
-you are multiple releases behind. You should always pick the latest available patch release.
+installation. Its required to update from one stable version to the next (e.g. 0.26.0 → 0.27.1 → 0.28.0) without
+skipping minor versions in between. You should always pick the latest available patch release.
 
 Be sure to check the [changelog](https://github.com/juanfont/headscale/blob/main/CHANGELOG.md) for version specific
 upgrade instructions and breaking changes.
@@ -70,12 +73,12 @@ of Headscale:
 
 1. An environment with 1000 servers
 
-   - they rarely "move" (change their endpoints)
-   - new nodes are added rarely
+    - they rarely "move" (change their endpoints)
+    - new nodes are added rarely
 
-2. An environment with 80 laptops/phones (end user devices)
+1. An environment with 80 laptops/phones (end user devices)
 
-   - nodes move often, e.g. switching from home to office
+    - nodes move often, e.g. switching from home to office
 
 Headscale calculates a map of all nodes that need to talk to each other,
 creating this "world map" requires a lot of CPU time. When an event that
@@ -139,10 +142,10 @@ connect back to the administrator's node. Why do all nodes see the administrator
 `tailscale status`?
 
 This is essentially how Tailscale works. If traffic is allowed to flow in one direction, then both nodes see each other
-in their output of `tailscale status`. Traffic is still filtered according to the ACL, with the exception of `tailscale
-ping` which is always allowed in either direction.
+in their output of `tailscale status`. Traffic is still filtered according to the ACL, with the exception of
+`tailscale ping` which is always allowed in either direction.
 
-See also <https://tailscale.com/kb/1087/device-visibility>.
+See also <https://tailscale.com/docs/concepts/device-visibility>.
 
 ## My policy is stored in the database and Headscale refuses to start due to an invalid policy. How can I recover?
 
@@ -157,13 +160,46 @@ indicates which part of the policy is invalid. Follow these steps to fix your po
 !!! warning "Full server configuration required"
 
     The above commands to get/set the policy require a complete server configuration file including database settings. A
-    minimal config to [control Headscale via remote CLI](../ref/remote-cli.md) is not sufficient. You may use `headscale
-    -c /path/to/config.yaml` to specify the path to an alternative configuration file.
+    minimal config to [control Headscale via remote CLI](../ref/api.md#grpc) is not sufficient. You may use
+    `headscale -c /path/to/config.yaml` to specify the path to an alternative configuration file.
+
+## How can I migrate back to the recommended IP prefixes?
+
+Tailscale only supports the IP prefixes `100.64.0.0/10` and `fd7a:115c:a1e0::/48` or smaller subnets thereof. The
+following steps can be used to migrate from unsupported IP prefixes back to the supported and recommended ones.
+
+!!! warning "Backup and test in a demo environment required"
+
+    The commands below update the IP addresses of all nodes in your tailnet and this might have a severe impact in your
+    specific environment. At a minimum:
+
+    - [Create a backup of your database](../setup/upgrade.md#backup)
+    - Test the commands below in a representive demo environment. This allows to catch subsequent connectivity errors
+      early and see how the tailnet behaves in your specific environment.
+
+- Stop Headscale
+- Restore the default prefixes in the [configuration file](../ref/configuration.md):
+    ```yaml
+    prefixes:
+      v4: 100.64.0.0/10
+      v6: fd7a:115c:a1e0::/48
+    ```
+- Update the `nodes.ipv4` and `nodes.ipv6` columns in the database and assign each node a unique IPv4 and IPv6 address.
+  The following SQL statement assigns IP addresses based on the node ID:
+    ```sql
+    UPDATE nodes
+    SET ipv4=concat('100.64.', id/256, '.', id%256),
+        ipv6=concat('fd7a:115c:a1e0::', format('%x', id));
+    ```
+- Update the [policy](../ref/acls.md) to reflect the IP address changes (if any)
+- Start Headscale
+
+Nodes should reconnect within a few seconds and pickup their newly assigned IP addresses.
 
 ## How can I avoid to send logs to Tailscale Inc?
 
 A Tailscale client [collects logs about its operation and connection attempts with other
-clients](https://tailscale.com/kb/1011/log-mesh-traffic#client-logs) and sends them to a central log service operated by
+clients](https://tailscale.com/docs/features/logging#client-logs) and sends them to a central log service operated by
 Tailscale Inc.
 
 Headscale, by default, instructs clients to disable log submission to the central log service. This configuration is
@@ -173,5 +209,5 @@ applied by a client once it successfully connected with Headscale. See the confi
 Alternatively, logging can also be disabled on the client side. This is independent of Headscale and opting out of
 client logging disables log submission early during client startup. The configuration is operating system specific and
 is usually achieved by setting the environment variable `TS_NO_LOGS_NO_SUPPORT=true` or by passing the flag
-`--no-logs-no-support` to `tailscaled`. See
-<https://tailscale.com/kb/1011/log-mesh-traffic#opting-out-of-client-logging> for details.
+`--no-logs-no-support` to `tailscaled`. See <https://tailscale.com/docs/features/logging#opt-out-of-client-logging> for
+details.

docs/about/features.md

@@ -5,15 +5,16 @@ to provide self-hosters and hobbyists with an open-source server they can use fo
 provides on overview of Headscale's feature and compatibility with the Tailscale control server:
 
 - [x] Full "base" support of Tailscale's features
-- [x] Node registration
-    - [x] Interactive
-    - [x] Pre authenticated key
+- [x] [Node registration](../ref/registration.md)
+    - [x] [Web authentication](../ref/registration.md#web-authentication)
+    - [x] [Pre authenticated key](../ref/registration.md#pre-authenticated-key)
 - [x] [DNS](../ref/dns.md)
-    - [x] [MagicDNS](https://tailscale.com/kb/1081/magicdns)
-    - [x] [Global and restricted nameservers (split DNS)](https://tailscale.com/kb/1054/dns#nameservers)
-    - [x] [search domains](https://tailscale.com/kb/1054/dns#search-domains)
+    - [x] [MagicDNS](https://tailscale.com/docs/features/magicdns)
+    - [x] [Global and restricted nameservers (split DNS)](https://tailscale.com/docs/reference/dns-in-tailscale#nameservers)
+    - [x] [search domains](https://tailscale.com/docs/reference/dns-in-tailscale#search-domains)
     - [x] [Extra DNS records (Headscale only)](../ref/dns.md#setting-extra-dns-records)
-- [x] [Taildrop (File Sharing)](https://tailscale.com/kb/1106/taildrop)
+- [x] [Taildrop (File Sharing)](https://tailscale.com/docs/features/taildrop)
+- [x] [Tags](../ref/tags.md)
 - [x] [Routes](../ref/routes.md)
     - [x] [Subnet routers](../ref/routes.md#subnet-router)
     - [x] [Exit nodes](../ref/routes.md#exit-node)
@@ -22,16 +23,17 @@ provides on overview of Headscale's feature and compatibility with the Tailscale
 - [x] Embedded [DERP server](../ref/derp.md)
 - [x] Access control lists ([GitHub label "policy"](https://github.com/juanfont/headscale/labels/policy%20%F0%9F%93%9D))
     - [x] ACL management via API
-    - [x] Some [Autogroups](https://tailscale.com/kb/1396/targets#autogroups), currently: `autogroup:internet`,
-      `autogroup:nonroot`, `autogroup:member`, `autogroup:tagged`, `autogroup:self`
-    - [x] [Auto approvers](https://tailscale.com/kb/1337/acl-syntax#auto-approvers) for [subnet
+    - [x] Some [Autogroups](https://tailscale.com/docs/reference/targets-and-selectors#autogroups), currently:
+      `autogroup:internet`, `autogroup:nonroot`, `autogroup:member`, `autogroup:tagged`, `autogroup:self`,
+      `autogroup:danger-all`
+    - [x] [Auto approvers](https://tailscale.com/docs/reference/syntax/policy-file#auto-approvers) for [subnet
       routers](../ref/routes.md#automatically-approve-routes-of-a-subnet-router) and [exit
       nodes](../ref/routes.md#automatically-approve-an-exit-node-with-auto-approvers)
-    - [x] [Tailscale SSH](https://tailscale.com/kb/1193/tailscale-ssh)
-* [x] [Node registration using Single-Sign-On (OpenID Connect)](../ref/oidc.md) ([GitHub label "OIDC"](https://github.com/juanfont/headscale/labels/OIDC))
+    - [x] [Tailscale SSH](https://tailscale.com/docs/features/tailscale-ssh)
+- [x] [Node registration using Single-Sign-On (OpenID Connect)](../ref/oidc.md) ([GitHub label "OIDC"](https://github.com/juanfont/headscale/labels/OIDC))
     - [x] Basic registration
     - [x] Update user profile from identity provider
     - [ ] OIDC groups cannot be used in ACLs
-- [ ] [Funnel](https://tailscale.com/kb/1223/funnel) ([#1040](https://github.com/juanfont/headscale/issues/1040))
-- [ ] [Serve](https://tailscale.com/kb/1312/serve) ([#1234](https://github.com/juanfont/headscale/issues/1921))
-- [ ] [Network flow logs](https://tailscale.com/kb/1219/network-flow-logs) ([#1687](https://github.com/juanfont/headscale/issues/1687))
+- [ ] [Funnel](https://tailscale.com/docs/features/tailscale-funnel) ([#1040](https://github.com/juanfont/headscale/issues/1040))
+- [ ] [Serve](https://tailscale.com/docs/features/tailscale-serve) ([#1234](https://github.com/juanfont/headscale/issues/1921))
+- [ ] [Network flow logs](https://tailscale.com/docs/features/logging/network-flow-logs) ([#1687](https://github.com/juanfont/headscale/issues/1687))

docs/assets/logo/headscale3-dots.svg

@@ -1 +1 @@
-<svg xmlns="http://www.w3.org/2000/svg" xml:space="preserve" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:2" viewBox="0 0 1280 640"><circle cx="141.023" cy="338.36" r="117.472" style="fill:#f8b5cb" transform="matrix(.997276 0 0 1.00556 10.0024 -14.823)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 -3.15847 0)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 -3.15847 115.914)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 148.43 115.914)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 148.851 0)"/><circle cx="805.557" cy="336.915" r="118.199" style="fill:#8d8d8d" transform="matrix(.99196 0 0 1 3.36978 -10.2458)"/><circle cx="805.557" cy="336.915" r="118.199" style="fill:#8d8d8d" transform="matrix(.99196 0 0 1 255.633 -10.2458)"/><path d="M680.282 124.808h-68.093v390.325h68.081v-28.23H640V153.228h40.282v-28.42Z" style="fill:#303030"/><path d="M680.282 124.808h-68.093v390.325h68.081v-28.23H640V153.228h40.282v-28.42Z" style="fill:#303030" transform="matrix(-1 0 0 1 1857.19 0)"/></svg>
+<svg xmlns="http://www.w3.org/2000/svg" xml:space="preserve" style="fill-rule:evenodd;clip-rule:evenodd;stroke-linejoin:round;stroke-miterlimit:2" viewBox="0 0 1280 640"><circle cx="141.023" cy="338.36" r="117.472" style="fill:#f8b5cb" transform="matrix(.997276 0 0 1.00556 10.0024 -14.823)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 -3.15847 0)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 -3.15847 115.914)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 148.43 115.914)"/><circle cx="352.014" cy="268.302" r="33.095" style="fill:#a2a2a2" transform="matrix(1.01749 0 0 1 148.851 0)"/><circle cx="805.557" cy="336.915" r="118.199" style="fill:#8d8d8d" transform="matrix(.99196 0 0 1 3.36978 -10.2458)"/><circle cx="805.557" cy="336.915" r="118.199" style="fill:#8d8d8d" transform="matrix(.99196 0 0 1 255.633 -10.2458)"/><path d="M680.282 124.808h-68.093v390.325h68.081v-28.23H640V153.228h40.282v-28.42Z" style="fill:#303030"/><path d="M680.282 124.808h-68.093v390.325h68.081v-28.23H640V153.228h40.282v-28.42Z" style="fill:#303030" transform="matrix(-1 0 0 1 1857.19 0)"/></svg>

docs/ref/acls.md

@@ -3,7 +3,8 @@ Headscale implements the same policy ACLs as Tailscale.com, adapted to the self-
 For instance, instead of referring to users when defining groups you must
 use users (which are the equivalent to user/logins in Tailscale.com).
 
-Please check https://tailscale.com/kb/1018/acls/ for further information.
+Please check [manage permissions using ACLs](https://tailscale.com/docs/features/access-control/acls) for further
+information.
 
 When using ACL's the User borders are no longer applied. All machines
 whichever the User have the ability to communicate with other hosts as
@@ -15,8 +16,8 @@ To enable and configure ACLs in Headscale, you need to specify the path to your
 
 Your ACL policy file must be formatted using [huJSON](https://github.com/tailscale/hujson).
 
-Info on how these policies are written can be found
-[here](https://tailscale.com/kb/1018/acls/).
+Info on how these policies are written can be found in [Tailscale's ACL
+documentation](https://tailscale.com/docs/features/access-control/acls).
 
 Please reload or restart Headscale after updating the ACL file. Headscale may be reloaded either via its systemd service
 (`sudo systemctl reload headscale`) or by sending a SIGHUP signal (`sudo kill -HUP $(pidof headscale)`) to the main
@@ -24,13 +25,13 @@ process. Headscale logs the result of ACL policy processing after each reload.
 
 ## Simple Examples
 
-- [**Allow All**](https://tailscale.com/kb/1192/acl-samples#allow-all-default-acl): If you define an ACL file but completely omit the `"acls"` field from its content, Headscale will default to an "allow all" policy. This means all devices connected to your tailnet will be able to communicate freely with each other.
+- [**Allow All**](https://tailscale.com/docs/reference/examples/acls#allow-all-default-acl): If you define an ACL file but completely omit the `"acls"` field from its content, Headscale will default to an "allow all" policy. This means all devices connected to your tailnet will be able to communicate freely with each other.
 
     ```json
     {}
     ```
 
-- [**Deny All**](https://tailscale.com/kb/1192/acl-samples#deny-all): To prevent all communication within your tailnet, you can include an empty array for the `"acls"` field in your policy file.
+- [**Deny All**](https://tailscale.com/docs/reference/examples/acls#deny-all): To prevent all communication within your tailnet, you can include an empty array for the `"acls"` field in your policy file.
 
     ```json
     {
@@ -65,7 +66,7 @@ servers.
 - billing.internal
 - router.internal
 
-![ACL implementation example](../images/headscale-acl-network.png)
+![ACL implementation example](../assets/images/headscale-acl-network.png)
 
 When [registering the servers](../usage/getting-started.md#register-a-node) we
 will need to add the flag `--advertise-tags=tag:<tag1>,tag:<tag2>`, and the user
@@ -87,7 +88,7 @@ Here are the ACL's to implement the same permissions as above:
     "group:intern": ["intern1@"]
   },
   // tagOwners in tailscale is an association between a TAG and the people allowed to set this TAG on a server.
-  // This is documented [here](https://tailscale.com/kb/1068/acl-tags#defining-a-tag)
+  // This is documented [here](https://tailscale.com/docs/features/tags)
   // and explained [here](https://tailscale.com/blog/rbac-like-it-was-meant-to-be/)
   "tagOwners": {
     // the administrators can add servers in production
@@ -222,7 +223,7 @@ Allows access to the internet through [exit nodes](routes.md#exit-node). Can onl
 
 ### `autogroup:member`
 
-Includes all users who are direct members of the tailnet. Does not include users from shared devices.
+Includes all [personal (untagged) devices](registration.md/#identity-model).
 
 ```json
 {
@@ -234,7 +235,7 @@ Includes all users who are direct members of the tailnet. Does not include users
 
 ### `autogroup:tagged`
 
-Includes all devices that have at least one tag.
+Includes all devices that [have at least one tag](registration.md/#identity-model).
 
 ```json
 {
@@ -245,7 +246,6 @@ Includes all devices that have at least one tag.
 ```
 
 ### `autogroup:self`
-**(EXPERIMENTAL)**
 
 !!! warning "The current implementation of `autogroup:self` is inefficient"
 
@@ -258,9 +258,11 @@ Includes devices where the same user is authenticated on both the source and des
   "dst": ["autogroup:self:*"]
 }
 ```
+
 *Using `autogroup:self` may cause performance degradation on the Headscale coordinator server in large deployments, as filter rules must be compiled per-node rather than globally and the current implementation is not very efficient.*
 
 If you experience performance issues, consider using more specific ACL rules or limiting the use of `autogroup:self`.
+
 ```json
 {
   // The following rules allow internal users to communicate with their
@@ -285,3 +287,9 @@ Used in Tailscale SSH rules to allow access to any user except root. Can only be
   "users": ["autogroup:nonroot"]
 }
 ```
+
+### `autogroup:danger-all`
+
+This autogroup resolves to all IP addresses (`0.0.0.0/0` and `::/0`) which also includes all IP addresses outside the
+standard Tailscale IP ranges. [This autogroup can only be used as
+source](https://tailscale.com/docs/reference/targets-and-selectors#autogroupdanger-all).

docs/ref/api.md

@@ -0,0 +1,129 @@
+# API
+
+Headscale provides a [HTTP REST API](#rest-api) and a [gRPC interface](#grpc) which may be used to integrate a [web
+interface](integration/web-ui.md), [remote control Headscale](#setup-remote-control) or provide a base for custom
+integration and tooling.
+
+Both interfaces require a valid API key before use. To create an API key, log into your Headscale server and generate
+one with the default expiration of 90 days:
+
+```shell
+headscale apikeys create
+```
+
+Copy the output of the command and save it for later. Please note that you can not retrieve an API key again. If the API
+key is lost, expire the old one, and create a new one.
+
+To list the API keys currently associated with the server:
+
+```shell
+headscale apikeys list
+```
+
+and to expire an API key:
+
+```shell
+headscale apikeys expire --prefix <PREFIX>
+```
+
+## REST API
+
+- API endpoint: `/api/v1`, e.g. `https://headscale.example.com/api/v1`
+- Documentation: `/swagger`, e.g. `https://headscale.example.com/swagger`
+- Headscale Version: `/version`, e.g. `https://headscale.example.com/version`
+- Authenticate using HTTP Bearer authentication by sending the [API key](#api) with the HTTP `Authorization: Bearer <API_KEY>` header.
+
+Start by [creating an API key](#api) and test it with the examples below. Read the API documentation provided by your
+Headscale server at `/swagger` for details.
+
+=== "Get details for all users"
+
+    ```console
+    curl -H "Authorization: Bearer <API_KEY>" \
+        https://headscale.example.com/api/v1/user
+    ```
+
+=== "Get details for user 'bob'"
+
+    ```console
+    curl -H "Authorization: Bearer <API_KEY>" \
+        https://headscale.example.com/api/v1/user?name=bob
+    ```
+
+=== "Register a node"
+
+    ```console
+    curl -H "Authorization: Bearer <API_KEY>" \
+        --json '{"user": "<USER>", "authId": "AUTH_ID>"}' \
+        https://headscale.example.com/api/v1/auth/register
+    ```
+
+## gRPC
+
+The gRPC interface can be used to control a Headscale instance from a remote machine with the `headscale` binary.
+
+### Prerequisite
+
+- A workstation to run `headscale` (any supported platform, e.g. Linux).
+- A Headscale server with gRPC enabled.
+- Connections to the gRPC port (default: `50443`) are allowed.
+- Remote access requires an encrypted connection via TLS.
+- An [API key](#api) to authenticate with the Headscale server.
+
+### Setup remote control
+
+1. Download the [`headscale` binary from GitHub's release page](https://github.com/juanfont/headscale/releases). Make
+   sure to use the same version as on the server.
+
+1. Put the binary somewhere in your `PATH`, e.g. `/usr/local/bin/headscale`
+
+1. Make `headscale` executable: `chmod +x /usr/local/bin/headscale`
+
+1. [Create an API key](#api) on the Headscale server.
+
+1. Provide the connection parameters for the remote Headscale server either via a minimal YAML configuration file or
+   via environment variables:
+
+    === "Minimal YAML configuration file"
+
+        ```yaml title="config.yaml"
+        cli:
+            address: <HEADSCALE_ADDRESS>:<PORT>
+            api_key: <API_KEY>
+        ```
+
+    === "Environment variables"
+
+        ```shell
+        export HEADSCALE_CLI_ADDRESS="<HEADSCALE_ADDRESS>:<PORT>"
+        export HEADSCALE_CLI_API_KEY="<API_KEY>"
+        ```
+
+    This instructs the `headscale` binary to connect to a remote instance at `<HEADSCALE_ADDRESS>:<PORT>`, instead of
+    connecting to the local instance.
+
+1. Test the connection by listing all nodes:
+
+    ```shell
+    headscale nodes list
+    ```
+
+    You should now be able to see a list of your nodes from your workstation, and you can
+    now control the Headscale server from your workstation.
+
+### Behind a proxy
+
+It's possible to run the gRPC remote endpoint behind a reverse proxy, like Nginx, and have it run on the _same_ port as Headscale.
+
+While this is _not a supported_ feature, an example on how this can be set up on
+[NixOS is shown here](https://github.com/kradalby/dotfiles/blob/4489cdbb19cddfbfae82cd70448a38fde5a76711/machines/headscale.oracldn/headscale.nix#L61-L91).
+
+### Troubleshooting
+
+- Make sure you have the _same_ Headscale version on your server and workstation.
+- Ensure that connections to the gRPC port are allowed.
+- Verify that your TLS certificate is valid and trusted.
+- If you don't have access to a trusted certificate (e.g. from Let's Encrypt), either:
+    - Add your self-signed certificate to the trust store of your OS _or_
+    - Disable certificate verification by either setting `cli.insecure: true` in the configuration file or by setting
+      `HEADSCALE_CLI_INSECURE=1` via an environment variable. We do **not** recommend to disable certificate validation.

docs/ref/configuration.md

@@ -17,8 +17,8 @@
 
     === "View on GitHub"
 
-        * Development version: <https://github.com/juanfont/headscale/blob/main/config-example.yaml>
-        * Version {{ headscale.version }}: <https://github.com/juanfont/headscale/blob/v{{ headscale.version }}/config-example.yaml>
+        - Development version: <https://github.com/juanfont/headscale/blob/main/config-example.yaml>
+        - Version {{ headscale.version }}: https://github.com/juanfont/headscale/blob/v{{ headscale.version }}/config-example.yaml
 
     === "Download with `wget`"
 

docs/ref/debug.md

@@ -3,16 +3,16 @@
 Headscale and Tailscale provide debug and introspection capabilities that can be helpful when things don't work as
 expected. This page explains some debugging techniques to help pinpoint problems.
 
-Please also have a look at [Tailscale's Troubleshooting guide](https://tailscale.com/kb/1023/troubleshooting). It offers
-a many tips and suggestions to troubleshoot common issues.
+Please also have a look at [Tailscale's Troubleshooting guide](https://tailscale.com/docs/reference/troubleshooting). It
+offers a many tips and suggestions to troubleshoot common issues.
 
 ## Tailscale
 
 The Tailscale client itself offers many commands to introspect its state as well as the state of the network:
 
-- [Check local network conditions](https://tailscale.com/kb/1080/cli#netcheck): `tailscale netcheck`
-- [Get the client status](https://tailscale.com/kb/1080/cli#status): `tailscale status --json`
-- [Get DNS status](https://tailscale.com/kb/1080/cli#dns): `tailscale dns status --all`
+- [Check local network conditions](https://tailscale.com/docs/reference/tailscale-cli#netcheck): `tailscale netcheck`
+- [Get the client status](https://tailscale.com/docs/reference/tailscale-cli#status): `tailscale status --json`
+- [Get DNS status](https://tailscale.com/docs/reference/tailscale-cli#dns): `tailscale dns status --all`
 - Client logs: `tailscale debug daemon-logs`
 - Client netmap: `tailscale debug netmap`
 - Test DERP connection: `tailscale debug derp headscale`
@@ -64,6 +64,9 @@ Headscale provides a metrics and debug endpoint. It allows to introspect differe
 
     Keep the metrics and debug endpoint private to your internal network and don't expose it to the Internet.
 
+    The metrics and debug interface can be disabled completely by setting `metrics_listen_addr: null` in the
+    [configuration file](./configuration.md).
+
 Query metrics via <http://localhost:9090/metrics> and get an overview of available debug information via
 <http://localhost:9090/debug/>. Metrics may be queried from outside localhost but the debug interface is subject to
 additional protection despite listening on all interfaces.

docs/ref/derp.md

@@ -1,8 +1,8 @@
 # DERP
 
-A [DERP (Designated Encrypted Relay for Packets) server](https://tailscale.com/kb/1232/derp-servers) is mainly used to
-relay traffic between two nodes in case a direct connection can't be established. Headscale provides an embedded DERP
-server to ensure seamless connectivity between nodes.
+A [DERP (Designated Encrypted Relay for Packets) server](https://tailscale.com/docs/reference/derp-servers) is mainly
+used to relay traffic between two nodes in case a direct connection can't be established. Headscale provides an embedded
+DERP server to ensure seamless connectivity between nodes.
 
 ## Configuration
 
@@ -31,8 +31,8 @@ traversal. [Check DERP server connectivity](#check-derp-server-connectivity) to
 ### Remove Tailscale's DERP servers
 
 Once enabled, Headscale's embedded DERP is added to the list of free-to-use [DERP
-servers](https://tailscale.com/kb/1232/derp-servers) offered by Tailscale Inc. To only use Headscale's embedded DERP
-server, disable the loading of the default DERP map:
+servers](https://tailscale.com/docs/reference/derp-servers) offered by Tailscale Inc. To only use Headscale's embedded
+DERP server, disable the loading of the default DERP map:
 
 ```yaml title="config.yaml" hl_lines="6"
 derp:
@@ -59,14 +59,14 @@ maps fetched via URL or to offer your own, custom DERP servers to nodes.
 
 === "Remove specific DERP regions"
 
-    The free-to-use [DERP servers](https://tailscale.com/kb/1232/derp-servers) are organized into regions via a region
-    ID. You can explicitly disable a specific region by setting its region ID to `null`. The following sample
+    The free-to-use [DERP servers](https://tailscale.com/docs/reference/derp-servers) are organized into regions via a
+    region ID. You can explicitly disable a specific region by setting its region ID to `null`. The following sample
     `derp.yaml` disables the New York DERP region (which has the region ID 1):
 
-     ```yaml title="derp.yaml"
-     regions:
-       1: null
-     ```
+    ```yaml title="derp.yaml"
+    regions:
+      1: null
+    ```
 
     Use the following configuration to serve the default DERP map (excluding New York) to nodes:
 
@@ -165,11 +165,10 @@ Any Tailscale client may be used to introspect the DERP map and to check for con
 Additional DERP related metrics and information is available via the [metrics and debug
 endpoint](./debug.md#metrics-and-debug-endpoint).
 
-[^1]:
-    This assumes that the default region code of the [configuration file](./configuration.md) is used.
-
 ## Limitations
 
 - The embedded DERP server can't be used for Tailscale's captive portal checks as it doesn't support the `/generate_204`
   endpoint via HTTP on port tcp/80.
 - There are no speed or throughput optimisations, the main purpose is to assist in node connectivity.
+
+[^1]: This assumes that the default region code of the [configuration file](./configuration.md) is used.

docs/ref/dns.md

@@ -6,8 +6,8 @@ within the `dns` section of the [configuration file](./configuration.md).
 ## Setting extra DNS records
 
 Headscale allows to set extra DNS records which are made available via
-[MagicDNS](https://tailscale.com/kb/1081/magicdns). Extra DNS records can be configured either via static entries in the
-[configuration file](./configuration.md) or from a JSON file that Headscale continuously watches for changes:
+[MagicDNS](https://tailscale.com/docs/features/magicdns). Extra DNS records can be configured either via static entries
+in the [configuration file](./configuration.md) or from a JSON file that Headscale continuously watches for changes:
 
 - Use the `dns.extra_records` option in the [configuration file](./configuration.md) for entries that are static and
   don't change while Headscale is running. Those entries are processed when Headscale is starting up and changes to the
@@ -25,7 +25,7 @@ hostname and port combination "http://hostname-in-magic-dns.myvpn.example.com:30
 
     Currently, [only A and AAAA records are processed by Tailscale](https://github.com/tailscale/tailscale/blob/v1.86.5/ipn/ipnlocal/node_backend.go#L662).
 
-1.  Configure extra DNS records using one of the available configuration options:
+1. Configure extra DNS records using one of the available configuration options:
 
     === "Static entries, via `dns.extra_records`"
 
@@ -66,12 +66,12 @@ hostname and port combination "http://hostname-in-magic-dns.myvpn.example.com:30
 
         !!! tip "Good to know"
 
-            * The `dns.extra_records_path` option in the [configuration file](./configuration.md) needs to reference the
+            - The `dns.extra_records_path` option in the [configuration file](./configuration.md) needs to reference the
               JSON file containing extra DNS records.
-            * Be sure to "sort keys" and produce a stable output in case you generate the JSON file with a script.
+            - Be sure to "sort keys" and produce a stable output in case you generate the JSON file with a script.
               Headscale uses a checksum to detect changes to the file and a stable output avoids unnecessary processing.
 
-1.  Verify that DNS records are properly set using the DNS querying tool of your choice:
+1. Verify that DNS records are properly set using the DNS querying tool of your choice:
 
     === "Query with dig"
 
@@ -87,7 +87,7 @@ hostname and port combination "http://hostname-in-magic-dns.myvpn.example.com:30
         100.64.0.3
         ```
 
-1.  Optional: Setup the reverse proxy
+1. Optional: Setup the reverse proxy
 
     The motivating example here was to be able to access internal monitoring services on the same host without
     specifying a port, depicted as NGINX configuration snippet:

docs/ref/integration/tools.md

@@ -7,10 +7,16 @@
 
 This page collects third-party tools, client libraries, and scripts related to headscale.
 
-| Name                  | Repository Link                                                 | Description                                                          |
-| --------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------- |
-| tailscale-manager     | [Github](https://github.com/singlestore-labs/tailscale-manager) | Dynamically manage Tailscale route advertisements                    |
-| headscalebacktosqlite | [Github](https://github.com/bigbozza/headscalebacktosqlite)     | Migrate headscale from PostgreSQL back to SQLite                     |
-| headscale-pf          | [Github](https://github.com/YouSysAdmin/headscale-pf)           | Populates user groups based on user groups in Jumpcloud or Authentik |
-| headscale-client-go   | [Github](https://github.com/hibare/headscale-client-go)         | A Go client implementation for the Headscale HTTP API.               |
-| headscale-zabbix      | [Github](https://github.com/dblanque/headscale-zabbix)          | A Zabbix Monitoring Template for the Headscale Service.              |
+- [headscale-operator](https://github.com/infradohq/headscale-operator) - Headscale Kubernetes Operator
+- [tailscale-manager](https://github.com/singlestore-labs/tailscale-manager) - Dynamically manage Tailscale route
+  advertisements
+- [headscalebacktosqlite](https://github.com/bigbozza/headscalebacktosqlite) - Migrate headscale from PostgreSQL back to
+  SQLite
+- [headscale-pf](https://github.com/YouSysAdmin/headscale-pf) - Populates user groups based on user groups in Jumpcloud
+  or Authentik
+- [headscale-client-go](https://github.com/hibare/headscale-client-go) - A Go client implementation for the Headscale
+  HTTP API.
+- [headscale-zabbix](https://github.com/dblanque/headscale-zabbix) - A Zabbix Monitoring Template for the Headscale
+  Service.
+- [tailscale-exporter](https://github.com/adinhodovic/tailscale-exporter) - A Prometheus exporter for Headscale that
+  provides network-level metrics using the Headscale API.

docs/ref/integration/web-ui.md

@@ -7,14 +7,21 @@
 
 Headscale doesn't provide a built-in web interface but users may pick one from the available options.
 
-| Name                   | Repository Link                                             | Description                                                                                  |
-| ---------------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
-| headscale-ui           | [Github](https://github.com/gurucomputing/headscale-ui)     | A web frontend for the headscale Tailscale-compatible coordination server                    |
-| HeadscaleUi            | [GitHub](https://github.com/simcu/headscale-ui)             | A static headscale admin ui, no backend environment required                                 |
-| Headplane              | [GitHub](https://github.com/tale/headplane)                 | An advanced Tailscale inspired frontend for headscale                                        |
-| headscale-admin        | [Github](https://github.com/GoodiesHQ/headscale-admin)      | Headscale-Admin is meant to be a simple, modern web interface for headscale                  |
-| ouroboros              | [Github](https://github.com/yellowsink/ouroboros)           | Ouroboros is designed for users to manage their own devices, rather than for admins          |
-| unraid-headscale-admin | [Github](https://github.com/ich777/unraid-headscale-admin)  | A simple headscale admin UI for Unraid, it offers Local (`docker exec`) and API Mode         |
-| headscale-console      | [Github](https://github.com/rickli-cloud/headscale-console) | WebAssembly-based client supporting SSH, VNC and RDP with optional self-service capabilities |
+- [headscale-ui](https://github.com/gurucomputing/headscale-ui) - A web frontend for the headscale Tailscale-compatible
+  coordination server
+- [HeadscaleUi](https://github.com/simcu/headscale-ui) - A static headscale admin ui, no backend environment required
+- [Headplane](https://github.com/tale/headplane) - An advanced Tailscale inspired frontend for headscale
+- [headscale-admin](https://github.com/GoodiesHQ/headscale-admin) - Headscale-Admin is meant to be a simple, modern web
+  interface for headscale
+- [ouroboros](https://github.com/yellowsink/ouroboros) - Ouroboros is designed for users to manage their own devices,
+  rather than for admins
+- [unraid-headscale-admin](https://github.com/ich777/unraid-headscale-admin) - A simple headscale admin UI for Unraid,
+  it offers Local (`docker exec`) and API Mode
+- [headscale-console](https://github.com/rickli-cloud/headscale-console) - WebAssembly-based client supporting SSH, VNC
+  and RDP with optional self-service capabilities
+- [headscale-piying](https://github.com/wszgrcy/headscale-piying) - headscale web ui,support visual ACL configuration
+- [HeadControl](https://github.com/ahmadzip/HeadControl) - Minimal Headscale admin dashboard, built with Go and HTMX
+- [Headscale Manager](https://github.com/hkdone/headscalemanager) - Headscale UI for Android
+- [Headscale UI](https://github.com/MunMunMiao/headscale-ui) - Headscale UI online and Self-hosting
 
 You can ask for support on our [Discord server](https://discord.gg/c84AZQhmpx) in the "web-interfaces" channel.

docs/ref/oidc.md

@@ -40,9 +40,9 @@ A basic configuration connects Headscale to an identity provider and typically r
 
 === "Identity provider"
 
-    * Create a new confidential client (`Client ID`, `Client secret`)
-    * Add Headscale's OIDC callback URL as valid redirect URL: `https://headscale.example.com/oidc/callback`
-    * Configure additional parameters to improve user experience such as: name, description, logo, …
+    - Create a new confidential client (`Client ID`, `Client secret`)
+    - Add Headscale's OIDC callback URL as valid redirect URL: `https://headscale.example.com/oidc/callback`
+    - Configure additional parameters to improve user experience such as: name, description, logo, …
 
 ### Enable PKCE (recommended)
 
@@ -63,8 +63,8 @@ recommended and needs to be configured for Headscale and the identity provider a
 
 === "Identity provider"
 
-    * Enable PKCE for the headscale client
-    * Set the PKCE challenge method to "S256"
+    - Enable PKCE for the headscale client
+    - Set the PKCE challenge method to "S256"
 
 ### Authorize users with filters
 
@@ -75,10 +75,11 @@ are configured, a user needs to pass all of them.
 
 === "Allowed domains"
 
-    * Check the email domain of each authenticating user against the list of allowed domains and only authorize users
+    - Check the email domain of each authenticating user against the list of allowed domains and only authorize users
       whose email domain matches `example.com`.
-    * Access allowed: `alice@example.com`
-    * Access denied: `bob@example.net`
+    - A verified email address is required [unless email verification is disabled](#control-email-verification).
+    - Access allowed: `alice@example.com`
+    - Access denied: `bob@example.net`
 
     ```yaml hl_lines="5-6"
     oidc:
@@ -91,10 +92,11 @@ are configured, a user needs to pass all of them.
 
 === "Allowed users/emails"
 
-    * Check the email address of each authenticating user against the list of allowed email addresses and only authorize
+    - Check the email address of each authenticating user against the list of allowed email addresses and only authorize
       users whose email is part of the `allowed_users` list.
-    * Access allowed: `alice@example.com`, `bob@example.net`
-    * Access denied: `mallory@example.net`
+    - A verified email address is required [unless email verification is disabled](#control-email-verification).
+    - Access allowed: `alice@example.com`, `bob@example.net`
+    - Access denied: `mallory@example.net`
 
     ```yaml hl_lines="5-7"
     oidc:
@@ -108,10 +110,10 @@ are configured, a user needs to pass all of them.
 
 === "Allowed groups"
 
-    * Use the OIDC `groups` claim of each authenticating user to get their group membership and only authorize users
+    - Use the OIDC `groups` claim of each authenticating user to get their group membership and only authorize users
       which are members in at least one of the referenced groups.
-    * Access allowed: users in the `headscale_users` group
-    * Access denied: users without groups, users with other groups
+    - Access allowed: users in the `headscale_users` group
+    - Access denied: users without groups, users with other groups
 
     ```yaml hl_lines="5-7"
     oidc:
@@ -123,19 +125,32 @@ are configured, a user needs to pass all of them.
         - "headscale_users"
     ```
 
+### Control email verification
+
+Headscale uses the `email` claim from the identity provider to synchronize the email address to its user profile. By
+default, a user's email address is only synchronized when the identity provider reports the email address as verified
+via the `email_verified: true` claim.
+
+Unverified emails may be allowed in case an identity provider does not send the `email_verified` claim or email
+verification is not required. In that case, a user's email address is always synchronized to the user profile.
+
+```yaml hl_lines="5"
+oidc:
+  issuer: "https://sso.example.com"
+  client_id: "headscale"
+  client_secret: "generated-secret"
+  email_verified_required: false
+```
+
 ### Customize node expiration
 
 The node expiration is the amount of time a node is authenticated with OpenID Connect until it expires and needs to
-reauthenticate. The default node expiration is 180 days. This can either be customized or set to the expiration from the
-Access Token.
+reauthenticate. The default node expiration can be configured via the top-level `node.expiry` setting.
 
 === "Customize node expiration"
 
-    ```yaml hl_lines="5"
-    oidc:
-      issuer: "https://sso.example.com"
-      client_id: "headscale"
-      client_secret: "generated-secret"
+    ```yaml hl_lines="2"
+    node:
       expiry: 30d   # Use 0 to disable node expiration
     ```
 
@@ -144,7 +159,6 @@ Access Token.
     Please keep in mind that the Access Token is typically a short-lived token that expires within a few minutes. You
     will have to configure token expiration in your identity provider to avoid frequent re-authentication.
 
-
     ```yaml hl_lines="5"
     oidc:
       issuer: "https://sso.example.com"
@@ -156,6 +170,7 @@ Access Token.
 !!! tip "Expire a node and force re-authentication"
 
     A node can be expired immediately via:
+
     ```console
     headscale node expire -i <NODE_ID>
     ```
@@ -166,13 +181,16 @@ You may refer to users in the Headscale policy via:
 
 - Email address
 - Username
-- Provider identifier (only available in the database or from your identity provider)
+- Provider identifier (this value is currently only available from the [API](api.md), database or directly from your
+  identity provider)
 
 !!! note "A user identifier in the policy must contain a single `@`"
 
     The Headscale policy requires a single `@` to reference a user. If the username or provider identifier doesn't
-    already contain a single `@`, it needs to be appended at the end. For example: the username `ssmith` has to be
-    written as `ssmith@` to be correctly identified as user within the policy.
+    already contain a single `@`, it needs to be appended at the end. For example: the Headscale username `ssmith` has
+    to be written as `ssmith@` to be correctly identified as user within the policy.
+
+    Ensure that the Headscale username itself does not end with `@`.
 
 !!! warning "Email address or username might be updated by users"
 
@@ -181,6 +199,34 @@ You may refer to users in the Headscale policy via:
     consequences for Headscale where a policy might no longer work or a user might obtain more access by hijacking an
     existing username or email address.
 
+!!! tip "Howto use the provider identifier in the policy"
+
+    The provider identifier uniquely identifies an OIDC user and a well-behaving identity provider guarantees that this
+    value never changes for a particular user. It is usually an opaque and long string and its value is currently only
+    available from the [API](api.md), database or directly from your identity provider).
+
+    Use the [API](api.md) with the `/api/v1/user` endpoint to fetch the provider identifier (`providerId`). The value
+    (be sure to append an `@` in case the provider identifier doesn't already contain an `@` somewhere) can be used
+    directly to reference a user in the policy. To improve readability of the policy, one may use the `groups` section
+    as an alias:
+
+    ```json
+    {
+      "groups": {
+        "group:alice": [
+          "https://soo.example.com/oauth2/openid/59ac9125-c31b-46c5-814e-06242908cf57@"
+        ]
+      },
+      "acls": [
+        {
+          "action": "accept",
+          "src": ["group:alice"],
+          "dst": ["*:*"]
+        }
+      ]
+    }
+    ```
+
 ## Supported OIDC claims
 
 Headscale uses [the standard OIDC claims](https://openid.net/specs/openid-connect-core-1_0.html#StandardClaims) to
@@ -189,7 +235,7 @@ endpoint.
 
 | Headscale profile   | OIDC claim           | Notes / examples                                                                                  |
 | ------------------- | -------------------- | ------------------------------------------------------------------------------------------------- |
-| email address       | `email`              | Only used when `email_verified: true`                                                             |
+| email address       | `email`              | Only verified emails are synchronized, unless `email_verified_required: false` is configured      |
 | display name        | `name`               | eg: `Sam Smith`                                                                                   |
 | username            | `preferred_username` | Depends on identity provider, eg: `ssmith`, `ssmith@idp.example.com`, `\\example.com\ssmith`      |
 | profile picture     | `picture`            | URL to a profile picture or avatar                                                                |
@@ -205,8 +251,6 @@ endpoint.
     - The username must be at least two characters long.
     - It must only contain letters, digits, hyphens, dots, underscores, and up to a single `@`.
     - The username must start with a letter.
-- A user's email address is only synchronized to the local user profile when the identity provider marks the email
-  address as verified (`email_verified: true`).
 
 Please see the [GitHub label "OIDC"](https://github.com/juanfont/headscale/labels/OIDC) for OIDC related issues.
 
@@ -233,8 +277,9 @@ Authelia is fully supported by Headscale.
 ### Authentik
 
 - Authentik is fully supported by Headscale.
-- [Headscale does not JSON Web Encryption](https://github.com/juanfont/headscale/issues/2446). Leave the field
+- [Headscale does not support JSON Web Encryption](https://github.com/juanfont/headscale/issues/2446). Leave the field
   `Encryption Key` in the providers section unset.
+- See Authentik's [Integrate with Headscale](https://integrations.goauthentik.io/networking/headscale/)
 
 ### Google OAuth
 
@@ -256,22 +301,30 @@ Console.
 #### Steps
 
 1. Go to [Google Console](https://console.cloud.google.com) and login or create an account if you don't have one.
-2. Create a project (if you don't already have one).
-3. On the left hand menu, go to `APIs and services` -> `Credentials`
-4. Click `Create Credentials` -> `OAuth client ID`
-5. Under `Application Type`, choose `Web Application`
-6. For `Name`, enter whatever you like
-7. Under `Authorised redirect URIs`, add Headscale's OIDC callback URL: `https://headscale.example.com/oidc/callback`
-8. Click `Save` at the bottom of the form
-9. Take note of the `Client ID` and `Client secret`, you can also download it for reference if you need it.
-10. [Configure Headscale following the "Basic configuration" steps](#basic-configuration). The issuer URL for Google
-    OAuth is: `https://accounts.google.com`.
+1. Create a project (if you don't already have one).
+1. On the left hand menu, go to `APIs and services` -> `Credentials`
+1. Click `Create Credentials` -> `OAuth client ID`
+1. Under `Application Type`, choose `Web Application`
+1. For `Name`, enter whatever you like
+1. Under `Authorised redirect URIs`, add Headscale's OIDC callback URL: `https://headscale.example.com/oidc/callback`
+1. Click `Save` at the bottom of the form
+1. Take note of the `Client ID` and `Client secret`, you can also download it for reference if you need it.
+1. [Configure Headscale following the "Basic configuration" steps](#basic-configuration). The issuer URL for Google
+   OAuth is: `https://accounts.google.com`.
 
 ### Kanidm
 
 - Kanidm is fully supported by Headscale.
 - Groups for the [allowed groups filter](#authorize-users-with-filters) need to be specified with their full SPN, for
   example: `headscale_users@sso.example.com`.
+- Kanidm sends the full SPN (`alice@sso.example.com`) as `preferred_username` by default. Headscale stores this value as
+  username which might be confusing as the username and email fields now contain values that look like an email address.
+  [Kanidm can be configured to send the short username as `preferred_username` attribute
+  instead](https://kanidm.github.io/kanidm/stable/integrations/oauth2.html#short-names):
+    ```console
+    kanidm system oauth2 prefer-short-username <client name>
+    ```
+    Once configured, the short username in Headscale will be `alice` and can be referred to as `alice@` in the policy.
 
 ### Keycloak
 
@@ -305,5 +358,19 @@ Entra ID is: `https://login.microsoftonline.com/<tenant-UUID>/v2.0`. The followi
 - `domain_hint: example.com` to use your own domain
 - `prompt: select_account` to force an account picker during login
 
-Groups for the [allowed groups filter](#authorize-users-with-filters) need to be specified with their group ID instead
+When using Microsoft Entra ID together with the [allowed groups filter](#authorize-users-with-filters), configure the
+Headscale OIDC scope without the `groups` claim, for example:
+
+```yaml
+oidc:
+  scope: ["openid", "profile", "email"]
+```
+
+Groups for the [allowed groups filter](#authorize-users-with-filters) need to be specified with their group ID(UUID) instead
 of the group name.
+
+## Switching OIDC providers
+
+Headscale only supports a single OIDC provider in its configuration, but it does store the provider identifier of each user. When switching providers, this might lead to issues with existing users: all user details (name, email, groups) might be identical with the new provider, but the identifier will differ. Headscale will be unable to create a new user as the name and email will already be in use for the existing users.
+
+At this time, you will need to manually update the `provider_identifier` column in the `users` table for each user with the appropriate value for the new provider. The identifier is built from the `iss` and `sub` claims of the OIDC ID token, for example `https://id.example.com/12340987`.

docs/ref/registration.md

@@ -0,0 +1,144 @@
+# Registration methods
+
+Headscale supports multiple ways to register a node. The preferred registration method depends on the identity of a node
+and your use case.
+
+## Identity model
+
+Tailscale's identity model distinguishes between personal and tagged nodes:
+
+- A personal node (or user-owned node) is owned by a human and typically refers to end-user devices such as laptops,
+  workstations or mobile phones. End-user devices are managed by a single user.
+- A tagged node (or service-based node or non-human node) provides services to the network. Common examples include web-
+  and database servers. Those nodes are typically managed by a team of users. Some additional restrictions apply for
+  tagged nodes, e.g. a tagged node is not allowed to [Tailscale SSH](https://tailscale.com/docs/features/tailscale-ssh)
+  into a personal node.
+
+Headscale implements Tailscale's identity model and distinguishes between personal and tagged nodes where a personal
+node is owned by a Headscale user and a tagged node is owned by a tag. Tagged devices are grouped under the special user
+`tagged-devices`.
+
+## Registration methods
+
+There are two main ways to register new nodes, [web authentication](#web-authentication) and [registration with a pre
+authenticated key](#pre-authenticated-key). Both methods can be used to register personal and tagged nodes.
+
+### Web authentication
+
+Web authentication is the default method to register a new node. It's interactive, where the client initiates the
+registration and the Headscale administrator needs to approve the new node before it is allowed to join the network. A
+node can be approved with:
+
+- Headscale CLI (described in this documentation)
+- [Headscale API](api.md)
+- Or delegated to an identity provider via [OpenID Connect](oidc.md)
+
+Web authentication relies on the presence of a Headscale user. Use the `headscale users` command to create a new
+user[^1]:
+
+```console
+headscale users create <USER>
+```
+
+=== "Personal devices"
+
+    Run `tailscale up` to login your personal device:
+
+    ```console
+    tailscale up --login-server <YOUR_HEADSCALE_URL>
+    ```
+
+    Usually, a browser window with further instructions is opened. This page explains how to complete the registration
+    on your Headscale server and it also prints the Auth ID required to approve the node:
+
+    ```console
+    headscale auth register --user <USER> --auth-id <AUTH_ID>
+    ```
+
+    Congrations, the registration of your personal node is complete and it should be listed as "online" in the output of
+    `headscale nodes list`. The "User" column displays `<USER>` as the owner of the node.
+
+=== "Tagged devices"
+
+    Your Headscale user needs to be authorized to register tagged devices. This authorization is specified in the
+    [`tagOwners`](https://tailscale.com/docs/reference/syntax/policy-file#tag-owners) section of the [ACL](acls.md). A
+    simple example looks like this:
+
+    ```json title="The user alice can register nodes tagged with tag:server"
+    {
+      "tagOwners": {
+        "tag:server": ["alice@"]
+      },
+      // more rules
+    }
+    ```
+
+    Run `tailscale up` and provide at least one tag to login a tagged device:
+
+    ```console
+    tailscale up --login-server <YOUR_HEADSCALE_URL> --advertise-tags tag:<TAG>
+    ```
+
+    Usually, a browser window with further instructions is opened. This page explains how to complete the registration
+    on your Headscale server and it also prints the Auth ID required to approve the node:
+
+    ```console
+    headscale auth register --user <USER> --auth-id <AUTH_ID>
+    ```
+
+    Headscale checks that `<USER>` is allowed to register a node with the specified tag(s) and then transfers ownership
+    of the new node to the special user `tagged-devices`. The registration of a tagged node is complete and it should be
+    listed as "online" in the output of `headscale nodes list`. The "User" column displays `tagged-devices` as the owner
+    of the node. See the "Tags" column for the list of assigned tags.
+
+### Pre authenticated key
+
+Registration with a pre authenticated key (or auth key) is a non-interactive way to register a new node. The Headscale
+administrator creates a preauthkey upfront and this preauthkey can then be used to register a node non-interactively.
+Its best suited for automation.
+
+=== "Personal devices"
+
+    A personal node is always assigned to a Headscale user. Use the `headscale users` command to create a new user[^1]:
+
+    ```console
+    headscale users create <USER>
+    ```
+
+    Use the `headscale user list` command to learn its `<USER_ID>` and create a new pre authenticated key for your user:
+
+    ```console
+    headscale preauthkeys create --user <USER_ID>
+    ```
+
+    The above prints a pre authenticated key with the default settings (can be used once and is valid for one hour). Use
+    this auth key to register a node non-interactively:
+
+    ```console
+    tailscale up --login-server <YOUR_HEADSCALE_URL> --authkey <YOUR_AUTH_KEY>
+    ```
+
+    Congrations, the registration of your personal node is complete and it should be listed as "online" in the output of
+    `headscale nodes list`. The "User" column displays `<USER>` as the owner of the node.
+
+=== "Tagged devices"
+
+    Create a new pre authenticated key and provide at least one tag:
+
+    ```console
+    headscale preauthkeys create --tags tag:<TAG>
+    ```
+
+    The above prints a pre authenticated key with the default settings (can be used once and is valid for one hour). Use
+    this auth key to register a node non-interactively. You don't need to provide the `--advertise-tags` parameter as
+    the tags are automatically read from the pre authenticated key:
+
+    ```console
+    tailscale up --login-server <YOUR_HEADSCALE_URL> --authkey <YOUR_AUTH_KEY>
+    ```
+
+    The registration of a tagged node is complete and it should be listed as "online" in the output of
+    `headscale nodes list`. The "User" column displays `tagged-devices` as the owner of the node. See the "Tags" column for the list of
+    assigned tags.
+
+[^1]: [Ensure that the Headscale username does not end with `@`.](oidc.md#reference-a-user-in-the-policy)

docs/ref/remote-cli.md

@@ -1,99 +0,0 @@
-# Controlling headscale with remote CLI
-
-This documentation has the goal of showing a user how-to control a headscale instance
-from a remote machine with the `headscale` command line binary.
-
-## Prerequisite
-
-- A workstation to run `headscale` (any supported platform, e.g. Linux).
-- A headscale server with gRPC enabled.
-- Connections to the gRPC port (default: `50443`) are allowed.
-- Remote access requires an encrypted connection via TLS.
-- An API key to authenticate with the headscale server.
-
-## Create an API key
-
-We need to create an API key to authenticate with the remote headscale server when using it from our workstation.
-
-To create an API key, log into your headscale server and generate a key:
-
-```shell
-headscale apikeys create --expiration 90d
-```
-
-Copy the output of the command and save it for later. Please note that you can not retrieve a key again,
-if the key is lost, expire the old one, and create a new key.
-
-To list the keys currently associated with the server:
-
-```shell
-headscale apikeys list
-```
-
-and to expire a key:
-
-```shell
-headscale apikeys expire --prefix "<PREFIX>"
-```
-
-## Download and configure headscale
-
-1.  Download the [`headscale` binary from GitHub's release page](https://github.com/juanfont/headscale/releases). Make
-    sure to use the same version as on the server.
-
-1.  Put the binary somewhere in your `PATH`, e.g. `/usr/local/bin/headscale`
-
-1.  Make `headscale` executable:
-
-    ```shell
-    chmod +x /usr/local/bin/headscale
-    ```
-
-1.  Provide the connection parameters for the remote headscale server either via a minimal YAML configuration file or via
-    environment variables:
-
-    === "Minimal YAML configuration file"
-
-        ```yaml title="config.yaml"
-        cli:
-            address: <HEADSCALE_ADDRESS>:<PORT>
-            api_key: <API_KEY_FROM_PREVIOUS_STEP>
-        ```
-
-    === "Environment variables"
-
-        ```shell
-        export HEADSCALE_CLI_ADDRESS="<HEADSCALE_ADDRESS>:<PORT>"
-        export HEADSCALE_CLI_API_KEY="<API_KEY_FROM_PREVIOUS_STEP>"
-        ```
-
-    This instructs the `headscale` binary to connect to a remote instance at `<HEADSCALE_ADDRESS>:<PORT>`, instead of
-    connecting to the local instance.
-
-1.  Test the connection
-
-    Let us run the headscale command to verify that we can connect by listing our nodes:
-
-    ```shell
-    headscale nodes list
-    ```
-
-    You should now be able to see a list of your nodes from your workstation, and you can
-    now control the headscale server from your workstation.
-
-## Behind a proxy
-
-It is possible to run the gRPC remote endpoint behind a reverse proxy, like Nginx, and have it run on the _same_ port as headscale.
-
-While this is _not a supported_ feature, an example on how this can be set up on
-[NixOS is shown here](https://github.com/kradalby/dotfiles/blob/4489cdbb19cddfbfae82cd70448a38fde5a76711/machines/headscale.oracldn/headscale.nix#L61-L91).
-
-## Troubleshooting
-
-- Make sure you have the _same_ headscale version on your server and workstation.
-- Ensure that connections to the gRPC port are allowed.
-- Verify that your TLS certificate is valid and trusted.
-- If you don't have access to a trusted certificate (e.g. from Let's Encrypt), either:
-    - Add your self-signed certificate to the trust store of your OS _or_
-    - Disable certificate verification by either setting `cli.insecure: true` in the configuration file or by setting
-      `HEADSCALE_CLI_INSECURE=1` via an environment variable. We do **not** recommend to disable certificate validation.

docs/ref/routes.md

@@ -1,7 +1,8 @@
 # Routes
 
-Headscale supports route advertising and can be used to manage [subnet routers](https://tailscale.com/kb/1019/subnets)
-and [exit nodes](https://tailscale.com/kb/1103/exit-nodes) for a tailnet.
+Headscale supports route advertising and can be used to manage [subnet
+routers](https://tailscale.com/docs/features/subnet-routers) and [exit
+nodes](https://tailscale.com/docs/features/exit-nodes) for a tailnet.
 
 - [Subnet routers](#subnet-router) may be used to connect an existing network such as a virtual
   private cloud or an on-premise network with your tailnet. Use a subnet router to access devices where Tailscale can't
@@ -42,8 +43,9 @@ can be used.
 
 ```console
 $ headscale nodes list-routes
-ID | Hostname | Approved | Available                  | Serving (Primary)
-1  | myrouter |          | 10.0.0.0/8, 192.168.0.0/24 |
+ID | Hostname | Approved | Available      | Serving (Primary)
+1  | myrouter |          | 10.0.0.0/8     |
+   |          |          | 192.168.0.0/24 |
 ```
 
 Approve all desired routes of a subnet router by specifying them as comma separated list:
@@ -57,8 +59,9 @@ The node `myrouter` can now route the IPv4 networks `10.0.0.0/8` and `192.168.0.
 
 ```console
 $ headscale nodes list-routes
-ID | Hostname | Approved                   | Available                  | Serving (Primary)
-1  | myrouter | 10.0.0.0/8, 192.168.0.0/24 | 10.0.0.0/8, 192.168.0.0/24 | 10.0.0.0/8, 192.168.0.0/24
+ID | Hostname | Approved       | Available      | Serving (Primary)
+1  | myrouter | 10.0.0.0/8     | 10.0.0.0/8     | 10.0.0.0/8
+   |          | 192.168.0.0/24 | 192.168.0.0/24 | 192.168.0.0/24
 ```
 
 #### Use the subnet router
@@ -70,8 +73,8 @@ $ sudo tailscale set --accept-routes
 ```
 
 Please refer to the official [Tailscale
-documentation](https://tailscale.com/kb/1019/subnets#use-your-subnet-routes-from-other-devices) for how to use a subnet
-router on different operating systems.
+documentation](https://tailscale.com/docs/features/subnet-routers#use-your-subnet-routes-from-other-devices) for how to
+use a subnet router on different operating systems.
 
 ### Restrict the use of a subnet router with ACL
 
@@ -109,9 +112,9 @@ approval of routes served with a subnet router.
 
 The ACL snippet below defines the tag `tag:router` owned by the user `alice`. This tag is used for `routes` in the
 `autoApprovers` section. The IPv4 route `192.168.0.0/24` is automatically approved once announced by a subnet router
-owned by the user `alice` and that also advertises the tag `tag:router`.
+that advertises the tag `tag:router`.
 
-```json title="Subnet routers owned by alice and tagged with tag:router are automatically approved"
+```json title="Subnet routers tagged with tag:router are automatically approved"
 {
   "tagOwners": {
     "tag:router": ["alice@"]
@@ -133,8 +136,9 @@ Advertise the route `192.168.0.0/24` from a subnet router that also advertises t
 $ sudo tailscale up --login-server <YOUR_HEADSCALE_URL> --advertise-tags tag:router --advertise-routes 192.168.0.0/24
 ```
 
-Please see the [official Tailscale documentation](https://tailscale.com/kb/1337/acl-syntax#autoapprovers) for more
-information on auto approvers.
+Please see the [official Tailscale
+documentation](https://tailscale.com/docs/reference/syntax/policy-file#auto-approvers) for more information on auto
+approvers.
 
 ## Exit node
 
@@ -168,8 +172,9 @@ available, but needs to be approved:
 
 ```console
 $ headscale nodes list-routes
-ID | Hostname | Approved | Available       | Serving (Primary)
-1  | myexit   |          | 0.0.0.0/0, ::/0 |
+ID | Hostname | Approved | Available | Serving (Primary)
+1  | myexit   |          | 0.0.0.0/0 |
+   |          |          | ::/0      |
 ```
 
 For exit nodes, it is sufficient to approve either the IPv4 or IPv6 route. The other will be approved automatically.
@@ -183,8 +188,9 @@ The node `myexit` is now approved as exit node for the tailnet:
 
 ```console
 $ headscale nodes list-routes
-ID | Hostname | Approved        | Available       | Serving (Primary)
-1  | myexit   | 0.0.0.0/0, ::/0 | 0.0.0.0/0, ::/0 | 0.0.0.0/0, ::/0
+ID | Hostname | Approved  | Available | Serving (Primary)
+1  | myexit   | 0.0.0.0/0 | 0.0.0.0/0 | 0.0.0.0/0
+   |          | ::/0      | ::/0      | ::/0
 ```
 
 #### Use the exit node
@@ -195,8 +201,8 @@ The exit node can now be used on a node with:
 $ sudo tailscale set --exit-node myexit
 ```
 
-Please refer to the official [Tailscale documentation](https://tailscale.com/kb/1103/exit-nodes#use-the-exit-node) for
-how to use an exit node on different operating systems.
+Please refer to the official [Tailscale documentation](https://tailscale.com/docs/features/exit-nodes#use-the-exit-node)
+for how to use an exit node on different operating systems.
 
 ### Restrict the use of an exit node with ACL
 
@@ -256,10 +262,9 @@ in a tailnet. Headscale supports the `autoApprovers` section of an ACL to automa
 soon as it joins the tailnet.
 
 The ACL snippet below defines the tag `tag:exit` owned by the user `alice`. This tag is used for `exitNode` in the
-`autoApprovers` section. A new exit node which is owned by the user `alice` and that also advertises the tag `tag:exit`
-is automatically approved:
+`autoApprovers` section. A new exit node that advertises the tag `tag:exit` is automatically approved:
 
-```json title="Exit nodes owned by alice and tagged with tag:exit are automatically approved"
+```json title="Exit nodes tagged with tag:exit are automatically approved"
 {
   "tagOwners": {
     "tag:exit": ["alice@"]
@@ -279,26 +284,23 @@ Advertise a node as exit node and also advertise the tag `tag:exit` when joining
 $ sudo tailscale up --login-server <YOUR_HEADSCALE_URL> --advertise-tags tag:exit --advertise-exit-node
 ```
 
-Please see the [official Tailscale documentation](https://tailscale.com/kb/1337/acl-syntax#autoapprovers) for more
-information on auto approvers.
+Please see the [official Tailscale documentation](https://tailscale.com/docs/reference/syntax/policy-file#autoapprovers)
+for more information on auto approvers.
 
 ## High availability
 
-Headscale has limited support for high availability routing. Multiple subnet routers with overlapping routes or multiple
-exit nodes can be used to provide high availability for users. If one router node goes offline, another one can serve
-the same routes to clients. Please see the official [Tailscale documentation on high
-availability](https://tailscale.com/kb/1115/high-availability#subnet-router-high-availability) for details.
+Headscale supports high availability routing. Multiple subnet routers with overlapping routes or multiple exit nodes can
+be used to provide high availability for users. If one router node goes offline, another one can serve the same routes
+to clients. Please see the official [Tailscale documentation on high
+availability](https://tailscale.com/docs/how-to/set-up-high-availability#subnet-router-high-availability) for details.
 
-!!! bug
-
-    In certain situations it might take up to 16 minutes for Headscale to detect a node as offline. A failover node
-    might not be selected fast enough, if such a node is used as subnet router or exit node causing service
-    interruptions for clients. See [issue 2129](https://github.com/juanfont/headscale/issues/2129) for more information.
+This feature is enabled by default when at least two nodes advertise the same prefix. See the configuration options
+`node.routes.ha` in the [configuration file](./configuration.md) for details.
 
 ## Troubleshooting
 
 ### Enable IP forwarding
 
 A subnet router or exit node is routing traffic on behalf of other nodes and thus requires IP forwarding. Check the
-official [Tailscale documentation](https://tailscale.com/kb/1019/subnets/?tab=linux#enable-ip-forwarding) for how to
+official [Tailscale documentation](https://tailscale.com/docs/features/subnet-routers#enable-ip-forwarding) for how to
 enable IP forwarding.

docs/ref/tags.md

@@ -0,0 +1,54 @@
+# Tags
+
+Headscale supports Tailscale tags. Please read [Tailscale's tag documentation](https://tailscale.com/docs/features/tags)
+to learn how tags work and how to use them.
+
+Tags can be applied during [node registration](registration.md):
+
+- using the `--advertise-tags` flag, see [web authentication for tagged devices](registration.md#__tabbed_1_2)
+- using a tagged pre authenticated key, see [how to create and use it](registration.md#__tabbed_2_2)
+
+Administrators can manage tags with:
+
+- Headscale CLI
+- [Headscale API](api.md)
+
+## Common operations
+
+### Manage tags for a node
+
+Run `headscale nodes list` to list the tags for a node.
+
+Use the `headscale nodes tag` command to modify the tags for a node. At least one tag is required and multiple tags can
+be provided as comma separated list. The following command sets the tags `tag:server` and `tag:prod` on node with ID 1:
+
+```console
+headscale nodes tag -i 1 -t tag:server,tag:prod
+```
+
+### Convert from personal to tagged node
+
+Use the `headscale nodes tag` command to convert a personal (user-owned) node to a tagged node:
+
+```console
+headscale nodes tag -i <NODE_ID> -t <TAG>
+```
+
+The node is now owned by the special user `tagged-devices` and has the specified tags assigned to it.
+
+### Convert from tagged to personal node
+
+Tagged nodes can return to personal (user-owned) nodes by re-authenticating with:
+
+```console
+tailscale up --login-server <YOUR_HEADSCALE_URL> --advertise-tags= --force-reauth
+```
+
+Usually, a browser window with further instructions is opened. This page explains how to complete the registration on
+your Headscale server and it also prints the Auth ID required to approve the node:
+
+```console
+headscale auth register --user <USER> --auth-id <AUTH_ID>
+```
+
+All previously assigned tags get removed and the node is now owned by the user specified in the above command.

docs/ref/tls.md

@@ -50,7 +50,7 @@ Headscale uses [autocert](https://pkg.go.dev/golang.org/x/crypto/acme/autocert),
 If you want to validate that certificate renewal completed successfully, this can be done either manually, or through external monitoring software. Two examples of doing this manually:
 
 1. Open the URL for your headscale server in your browser of choice, and manually inspecting the expiry date of the certificate you receive.
-2. Or, check remotely from CLI using `openssl`:
+1. Or, check remotely from CLI using `openssl`:
 
 ```console
 $ openssl s_client -servername [hostname] -connect [hostname]:443 | openssl x509 -noout -dates

docs/requirements.txt

@@ -1,6 +1,6 @@
 mike~=2.1
-mkdocs-include-markdown-plugin~=7.1
-mkdocs-macros-plugin~=1.3
-mkdocs-material[imaging]~=9.5
-mkdocs-minify-plugin~=0.7
+mkdocs-include-markdown-plugin~=7.2
+mkdocs-macros-plugin~=1.5
+mkdocs-materialx[imaging]~=10.1
+mkdocs-minify-plugin~=0.8
 mkdocs-redirects~=1.2

docs/setup/install/container.md

@@ -18,25 +18,26 @@ Registry](https://github.com/juanfont/headscale/pkgs/container/headscale). The c
 
 ## Configure and run headscale
 
-1.  Create a directory on the Docker host to store headscale's [configuration](../../ref/configuration.md) and the [SQLite](https://www.sqlite.org/) database:
+1. Create a directory on the container host to store headscale's [configuration](../../ref/configuration.md) and the [SQLite](https://www.sqlite.org/) database:
 
     ```shell
-    mkdir -p ./headscale/{config,lib,run}
+    mkdir -p ./headscale/{config,lib}
     cd ./headscale
     ```
 
-1.  Download the example configuration for your chosen version and save it as: `$(pwd)/config/config.yaml`. Adjust the
-    configuration to suit your local environment. See [Configuration](../../ref/configuration.md) for details.
+1. Download the example configuration for your chosen version and save it as: `$(pwd)/config/config.yaml`. Adjust the
+   configuration to suit your local environment. See [Configuration](../../ref/configuration.md) for details.
 
-1.  Start headscale from within the previously created `./headscale` directory:
+1. Start headscale from within the previously created `./headscale` directory:
 
     ```shell
     docker run \
       --name headscale \
       --detach \
-      --volume "$(pwd)/config:/etc/headscale" \
+      --read-only \
+      --tmpfs /var/run/headscale \
+      --volume "$(pwd)/config:/etc/headscale:ro" \
       --volume "$(pwd)/lib:/var/lib/headscale" \
-      --volume "$(pwd)/run:/var/run/headscale" \
       --publish 127.0.0.1:8080:8080 \
       --publish 127.0.0.1:9090:9090 \
       --health-cmd "CMD headscale health" \
@@ -57,21 +58,23 @@ Registry](https://github.com/juanfont/headscale/pkgs/container/headscale). The c
         image: docker.io/headscale/headscale:<VERSION>
         restart: unless-stopped
         container_name: headscale
+        read_only: true
+        tmpfs:
+          - /var/run/headscale
         ports:
           - "127.0.0.1:8080:8080"
           - "127.0.0.1:9090:9090"
         volumes:
           # Please set <HEADSCALE_PATH> to the absolute path
           # of the previously created headscale directory.
-          - <HEADSCALE_PATH>/config:/etc/headscale
+          - <HEADSCALE_PATH>/config:/etc/headscale:ro
           - <HEADSCALE_PATH>/lib:/var/lib/headscale
-          - <HEADSCALE_PATH>/run:/var/run/headscale
         command: serve
         healthcheck:
             test: ["CMD", "headscale", "health"]
     ```
 
-1.  Verify headscale is running:
+1. Verify headscale is running:
 
     Follow the container logs:
 
@@ -88,45 +91,10 @@ Registry](https://github.com/juanfont/headscale/pkgs/container/headscale). The c
     Verify headscale is available:
 
     ```shell
-    curl http://127.0.0.1:9090/metrics
+    curl http://127.0.0.1:8080/health
     ```
 
-1.  Create a headscale user:
-
-    ```shell
-    docker exec -it headscale \
-      headscale users create myfirstuser
-    ```
-
-### Register a machine (normal login)
-
-On a client machine, execute the `tailscale up` command to login:
-
-```shell
-tailscale up --login-server YOUR_HEADSCALE_URL
-```
-
-To register a machine when running headscale in a container, take the headscale command and pass it to the container:
-
-```shell
-docker exec -it headscale \
-  headscale nodes register --user myfirstuser --key <YOUR_MACHINE_KEY>
-```
-
-### Register a machine using a pre authenticated key
-
-Generate a key using the command line for the user with ID 1:
-
-```shell
-docker exec -it headscale \
-  headscale preauthkeys create --user 1 --reusable --expiration 24h
-```
-
-This will return a pre-authenticated key that can be used to connect a node to headscale with the `tailscale up` command:
-
-```shell
-tailscale up --login-server <YOUR_HEADSCALE_URL> --authkey <YOUR_AUTH_KEY>
-```
+Continue on the [getting started page](../../usage/getting-started.md) to register your first machine.
 
 ## Debugging headscale running in Docker