A raw coder/websocket dial and the real tailscale.com js/wasm control client under Node, both against headscale alongside normal Tailscale clients.
Updates #3357
The chi migration dropped GET; JS/WASM control clients open /ts2021 as a WebSocket GET and were rejected with 405 before reaching NoiseUpgradeHandler.
Fixes#3357
Scan a node-health check registry at boot and log each node whose name
can't form a valid FQDN, with the rename fix. Log-only, no mutation.
Updates #3346
A peer whose GivenName fails GetFQDN aborted the whole map for every node
that could see it. Drop and log it; SSH policy errors degrade too.
Fixes#3346
TestInitialMapNotStarvedByReconnectStorm reproduces the #3346 stall;
TestPolicyManagerConcurrentReads guards the RLock cache access under -race.
Updates #3346
One exclusive mutex serialized every policy read, so a mass reconnect on
autogroup:self/via/relay policies stalled clients into "unexpected EOF"
retries. Per-node caches become xsync.Maps for lazy population under RLock.
Fixes#3346
It needs no special runner (every integration container already runs privileged), so it joins the generated matrix; load br_netfilter only for that test.
Updates #1202
Run the real operator in a single-container k3s cluster against an in-test Headscale over plain HTTP. k3sic exposes reusable building blocks (InstallOperator, DeployConnector, DeployEchoServer, ExposeServiceToTailnet, DeployProxyGroup); the test covers operator registration, an egress connector, ingress connectivity from a tailnet node, and proxy groups. tls-ca-baking.md records the private-CA TLS variant.
Updates #1202
Mints an admin API key and creates a keyType=client OAuth credential through gen/client/v2, exposed on ControlServer. Reusable by tests needing OAuth client credentials.
Sets TS_DEBUG_DERP_WS_CLIENT + TS_DEBUG_USE_DERP_HTTP so a client can reach hsic's plain-HTTP embedded DERP over websockets; the counterpart to hsic.WithoutTLS.
Defence-in-depth fixes to the OIDC login flow.
Set the state/nonce cookie Secure flag from the configured server_url
scheme rather than req.TLS, so the cookies stay Secure behind a
TLS-terminating reverse proxy where the proxy-to-Headscale hop is plain
HTTP. Deriving it from config avoids trusting a spoofable
X-Forwarded-Proto header.
Make the OIDC state single-use: consume it from the cache on the
callback and clear the state/nonce cookies once validated, so a replayed
callback cannot resolve the same session and the cookies do not linger
until expiry.
Bound OIDC discovery to the caller's context so a slow or unreachable
issuer fails startup within the timeout instead of hanging, and validate
the issuer URL and required client_id/client_secret at config load so an
unworkable setup fails fast.
Add the OAuth client type, its database storage, the scope grant package,
policy tag-ownership exposure, and the state operations backing the v2
OAuth client-credentials flow.
NewAuthProviderOIDC received a context with a 30-second timeout from
its caller, but immediately discarded it by passing context.Background()
to oidc.NewProvider. This caused both `headscale serve` and
`headscale configtest` to hang indefinitely if the OIDC issuer was
unreachable, rather than timing out and returning an error.
Fix by passing the caller's ctx to oidc.NewProvider and remove the
nolint:contextcheck annotation that was suppressing the linter warning
about this bug. The annotation was introduced in ce580f82 as part of a
bulk lint-suppression pass without addressing the underlying issue.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Exercises the user data sources via the Go SDK, tscli, and OpenTofu, and
backfills tailscale_4via6 (provider-local compute), each with the
no-drift gate.
Pins the wire shape: honest constants, deviceCount aggregation, the
{"users":[...]} envelope, ignored type/role filters, and 404 on
unknown/non-numeric/pseudo-user ids.
Backs the tailscale_user and tailscale_users data sources. getUser maps
gorm/ErrUserNotFound to 404; unmodelled fields are honest constants,
deviceCount/lastSeen/currentlyConnected aggregate the user's nodes.
Tailscale's keys API has no separate expire verb: DELETE is the revoke.
Map it to a soft revoke so the key stays retrievable as invalid afterwards
instead of vanishing, matching the SDK and Terraform's expectations.
Add a revoked timestamp to pre-auth keys (migration plus schema), mark a
key invalid once revoked, and have the keys DELETE handler stamp it rather
than destroy the row. A background collector reaps revoked keys after a
configurable retention window (preauth_keys.revoked_retention, default
168h), so the table does not grow without bound.
Add the v2 foundation — Basic/Bearer auth, scope scaffolding, the
tailnet check, and the Tailscale error shape — plus the auth-key
endpoints mapped onto Headscale pre-auth keys.
Give API keys an optional owning user and pre-auth keys a free-text
description, plus the state accessors the v2 API needs to act as a
key's owner and to persist descriptions.
Express the maximum SSH check period as 7 * 24 * time.Hour instead of a
bare nanosecond constant, matching how the rest of the code spells out
magic durations.
Serialize node, user and pre-auth-key responses through the NodeView,
UserView and PreAuthKeyView accessors instead of AsStruct(), which clones
the whole record on every read. Document the convention in AGENTS.md.
Add StringID() on Node/NodeView and Username/Display/CreatedAt on
UserView, so the HTTP read paths render ids and read users through view
accessors instead of cloning with AsStruct().
Adopt kradalby/flake-checks: flake.nix exposes build, gotest,
golangci-lint and formatting checks over one shared, fileset-filtered
source set, and nix-checks.yml gates each with
nix build .#checks.<system>.<name>. Drop test.yml and lint.yml —
gotestsum, golangci-lint and prettier are now those checks. golangci-lint
runs full-tree; Go formatting stays in it while formatting adds prettier.
hscontrol/servertest is too slow and timing-sensitive for the sandboxed
gotest check, so servertest.yml runs it in the devShell instead.
The new full-tree golangci-lint check reports issues the --new-from-rev
diff lint hid: nine wsl_v5 whitespace gaps, a prealloc, and an unparam
(setCSRFCookie never errored, so drop the return and update callers).
gocyclo on the central UpdateNodeFromMapRequest and an SA1019 NetMap
deprecation in an integration helper are suppressed with reasons.
Swap nix-quick-install-action -> nix-installer-action and
cache-nix-action -> Mic92/hestia/action across every Nix workflow, drop
the now-unneeded cache keys, and run devShell tooling through a
defaults.run.shell. Add gc.yml to trim the hestia cache daily.
newRemoteClient prepended "https://" unconditionally, so
HEADSCALE_CLI_ADDRESS=https://host became https://https://host and dialed
the host "https". Default a bare host to https, keep an explicit scheme.
With the v1 API served by Huma and the CLI on the HTTP client, nothing
uses the gRPC service or its generated code. Delete proto/, the generated
gen/go and gen/openapiv2, grpcv1.go, the buf config, the proto .Proto()
helpers and gRPC config, and the proto build tooling from the flake and
CI.
Replace the gRPC client with the generated HTTP client across every
command: locally over the unix socket without auth (matching the previous
local gRPC socket), remotely over TLS with a Bearer API key. Output
rendering and integration tests move to the HTTP client types; the
transport changes, the assertions do not.
Generate a strongly-typed Go client (oapi-codegen) from the emitted spec, committed under gen/client/v1 as package clientv1. Tests exercise it against the in-memory Huma server over both TCP and the unix socket.
Pin every endpoint's behaviour with golden fixtures captured from the
retiring grpc-gateway (timestamps and secrets neutralised). Error cases
assert their HTTP status independently of the golden via assertStatus, so
a blind regenerate cannot re-pin a wrong code; HEADSCALE_UPDATE_GOLDEN
rewrites the fixtures. A unit test exercises the API-key auth middleware
on the real server router. Exclude the emitted spec and goldens from
pre-commit checks.
Reimplement the v1 API as a code-first Huma service in hscontrol/api/v1,
a thin adapter over the state layer. Huma emits the OpenAPI 3.1 document
(openapi/v1/headscale.yaml) from the Go operation and type definitions;
cmd/gen-openapi writes it and the 3.0.3 downgrade the client is generated
from. Responses reproduce the protojson wire contract (string-encoded
64-bit IDs, all fields emitted, RFC3339/null timestamps); errors map to
the correct HTTP status (404/400/409, 500 for server faults) via mapError.
Serve it on the chi router at /api/v1 behind the existing API-key
middleware, and point /swagger at the emitted spec.
Expiring or deleting a pre-auth key that does not exist updated zero rows
and returned no error, so the caller could not tell a missing key from a
successful no-op. Return ErrPreAuthKeyNotFound when RowsAffected is zero.
dockertest's bundled client builds against API v1.25; Docker Engine 29
rejects it (min 1.40), surfacing as a "broken pipe" that fails every local
image build. Pin to the daemon's reported version so builds use a live path.
Closes#2937
A nil tags slice marshals to JSON `null`; the clear-tagged migration
read that as tagged and cleared user_id. Exclude it, and recover
already-detached nodes from their pre-auth key.
Fixes#3323
The First()-by-machine-key getter returned an undefined node when a machine
key mapped to several nodes. It was used only by RegisterNodeForTest; match on
(machine_key, user_id) there instead and remove the getter.
Updates #3312
Collapse the single-pick machine-key lookups onto GetNodesByMachineKeyAllUsers
so callers see every node sharing a machine key and reject the ambiguous or
impossible cases (tagged and user-owned coexistence; a tagged key with several
user-owned candidates) instead of mutating an arbitrarily-picked node.
Updates #3312
Tagged nodes disable key expiry by default but can still have one set
explicitly, and changing tags leaves expiry unchanged, matching Tailscale.
Updates #3312
Concurrent registrations of one machine key each saw no existing node and
created their own, duplicating nodes and IPs. Hold a per-machine lock across
the find-then-create section.
Updates #3312
Re-registration mutated the node store before the database write and did not
revert on failure, so a restart could drop the client's current node key.
Snapshot the node and restore it if the write fails.
Updates #3312
An unknown or deleted key returned a bare error matching neither the
not-found nor pre-auth-key checks, so registration returned a server error
instead of 401. Wrap gorm.ErrRecordNotFound.
Updates #3312
A reusable key on a converted node, or a tagged key on a user-owned node,
fell through to new-node creation, leaving two nodes per machine. Match by
machine key and update or convert in place.
Updates #3312
The lookup returned the first map match, so re-auth branch choice varied
with map order once a machine key had more than one node. Prefer the tagged
node, else the lowest node ID.
Updates #3312
The pre-auth-key path wrote the client node key without the collision check
the auth path applies, so a re-registration could claim another node's key
and poison the node-key index. Reject keys bound to a different machine.
Updates #3312
The re-registration fast-path skipped validation for a matching node key
without checking expiry, so an expired node could re-auth with a spent key.
Gate it on the node not being expired.
Updates #3312
A node converted to tagged is re-indexed under no user, so re-registration
keyed on the key's owner missed it and rejected the spent one-shot key.
Match the existing tagged node by machine key.
Fixes#3312
Tailscale sends the sentinel time.Unix(123, 0) on logout; storing it
verbatim propagates a 1970 KeyExpiry to every peer's netmap. Same
semantics (expired as of now), saner logs and debug dumps.
NodeCanHaveTag, TagExists, ViaRoutesForPeer checked pm.pol before
taking pm.mu, racing SetPolicy's write (caught by -race in
TestRaceConcurrentServerMutations). DebugString read pol, filter, and
the derived maps with no lock at all.
AddNode registers the connection before sending the initial map, so a
batched delta could land first and become the stream's first frame;
Tailscale clients then kill the poll with "initial MapResponse lacked
Node" and a retry loop under steady change traffic leaves the netmap
empty. Skip connections awaiting their initial map and retry the
change next tick — the in-flight map may predate it, so it cannot be
dropped. Retries are prepended to keep patch order.
Mirrors the flaky integration relogin tests with production batcher
tuning. The churn variant restarts map polls around login like newer
tailscaled does; pre-fix it stranded nodes online after logout 2/2.
A cancelled map request whose handler ran late could Connect after the
live session, steal the newest SessionEpoch, then exit without
disconnecting (stillConnected path); the live session's final
Disconnect was rejected as stale and the node stayed online forever
(relogin flake). Counted releases are order-independent, so overlapping
sessions cannot strand a node in either direction.
Sentinel ErrNodeKeyInUse (err113); key the visible-peer set by tailcfg.NodeID
to drop an int64->uint64 cast (gosec G115); NewRequestWithContext (noctx); wsl.
filterVisiblePeerPatches and filterVisibleNodes now share one visiblePeerIDs
helper using the live MatchersForNode/ReduceNodes set, so paths cannot drift.
TestReAuthWithDifferentMachineKey asserted that a second machine claiming an existing NodeKey succeeds, but that hijack (now rejected by f8f08cf7) poisoned the NodeKey index and DoS'd the original node — the test only checked the hijacker's node, never the original's survival. Assert the registration is rejected and the original node survives intact.
noiseServer.nodeKey was assigned on every PollNetMap and Register request but never read anywhere. The inner HTTP/2 mux multiplexes concurrent requests over one Noise session, so those per-request writes to the shared field raced (caught by -race in servertest TestConnectDisconnectRace, reachable by a client issuing concurrent map requests). Remove the dead field and its two writes.
addNextDNSMetadata dereferenced node.Hostinfo().OS() without the .Valid() guard its siblings (RequestTags, TailNode) apply, so building the DNS config for a node with nil Hostinfo (a legacy NULL host_info row) panicked whenever a NextDNS resolver was configured. Guard the OS() dereference.
applyAuthNodeUpdate rotated the node's NodeKey to the client-supplied value without the 1:1 NodeKey/MachineKey check createAndSaveNewNode (f8f08cf7) and getAndValidateNode enforce. A re-authenticating node could thus rotate its key to a victim's and poison the NodeStore NodeKey index, denying the victim service. Apply the same uniqueness check on the re-auth path.
buildFromChange's PeersChanged path passed an unfiltered changed-node slice to WithUserProfiles, while the full-map path uses the BuildPeerMap-filtered ListPeers. A node thus received the identities (login name, display name, avatar) of users owning peers its ACL forbids accessing. Filter the UserProfiles peer set via the same ReduceNodes visibility check buildTailPeers applies.
buildFromChange added PeersChangedPatch (online/offline, endpoint, key-expiry) directly, skipping the policy.ReduceNodes visibility filter that buildTailPeers applies to full peers. A node thus received the existence, presence, and addresses of peers its ACL forbids accessing. Restrict patches to the recipient's visible peer set.
setCSRFCookie set no SameSite attribute despite a nolint comment claiming it did, so the OIDC state/nonce cookies relied on the browser default. Set Lax explicitly (Strict would drop the cookie on the cross-site IdP->callback navigation and break login), hardening browsers that do not default to Lax against OIDC login CSRF.
createAndSaveNewNode trusted the client NodeKey without checking it was already bound to a different machine, so an authenticated party could register a node carrying a victim's public NodeKey, poison the NodeStore NodeKey index, and make the victim's MapRequest resolve to the wrong node (rejected by getAndValidateNode = DoS). Enforce the 1:1 NodeKey/MachineKey binding at registration, as poll time already does.
The autogroup:self SSH-check branch dereferenced node.User().ID() guarded only by !IsTagged(); a non-tagged node with an unhydrated User (UserID set, association nil) crashed the server via the Noise SSH-check path. Gate on User().Valid() like filter.go, same shape as 171fd7a3.
ApplePlatformConfig still used gorilla mux.Vars after the chi
migration in 30338441, so every /apple/{platform} request returned
400 "no platform specified". Read the param via chi and add tests.
Fixes#3296
Don't send a full update when a subnet router goes up or down; the gated
policy change already recomputes peers and is a smaller payload.
Updates #3293
Connect and Disconnect appended change.PolicyChange() on every reconnect. PolicyChange sets RequiresRuntimePeerComputation, so the batcher rebuilt a full netmap (packet filters, SSH policy, peer serialization) for every connected node — O(N) per reconnect, O(N^2) on a restart storm. On a small VM this saturated CPU after the v0.28 -> v0.29 upgrade.
Emit it only when the node's online state changes what peers compute: subnet routers, relay targets, and via targets. An ordinary reconnect now sends just the lightweight online/offline peer patch. Relay and via targets still recompute, so peers drop a stale PeerRelay allocation when a relay goes offline.
Fixes#3293
Reports whether a node's online/offline transition forces peers to recompute their netmap. True for subnet routers, relay targets (tailscale.com/cap/relay), and via targets; false otherwise.
The relay-target IP set and via-target tag set are precompiled from the grants in updateLocked, alongside the existing filter, so the per-node check is a cheap set lookup. Keyed on the node itself, so an ordinary node in a tailnet that uses relay or via for other nodes is still classified as not needing a recompute.
Updates #3293
invalidateAutogroupSelfCache derived the owning user from
node.User().ID(), dereferencing the User association view. The
NodeStore stores nodes by value with User as a *User pointer, and not
every write path hydrates that association, so a non-tagged node could
reach SetNodes with UserID set but User nil. UserView.ID() then
dereferenced a nil *User and panicked on /machine/map whenever an
autogroup:self policy was active and a node restarted tailscaled.
Group affected nodes by node.TypedUserID(), which reads the UserID
field directly. UserID is the authoritative ownership field for
non-tagged nodes, so this needs no hydrated association and fixes every
.User().ID() site in the function.
Versions before 0.28 persisted a pointer to a zero time.Time as
'0001-01-01 00:00:00+00:00' in nodes.expiry rather than NULL. 0.29
reports those rows as expired. #3197 and #3199 stopped new writes;
this normalises the existing rows so the column once again means
"no expiry" when unset.
Fixes: #3284
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Untagged main-sha builds inherit a Go module pseudo-version from
runtime/debug.BuildInfo (vX.Y.Z-<timestamp>-<hash>). isDev only
filtered "", "dev", and "(devel)", so the pseudo-version was stored
in database_versions and the next real release tripped the
multi-minor upgrade guard:
headscale version v0.29.0-beta.1 cannot be used with a database
last used by v0.0.0-20260520093041-e4e742c776ee, upgrading more
than one minor version at a time is not supported
Add pseudoVersionTime, a regex + time.Parse predicate covering all
three Go pseudo-version forms (v0.0.0 base, pre-release ancestor,
release ancestor), and delegate from isDev. The dev gate at
db.go:790 already prevents pseudo-versions from being written, so
already-poisoned databases self-heal on the next real-release start.
Fixes#3281
ghcr.io/tailscale/tailscale:unstable is stale (last updated 2022,
points to v1.35.25). Only tailscale/tailscale on Docker Hub publishes
current unstable builds, so tsic.go reverts the unstable case and
build-tailscale-released pulls unstable from Docker Hub while keeping
release tags on ghcr.io.
Release tags on ghcr.io match Docker Hub by digest (verified v1.96),
so the rate-limit avoidance for the bulk of pulls is preserved.
Add the conditional docker/login-action step to the new job so the
main repo gets authenticated pulls; fork PRs fall through to anonymous
DH for the single unstable pull per CI run, well under the 100/6h
anonymous limit.
Fork PRs anonymous-pull tailscale/tailscale:vX.Y at test time and hit
Docker Hub rate limits, causing flakes. A new build-tailscale-released
job pulls the MustTestVersions set from ghcr.io once per CI run and
ships them to every test job as a gzipped tarball artifact, matching
the shape of the existing headscale/HEAD/postgres caches.
The version list is driven by 'hi list-versions' so capver is the
single source of truth; adding a version there propagates to CI
without YAML edits.
ghcr.io public reads need no auth, so the new job has no docker login
step.
Anonymous reads on ghcr.io are unmetered. Pulling tailscale/tailscale
from Docker Hub fails on fork PRs without DOCKERHUB_USERNAME because
the unauthenticated rate limit is hit at test time.
ghcr.io/tailscale/tailscale publishes the same tags. The cache-hit
short-circuit in dockertestutil.PullWithAuth still skips the pull when
the image is already loaded locally, so the change is a no-op once CI
pre-pulls the images.
Prints the tailscale versions integration tests use, with optional
filter and format flags. Source of truth stays in hscontrol/capver;
CI shells out to this instead of duplicating the version list in
YAML.
Examples:
hi list-versions
hi list-versions --set=must --exclude=head
hi list-versions --set=all --format=json
Regenerate capver for tailscale v1.98 — `MinSupportedCapabilityVersion`
slides 109 → 113 (v1.78 → v1.80). v1.80+ clients understand
`Node.HomeDERP`, which superseded `LegacyDERPString` upstream at capver
111. Drops the emit, plumbing, the trip-wire that caught this cleanup,
the golden test fixtures, and the integration check on
`peer.LegacyDERPString()` — the sibling `HomeDERP` check covers intent.
`v1.98` was added by hand to `capver_generated.go` because tailscale
skipped publishing v1.97/v1.98 to ghcr until late in the cycle; a clean
regeneration produces the same content.
Refresh direct deps (tailscale v1.98.1, modernc.org/sqlite v1.50.1 +
libc v1.72.3 lockstep, otel cluster v1.43.0, x/* family, pgx v5.9.2,
go-jose v3.0.5, grpc v1.81.1, grpc-gateway v2.29.0) and bulk-update
remaining direct deps. tailscale held at v1.98.1 since v1.98.2 demands
go 1.26.3 which nixpkgs unstable does not yet ship.
Move HA subnet router health probing above BREAKING so the layout
matches every other release. Drop **User deletion**: / **Node Expiry**:
bold prefixes redundant with the #### subgrouping. Fill missing PR
refs: #3202 (hostname rewrite), #3263 (sshTests + SSH rule validation),
#3194 (HA probe), #3251 (randomize_client_port removal), #3268
(trusted_proxies).
The HTTP/2 server inside the Noise tunnel fills r.RemoteAddr from the
hijacked TCP socket, so /machine/register and /machine/map logged the
reverse proxy's loopback peer (e.g. 127.0.0.1:44388) even with
trusted_proxies set. The outer router's realIPMiddleware had already
resolved the client IP onto req.RemoteAddr; that value never crossed
the hijack.
Replace the inner realIPMiddleware mount — dead inside the encrypted
tunnel — with overrideRemoteAddr(req.RemoteAddr) so requests served
over the tunnel report the outer-resolved client IP.
Every Go-identifier reference in // and /* */ comments now uses
godoc's [Name] linking syntax so pkg.go.dev and `go doc` render
them as clickable cross-references. No behaviour change.
Pattern applied across the tree:
In-package [Foo], [Foo.Bar]
Cross-package [pkg.Foo], [pkg.Foo.Bar]
Stdlib [netip.Prefix], [errors.Is], [context.Context]
Tailscale [tailcfg.MapResponse], [tailcfg.Node.CapMap],
[tailcfg.NodeAttrSuggestExitNode]
Skip rules:
- File:line refs left as plain text
- HuJSON wire keys inside backtick raw strings untouched
- ACL/policy syntax tokens (tag:foo, autogroup:self, ...) not Go
symbols, left as plain text
- JSON/OIDC wire keys, gorm tags, RFC IPv6 placeholders, markdown
link tags, decorative dividers — all left as-is
Splitting these functions does not buy clarity — each has been
extracted before and put back. Pin the //nolint:gocyclo on each
with the reason their shape resists clean decomposition.
policy/v2/policy.go ViaRoutesForPeer — three-pass
via-grant resolution
policy/v2/filter.go compileSSHPolicy — per-rule
branches with
intertwined
autogroup:self
handling
(annotated in the
earlier nil-error
commit)
state/state.go HandleNodeFromPreAuthKey — security-
sensitive
sequential
validation order
servertest/routes_test TestRoutes — table-driven
test driver with
many independent
subtests
Also: //nolint:recvcheck on policy/v2.SSHUser — UnmarshalJSON
requires a pointer receiver; the other methods on this string
newtype use value receivers by convention.
Each //nolint:gosec carries the gosec code and one line on why
the finding is a false positive or already mitigated.
G124 cookies (oidc.go x3, oidc_confirm_test.go)
Secure is set conditionally on req.TLS != nil; HttpOnly and
SameSiteStrictMode already on. gosec misses the conditional.
Test fixture cookie is explicitly a test fixture.
G705 (debug.go)
templates.PingPage(...).Render() is a templ component that
auto-escapes user input.
G706 (scenario.go)
Integration log emits trusted scenario state. The pre-built
image G706 sites in hsic.go / tsic.go ride along with the
earlier constants commit.
G710 (app.go, tailsql.go)
Redirect target is "trusted ServerURL prefix + path". gosec
cannot see past the prefix.
hscontrol/debug.go — pre-size nodes []nodeStatus to len(debugInfo)
so the loop does not grow under append.
hscontrol/mapper/batcher_test.go — testing.TB parameter on
setupBatcherWithTestData renamed t → tb so thelper sees the
expected name.
hscontrol/db/text_serialiser.go — reflect.Ptr → reflect.Pointer
(deprecated alias).
Constants the operator/test reader benefits from centralising.
Tests stay verbatim (the .golangci.yaml goconst tune skips them);
extraction here applies only where the same literal acts as
shared vocabulary across files.
cmd/headscale/cli/strings.go (new)
Cobra subcommand verbs and aliases shared by every list / show
/ new / delete / expire command across api_key, nodes, policy,
preauthkeys, users — plus the Result / Created / Expiration
column headers used in printOutput maps.
hscontrol/templates/design.go
cssBorderHS, cssBreakWord, cssCenter, cssOverflowWrap — shared
styles applied across design.go, ping.go, register_confirm.go.
spaceS already existed; switch raw "0.5rem" literals to it.
integration/hsic/hsic.go
binHeadscale, flagOutput, acceptJSON — names invoked across
hsic.go and config.go.
integration/tsic/tsic.go
tailscaleBin — used across docker exec call sites.
Two cleanups in the same package boundary:
policy/v2.Policy.compileFilterRules and compileFilterRulesForNode
returned (rules, error) where the error was always nil. Drop the
error return and the dead error handling at 15 call sites in the
compat tests.
Delete code with no callers:
hscontrol/metrics.go
prometheusMiddleware + respWriterProm — custom HTTP timer
middleware that was never registered. /metrics stays mounted
via debug.go and noise.go; mapresponse_* counters keep emitting.
httpDuration and httpCounter — backing metrics for the deleted
middleware.
hscontrol/policy/v2/filter.go
resolvedAddrsToPrincipals — superseded by ipSetToPrincipals.
ipSetToPrefixStringList — superseded by inline prefix
formatting at the surviving callers.
hscontrol/policy/v2/tailscale_{acl,grants}_data_compat_test.go
setupACLCompatNodes / setupGrantsCompatNodes — the data-driven
tests switched to per-scenario topology reconstruction.
Test fixtures repeat strings (IPs, tags, hostnames, user/email
names) by their nature; extracting each into a named constant
adds indirection without adding clarity. Keep goconst strict on
production code, off on tests.
ignore-tests: true drop test-fixture noise entirely
min-occurrences: 5 raise from default 3 to filter
"happens thrice" non-vocabulary cases
min-len: 6 skip short literals like "set", "get",
"new" that read better at call sites
Expand TestPrimaryRoutesProperty (5 -> 9 ops). New ops mirror the
production shapes the failure cases hit: BatchProbeResults via
UpdateNodes, SimultaneousDisconnect via UpdateNodes, SetApprovedRoutes
that leaves announced RoutableIPs intact, OfflineExpiry that keeps
Unhealthy set. The model now tracks announced and approved separately
and recomputes the intersection.
Strengthen the per-op assertions to cover invariants the model alone
cannot prove: every primary must be online, every primary must
currently advertise its prefix, no flap onto an unhealthy candidate
when a healthy one was available, no flap off a previous primary that
remains a healthy candidate. The check now takes a pre-op snapshot so
the anti-flap rule has a stable reference.
Add TestHAProberProperty in servertest. It drives a real TestServer
with three HA-route-advertising clients through rapid-drawn sequences
of ClientDisconnect / ClientReconnect / ProberTick / WaitForSnapshot
ops and re-checks the same shape invariants after every step.
Document the system in hscontrol/state/HA_INVARIANTS.md: a state
machine over (Healthy+Online, Unhealthy+Online, Offline,
OfflineExpired), fifteen numbered invariants with predicates and
violation paths, and a coverage matrix mapping each invariant to its
unit, servertest, and integration tests. Three rows pin the recent
fixes to the invariants they enforce.
electPrimaryRoutes' all-unhealthy fallback picked candidates[0] when
the previous primary was no longer a candidate. The Phase-5
simultaneous dual-disconnect path in TestHASubnetRouterFailoverDocker
Disconnect hits this asymmetrically: a batched probe cycle marks both
routers unhealthy with prev=r2 preserved, then the grace-period
Disconnect for r2 drops it from candidates. With prev gone and the
remaining r1 still carrying its Unhealthy bit, the fallback pointed
peers at the cable-pulled r1 — flapping primary to an unreachable
node and tripping requirePrimaryStable.
Leave the prefix unmapped when prev is gone and every candidate is
unhealthy. Peers see no advertiser instead of an unreachable one,
which is honest: the next probe cycle re-evaluates and picks
whichever node responds. The property-test model that mirrored the
old behaviour is updated to match.
requirePrimaryStable in TestHASubnetRouterFailoverDockerDisconnect
Phase 5a (simultaneous cable-pull of both routers) intermittently
caught the primary flipping to the offline r1. Both probe goroutines
mark their target unhealthy back-to-back; SetNodeUnhealthy publishes a
fresh NodeStore snapshot each call, so the intermediate snapshot — r1
unhealthy, r2 still healthy — runs the election with one healthy
candidate left and picks it. The next snapshot then enters the
all-unhealthy preserve-prev path, which preserves the wrong choice.
Collect probe results from the cycle and apply them through a new
NodeStore.UpdateNodes batched op so the election only runs once, with
the cycle's final health state. PolicyChange dispatch moves outside
the wg.Go goroutines and fires once if the primary assignment
actually changed.
The HA prober dispatches a PingRequest, waits ProbeTimeout (5s), and
marks the node unhealthy if no callback arrives. A node that bounced
its poll session between probe cycles satisfies two conditions that
conspire to fail TestHASubnetRouterFailover: a probe queued against
the previous session is silently dropped when the worker writes to
the closed connection (timeout always fires), and a probe sent
immediately after reconnect lands while wgengine is still rebuilding
magicsock state from the new netmap. Either path installs a spurious
unhealthy bit, which sends the preserved-primary anti-flap the wrong
way.
Record the session observed at dispatch time and drop the timeout
path if the node reconnected since. Require the session to survive
a full probe cycle before a timeout can drive a failover.
policyChangeResponse already includes everything else; carry DNSConfig
too so the client's netmap DNS is anchored on every policy change
rather than relying on the previous snapshot.
Send the MagicDNS root domains as empty non-nil Resolver slices instead
of nil values. tailcfg.DNSConfig.Clone and net/dns.Config.Clone in
tailscale drop map entries whose value is nil (tailcfg_clone.go and
dns_clone.go both contain `if sv == nil { continue }`). On a major
LinkChange the client's wgengine handler clones lastDNSConfig and
re-applies it; with nil values the cloned config has Routes:{}, dns.Set
wipes Nameservers in /etc/resolv.conf, and curl-by-FQDN fails until
the next route-changing netmap, typically about six minutes later.
Empty slice survives Clone and carries the same "resolve locally"
semantics for Routes entries.
tsic.Curl returned ("", nil) when curl exited 0 with a zero-byte body —
the usual signature of a mid-stream reset — so EventuallyWithT could
not retry. Return an error on empty body instead.
Replace the 56 inline curl + assert.Len(13) blocks with
assertCurlDockerHostname so the empty-body fix benefits every callsite
without further touch-ups.
Gate ACL waits on actual filter visibility: snapshotClientFilters +
waitForClientFilterChange ensure the new PacketFilter has reached the
client before assertions fire; SyncOption + WithPreBarrier feeds a
server-side policy-loaded check into WaitForTailscaleSyncPerUser.
Move advertise-routes mutation out of EventuallyWithT in route_test
(cmd/hi/README forbids retrying state-mutating calls). Pace the
TestNodeOnlineStatus outer loop with a Ticker, not a Sleep.
Categorised timeouts in integrationutil/timeouts.go remove the drift
opportunity between same-purpose budgets repeated across the suite.
The auth, cli, dns, derp, ssh, and tags tests are swept; acl, route,
and general tests follow in later commits alongside their other
ergonomic fixes.
DisconnectContainerFromNetwork and ReconnectContainerToNetwork
returned as soon as the docker API call completed, but libnetwork
bridge reprogramming continued for several seconds after. The HA
disconnect tests then raced and bounced between healthy and broken
bridges. Poll until the container's endpoint is gone (on
disconnect) or reconciled (on reconnect), and on the
"conflicts with existing route" surface clear the stale subnet
route from the netns and retry. Settle is now baked into the
primitive so every caller benefits.
Anonymous Hub pulls trip the 100/6h IP cap on shared CI runners, turning
into singleton FAIL reports whenever the runner egress IP crosses the
quota. Route every pull through Docker Hub credentials when present, and
retry transient errors with backoff. tsic and hi use the same helper so
both surfaces honour ~/.docker/config.json and the GHA secrets.
Docker 29 itself works; the breakage on the GHA runner image was the
overlayfs default. Setting storage-driver=overlay2 restores the
long-standing default and lets the suite run on the current Docker
without the apt downgrade dance.
Fixes#3094
chi middleware.RealIP was mounted unconditionally on both the
public router and the noise router, so any client could send
X-Real-IP or X-Forwarded-For and have the spoofed value land in
r.RemoteAddr and the access-log remote= field.
Add a top-level trusted_proxies config option (list of CIDRs) and
replace middleware.RealIP with a gated middleware that:
- honours True-Client-IP / X-Real-IP / X-Forwarded-For only when
r.RemoteAddr is inside one of the configured prefixes;
- strips those three headers from every request whose peer is
not trusted, so downstream handlers cannot read them.
X-Forwarded-For is parsed via realclientip-go's
RightmostTrustedRangeStrategy so a prepended value cannot win in a
proxy chain. trustedProxies() rejects 0.0.0.0/0 and ::/0 at config
load.
Empty trusted_proxies (the default) skips the mount entirely;
r.RemoteAddr is the directly-connecting TCP peer.
TestGrantViaSubnetFilterRules pins exact-equality dst. Add a sibling
for the broader-dst case so the regression sits at the server level
alongside the policy-engine unit test.
Updates #3267
slices.Contains required exact equality between grant dst and the
advertised subnet route. Any non-identical pair was rejected, so a
via grant with broader (or narrower) dst emitted no filter rule and
added no route to the viewer's AllowedIPs. Tailscale SaaS uses
containment in either direction.
Switch to slices.ContainsFunc(routes, dst.Overlaps) for filter rule
emission (keep dst literal in DstPorts), and append overlapping
advertised routes to ViaRoutesForPeer.Include / Exclude. Rewrite the
multi-router HA election and regular-grant overlap detection to key
off the matched routes rather than the dst. Resolve *Host aliases to
*Prefix once in compileOneViaGrant and at the top of ViaRoutesForPeer
so the switch arms reach them.
Fixes#3267
Both helpers existed to write the literal "[]" when clearing a slice
column — a workaround for GORM's struct-Updates skipping nil slices.
The State path goes exclusively through persistNodeToDB, which is now
correct end-to-end thanks to the named IsZero slice types, so the
helpers are dead in production. The remaining callers were tests.
TestSetTags is dropped — TestSetTags_* in hscontrol/grpcv1_test.go
already covers the State path that production uses. TestAutoApproveRoutes
now writes routes via DB.Save on the loaded node, which is the path
gRPC SetApprovedRoutes drives in production.
Updates #3110
Drives the persist path for ApprovedRoutes, Tags and Endpoints —
seed a non-empty value, clear to nil, read the column back from disk,
then close the State and reopen one against the same sqlite file to
simulate a server restart. Pins the contract the named IsZero slice
types enforce so future changes to the persist path cannot silently
drop a cleared slice column.
Updates #3110
Endpoints, Tags and ApprovedRoutes serialize as JSON on Node. GORM's
struct Updates path skips fields it considers zero, and reflect treats
a nil slice as zero — clearing any of these columns via the State
persist path would leave the previous value in the database.
Introduce Strings, Prefixes and AddrPorts as named slice types whose
IsZero() always reports false, so GORM keeps the column in the UPDATE
regardless of the slice being nil or empty. JSON marshalling is
unchanged: nil serializes to null, empty to []. List() returns the
underlying unnamed slice for callers (mainly testify assertions over
reflect.DeepEqual) that distinguish the named type from its base.
Regenerated types_clone.go and types_view.go follow the field-type
swap. Test assertions across hscontrol/{db,state,servertest} updated
to call .List() where reflect.DeepEqual previously matched the raw
slice type.
Fixes#3110
Replays recorded policy responses for the sshTests block. 200 captures must evaluate; non-200 captures must reject with the recorded body as a substring of the headscale error. Divergences are listed in knownSSHTesterDivergences.
SetPolicy and policy check now compile per-dst SSH rules and replay each sshTests entry. The accept assertion treats check-action rules as reachable; the check assertion requires HoldAndDelegate on the matching rule. Boot reload warns and continues.
Adds SSHPolicyTest plus parse-time validation: empty src/dst, port/CIDR/autogroup-internet destinations, and tag references missing from tagOwners are rejected. Engine evaluation comes in a follow-up.
The guard added for #2862 in handleRegister checked
node.Expiry().Valid() before preserving node state on
Auth=nil + Expiry=zero registration requests. Valid() returns false
when node.Expiry is nil, the default for tagged nodes and for untagged
nodes registered against a preauth key with no default node.expiry
configured. Both fell through to handleLogout, which wrote
&time.Time{} (0001-01-01T00:00:00Z) over the original nil — the
user-visible 0001-01-01 expiry that `headscale nodes list` reports
after restart.
IsExpired() already returns false for both nil and zero-time, so the
Valid() check was redundant. Drop it so all nil-expiry nodes are
covered by the same preservation path.
Fixes#3170Fixes#3262
When a node carries the disable-ipv4 nodeAttr documented at
https://tailscale.com/docs/reference/troubleshooting/network-configuration/cgnat-conflicts,
SaaS stops sending the node's CGNAT IPv4 prefix in MapResponse. The
allocator keeps assigning IPv4 server-side; only the wire-shape
delivery is filtered. Subnet routes the node advertises -- including
IPv4 prefixes -- survive in AllowedIPs and PrimaryRoutes.
TailNode now drops Is4 prefixes from Addresses and from the node's
own /32 slot in AllowedIPs when selfPolicyCaps carries
disable-ipv4. Mapper.buildTailPeers passes each peer's policy
CapMap so the filter applies in viewer netmaps too; the CapMap
merge that follows is overwritten by PeerCapMap so only the address
filter survives on the peer path.
Two captures land in testdata/nodeattrs_results to anchor the
behaviour:
- nodeattrs-attr-c15-disable-ipv4 (on tag:client)
- nodeattrs-attr-c16-disable-ipv4-router (on tag:router, which
advertises 10.33.0.0/16, confirming subnet routes survive)
Tailscale stamps tailcfg.NodeAttrDefaultAutoUpdate on every node's
CapMap with a JSON bool reflecting the tailnet-wide auto-update
default. Headscale grows an auto_update.enabled config option and
emits the cap accordingly from TailNode -- the cap leaves the
unmodelledTailnetStateCaps strip list and is compared in full by the
nodeAttrs compat suite.
testNodeAttrsSuccess drives cfg.AutoUpdate.Enabled from
tf.Input.Tailnet.Settings.DevicesAutoUpdatesOn so each capture's
expected emission matches the SaaS state it was taken under. Two
captures cover both branches:
- nodeattrs-tailnet-devices-auto-updates-on -> [true]
- nodeattrs-tailnet-devices-auto-updates-off -> [false]
The Tailscale v2 TailnetSettings API does not expose the Send Files
toggle, so the compat suite cannot vary cfg.Taildrop.Enabled per
capture. TestTaildropDisabledWithholdsFileSharingCap covers the off
path directly in servertest.
Taildrive (drive:share and drive:access) is policy-driven per
Tailscale's documented behaviour
(https://tailscale.com/docs/features/taildrive). The previous
always-on baseline emission diverged from SaaS for every node not
targeted by a drive nodeAttr -- a real semantic divergence that the
compat suite caught once the test moved to comparing TailNode output
against the captured netmaps.
types.Node.TailNode no longer stamps the drive pair. Operators
wanting taildrive add a nodeAttrs entry:
"nodeAttrs": [
{ "target": ["*"], "attr": ["drive:share", "drive:access"] }
]
unmodelledTailnetStateCaps shrinks accordingly. The baseline-divergence
group is gone; every entry left in the list is genuinely unmodelled
(user-role caps, unimplemented features, tailnet metadata, internal
tuning).
servertest's TestNodeAttrsBaselineCapsAlwaysOn expects the smaller
baseline (admin + ssh + file-sharing). Integration TestGrantCapDrive
grants the drive caps explicitly via NodeAttrs to exercise the
policy-driven emission path.
types.NodeView.TailNode takes a selfPolicyCaps tailcfg.NodeCapMap
parameter and merges it into the baseline. The mapper's WithSelfNode
hands it the policy result via state.NodeCapMap; peer-path callers
pass nil because peer-side CapMap is set downstream via
policyv2.PeerCapMap.
The nodeAttrs compat test now diffs the full TailNode self-view
output against captured SaaS netmaps. Before this change the test
compared compileNodeAttrs alone -- the policy-only output -- and
needed a strip list to compensate for the missing baseline. With
TailNode on the diff path, baseline emission is exercised end-to-end
by every capture; a regression in TailNode breaks the suite.
unmodelledTailnetStateCaps drops cap/ssh and cap/file-sharing now
that both sides emit them identically. The file header is rewritten
to read as 'caps SaaS emits where headscale has no equivalent yet'
rather than the more confusing 'shape divergence' framing.
The Tailscale client surfaces 'use this peer as your exit node' when
the peer's CapMap carries the tailcfg.NodeAttrSuggestExitNode cap.
SaaS emits it only on peers whose advertised exit routes are
approved -- not every peer that just has the cap in its own
nodeAttrs slot.
policyv2.PeerCapMap encodes that emission rule: it walks the
peer's own self-CapMap (built from compileNodeAttrs) and surfaces
the gated entries (today just suggest-exit-node when the peer
IsExitNode). Mapper.buildTailPeers calls it for each peer instead
of merging the peer's full nodeAttrs CapMap onto its peer view.
allCapMaps snapshots the full per-node CapMap once per peer-list
build so pm.mu is acquired once rather than per peer.
Adds a data-driven test that loads testdata/nodeattrs_results/*.hujson
and diffs the captured SaaS-rendered netmaps against headscale's
compileNodeAttrs output. Each capture is one scenario the SaaS
control plane has rendered against the same policy headscale is asked
to compile -- the test enforces shape parity per node.
tailnet_state_caps.go enumerates the caps SaaS emits where headscale
has no equivalent concept yet (user-role admin/owner, tailnet lock,
services host, app connectors, internal magicsock and SSH tuning,
tailnet-state metadata) plus the always-on baseline (admin, ssh,
file-sharing) and the taildrive pair. stripUnmodelledTailnetStateCaps
filters both sides of cmp.Diff so the comparison focuses on the
policy-driven caps. PeerCapMap encodes which caps the Tailscale
client reads from the peer view (suggest-exit-node when exit routes
are approved, etc.) for use by the mapper.
testcapture switches to typed tailcfg/netmap/filtertype/apitype
values so schema drift between the capture tool and headscale
becomes a compile error rather than a silent test failure. Existing
compat suites (acl, grants, routes, ssh, issue_3212) move to the
typed shape.
The 53 SelfNode netmap captures and the 7 anonymizer-corrupted
suggest-charmander -> suggest-exit-node restorations in
routes_results / issue_3212 ride along.
Tailscale models the randomize-client-port toggle as a top-level
field on the ACL policy. Headscale now matches that shape: the
server-config randomize_client_port key is removed, the toggle
lives in the policy file as randomizeClientPort, and per-node
opt-in via nodeAttrs is also supported.
Operators upgrading from a config-set randomize_client_port hit
depr.fatalWithHint at startup, which prints the deprecation message
and points at the new policy field rather than silently dropping
the toggle. The default carries over (false) so operators who never
set it are unaffected. config-example.yaml ships a REMOVED stanza
showing the migration.
types/node.go drops the cfg.RandomizeClientPort read from
TailNode -- the cap is now policy-driven through compileNodeAttrs
and the tail_test.go expectations follow.
WithSelfNode and buildTailPeers merge each node's policy CapMap
into the tailcfg.Node.CapMap they emit. State.NodeCapMap and
State.NodeCapMaps wrap the policy manager: NodeCapMap returns a
defensive clone per call; NodeCapMaps snapshots the full per-node
map once for batched callers, amortising pm.mu acquisition across
a peer build.
generateDNSConfig grew a per-node CapMap argument so it can apply
nodeAttr-driven DNS overlays. The nextdns DoH rewrite hardens against
policy-controlled inputs:
- nextDNSDoHHost anchors the prefix match instead of substring,
so a hostile resolver URL cannot smuggle a nextdns hostname in
a path or query.
- nextDNSProfileFromCapMap accepts only profile names matching
[A-Za-z0-9._-]{1,64} and picks the lexicographically first when
multiple are granted -- deterministic, no shell metacharacters
or URL fragments through.
- addNextDNSMetadata composes the rewritten URL via url.Parse +
url.Values rather than fmt.Sprintf, so existing query strings
on the resolver URL survive and metadata cannot inject a new
component.
WithTaildropEnabled in servertest controls cfg.Taildrop.Enabled per
test so cap/file-sharing emission can be toggled in tests that need
to verify the off path.
ACL policies now accept a top-level nodeAttrs block. Each entry hands
a list of tailcfg node capabilities to every node matching target.
Accepted target forms are the same as acls.src and grants.src: users,
groups, tags, hosts, prefixes, autogroup:member, autogroup:tagged,
and *. autogroup:self, autogroup:internet, and autogroup:danger-all
are rejected at validate time because none describes a stable
identity set a node-level attribute can attach to.
NodeAttrGrant carries Targets, Attrs, and IPPool. IPPool is parsed
but rejected at validate time -- the allocator that consumes it is
not yet implemented. nodeAttrUnsupportedCaps lists caps SaaS accepts
that headscale cannot act on (funnel today) and rejects them with a
tracking-issue link in the error.
compileNodeAttrs resolves each entry's targets, then maps every
targeted node to a tailcfg.NodeCapMap of the entry's attrs. Per-node
IPs are cached once per call so the inner attr loop is O(grants)
instead of O(grants * nodes) IP allocations.
PolicyManager grows NodeCapMap (per-node), NodeCapMaps (snapshot for
batched callers), and NodesWithChangedCapMap (drain buffer for the
self-broadcast diff). refreshNodeAttrsLocked appends to the drain
rather than overwriting so a SetUsers/SetNodes between SetPolicy and
the drain cannot lose the policy-reload diff.
- Mention policy as generic term that covers ACLs or Grants
- Refresh routes policy examples
- Remove Headscale specific exit node separation. Use via instead.
Fixes: #3087
- Rename from acl.md to policy.md and setup redirect links
- Mention both ACLs and Grants
- Remove most old ACL docs and replace with links to Tailscale docs
- Add "Getting started" section
- Add section about notable differences
The policy `tests` block lets entries omit `proto`. Tailscale's client
maps that to the default protocol set {TCP, UDP, ICMP, ICMPv6} — the
captured packet_filter_matches show all four IANA numbers explicitly
when no proto is set — and a rule restricted to any one of them
satisfies an empty-proto reachability test.
srcReachesDst was passing the empty Protocol through unchanged, which
landed an empty []int in ruleMatchesProto. The matcher then short-
circuited to "no match" for every rule with a non-empty IPProto
restriction, including TCP-only grants compiled from `ip: ["tcp:80"]`.
The bug surfaced in the captured allpass-acls-and-grants-mixed
scenario: the grant `tag:client → webserver:80` was reachable in the
compiled filter but the empty-proto test could not see it.
Expand the empty Protocol to the default set at the call site so
ruleMatchesProto's intersection check sees the right requested
protocols. Drop the now-dead empty-requestedProtos branch from the
matcher. The last divergence drops out of knownPolicyTesterDivergences
as a result.
Updates #1803
Tailscale accepts both named ("tcp") and numeric IANA ("6") protocol
forms wherever a Protocol value is allowed. Headscale stored whichever
form the user wrote, leaving downstream code with two equivalents to
handle separately. validateProtocolPortCompatibility only recognised
the named constants and rejected the numeric form, so a policy with
`proto: "6", dst: ["host:443"]` was rejected at parse time even though
SaaS accepts it.
Resolve the disagreement by normalising to the named form during
Protocol.UnmarshalJSON. Every downstream consumer now sees one form
regardless of what the user wrote, so layered guards like
`|| protocol == "6"` in the validator are unnecessary.
Updates #1803
A `tests` entry describes one connection attempt to one specific
host on one specific port over a connection-oriented protocol, and
asserts whether it is allowed or denied. Five shape rules follow —
single-port dst, proto in {tcp, udp, sctp, ""}, no
autogroup:internet dst, no CIDR-typed dst (raw `/N` or hosts:-alias
to a multi-host prefix), at least one of accept/deny — and every
one was previously silently accepted by headscale even though
Tailscale SaaS rejects them as "test(s) failed".
Enforce them in one pass over `pol.Tests` from `Policy.validate()`,
reusing the existing parse-time multierr aggregation. The same
shapes remain valid inside ACL or Grant destinations where the rule
does not apply; the validator only walks the tests array.
The compat runner now treats parse-time errors equivalently to
SetPolicy errors so the captured Tailscale body still matches via
substring regardless of which step surfaces the rejection. Nine
divergences resolved by this validation pass drop out of
knownPolicyTesterDivergences.
Updates #1803
57 captures covering the alias × outcome matrix for the tests block,
recorded against a real Tailscale SaaS tailnet. Replayed by
TestPolicyTesterCompat.
Bump the check-added-large-files pre-commit threshold to 1024 KB —
captures include verbose per-node netmaps and one is 620 KB.
Updates #1803
Pin headscale's accept/reject decision and error body against
Tailscale SaaS by replaying captures recorded from a real tailnet.
Mirrors the tailscale_grants_compat_test.go pattern: glob over
testdata/policytest_results/, one t.Run per file, parse-or-SetPolicy
error must contain the captured api_response_body.message.
errPolicyTestsFailed is "test(s) failed" — Tailscale's literal body —
so substring match works against captured response bodies. Per-test
detail (src, dst, expected vs got) is preserved below the prefix for
the CLI / config-reload paths that don't have an audit endpoint.
knownPolicyTesterDivergences gates the 12 mismatches the captures
will surface so the suite stays green; engine fixes in follow-up
commits drop the entries as each is resolved.
Updates #1803
v2 silently dropped policy.tests, so a policy that contradicted its
own assertions still applied. Resolve src/dst via the existing Alias
machinery, walk the compiled global filter rules (acls and grants
both contribute), and run on every user-write boundary: SetPolicy,
the file watcher, and `headscale policy check`. A failing test
rejects the write before it mutates live state.
Boot-time reload skips evaluation; an already-stored policy that
references a deleted user shouldn't lock the server out.
`headscale policy check` is a thin frontend for the new CheckPolicy
gRPC method. The server-side handler builds a fresh PolicyManager
from the request bytes and the state's live users/nodes, runs
SetPolicy on the sandbox so the tests block executes, and returns
the result through gRPC status. No persistence, no policy_mode
coupling. --bypass-grpc-and-access-database-directly opens the DB
directly when the server is not running.
cmd/headscale/cli/root.go no longer special-cases `policy check` in
init() (the early return from PR #2580 broke --config registration
and viper priming for --bypass).
integration/cli_policy_test.go covers policy_mode={file,database} x
fixture={acl-only, acl+passing-tests, acl+failing-tests} x
bypass={false,true} = 12 rows.
Updates #1803
Co-authored-by: Janis Jansons <janhouse@gmail.com>
CheckPolicy validates a candidate policy against a running server's
live users and nodes (running its tests block) without persisting
anything. Used by 'headscale policy check' to replace the in-process
validation path the CLI runs today, which would otherwise need its
own database connection.
Updates #1803
MatchFromFilterRule only read DstPorts[].IP into the destination
IPSet. Cap-grant-only filter rules (e.g. tailscale.com/cap/relay)
carry their destinations in CapGrant[].Dsts, so the derived matchers
had empty dest sets and BuildPeerMap / ReduceNodes never exposed the
cap target to its source nodes. Without a companion IP-level grant
the relay node stayed invisible, so clients never tried to use it
and connections sat on DERP.
Union CapGrant[].Dsts into the destination IPSet alongside DstPorts.
Restores peer-visibility for any cap-grant-only relationship; the
peer-relay flow is the most visible instance.
Fixes#3256
revgrep needs pull_request.base.sha in the local clone to compute
the diff against new code. With fetch-depth: 2, only HEAD and one
parent are fetched, so a stale base SHA (when main moves between
PR syncs) is not reachable and revgrep falls through, surfacing
pre-existing issues outside the PR scope.
Replace the grep/awk hash extraction in build.yml with a structured
vendorhash check step; the PR review comment now reads expected/
actual values directly from $GITHUB_OUTPUT instead of scraping Nix
stderr. Add a prek hook so divergence is caught locally before push.
Move the headscale vendorHash out of flake.nix into a content-
addressed flakehashes.json maintained by a small Go tool. The
schema and goModFingerprint algorithm mirror upstream tailscale's
tool/updateflakes so a future shared library extraction is trivial.
vendorhash check verifies flakehashes.json against the current
go.mod/go.sum. Hot path is a sha256 over those two files, so
re-runs without input change are essentially free; only an actual
fingerprint drift triggers go mod vendor + nardump.SRI.
vendorhash update recomputes both fields and rewrites the JSON.
The nix-vendor-sri devShell shim now wraps it.
Pulls in the cmd/nardump library split (tailscale/tailscale#19551)
so flakehashes.json tooling can import nardump.SRI directly.
Side effects: Go directive bumps to 1.26.2 and the nixpkgs lock
advances to a revision shipping go 1.26.2.
Add --bypass-grpc-and-access-database-directly to policy check so
the new ambiguous-user validator runs against the live user list.
Without the flag, policy check stays a syntax-only check and the
success message says so.
Updates #3160
When a user@ token resolved to more than one DB row, ACL and SSH
rules referencing it were silently dropped at compile time, leaving
clients with SSHPolicy={rules: null} and no signal to the admin.
Validate every Username reference in groups, tagOwners,
autoApprovers, ACLs and SSH rules at NewPolicyManager and SetPolicy
and return ErrMultipleUsersFound. Missing-user tokens stay tolerant
per #2863.
Updates #3160
Mirror the guard from HandleNodeFromPreAuthKey in HandleNodeFromAuthPath.
Both functions log the old user's name in the "different user" branch
when an existing NodeStore entry under the same machine key belongs to
another user. UserView.Name dereferences the backing User pointer
unconditionally, so when the cached node was loaded with a non-nil
UserID but a nil User (Preload join missed the row, or upstream code
left the snapshot in that shape), the log call panics with a nil-pointer
dereference at hscontrol/types/types_view.go:97.
The panic is caught by the http2 server's runHandler for the noise
control plane, so the process keeps running but every retry produces a
new panic — production has observed bursts of ~1.9k panics per hour
during a tailscaled reconnect loop. The gRPC/OIDC entry has no equivalent
recover and would surface the panic to the caller.
Guard both call sites with oldUser.Valid() and fall back to an empty
old-user name when the pointer is nil. The "Creating new node for
different user" log line still includes the existing node ID, hostname,
machine key, and new user, so operator visibility is preserved.
Add reproduction tests for both handlers seeding the orphan shape
directly into NodeStore via PutNodeInStoreForTest.
Co-Authored-By: Kristoffer Dalby <kristoffer@dalby.cc>
The 39 SSH-*.hujson files in hscontrol/policy/v2/testdata/ssh_results/
were legacy hand-written "expected SSH rules" snippets superseded by
the lowercase tscap captures (ssh-*.hujson). The active loader in
TestSSHDataCompat globs ssh-*.hujson; filepath.Glob is case-sensitive
on Linux so the uppercase set was loaded by no test.
The duplication caused permanent dirty git state on case-insensitive
filesystems (APFS, NTFS) where only one of SSH-A1.hujson and
ssh-a1.hujson can physically exist in the working tree.
Add an assertion to TestSSHDataCompat that the loader picks up every
*.hujson under ssh_results/ so future fixture migrations cannot leave
stranded files behind.
Fixes#3240
Some OIDC providers (notably JumpCloud) return the `groups` claim as
a plain string when the user belongs to a single group, rather than
a single-element array:
Single group: {"groups": "MyGroup"}
Multiple groups: {"groups": ["Group1", "Group2"]}
This causes `json.Unmarshal` to fail with:
cannot unmarshal string into Go struct field OIDCClaims.groups of type []string
This is the same class of issue as juanfont#2293 (FlexibleBoolean for
email_verified). The fix follows the same pattern: introduce a
FlexibleStringSlice type with a custom UnmarshalJSON that accepts
both a string and a []string, and use it for the Groups field in
both OIDCClaims and OIDCUserInfo.
TestGrantViaExitNodeInternetVisibility boots a server, applies a
policy that scopes autogroup:internet to a tag, registers a tagged
exit advertiser and a regular client, and asserts the client's netmap
surfaces the exit node with 0.0.0.0/0 and ::/0 in AllowedIPs — the
substrate the Tailscale client reads to populate
`tailscale exit-node list`.
TestGrantViaExitNodeNoFilterRules retains its assertion (literal /0
absent from the exit node's PacketFilter, matching SaaS PacketFilter
encoding); only its docstring is updated to reflect that the exit
node now does receive a TheInternet-shaped rule, just not the
literal /0 form.
Updates #3233
A grant of the form `{src: alice, dst: autogroup:internet, via:
tag:exit1}` was loading without error but stripping every exit node
from alice's view: `tailscale exit-node list` returned "no exit nodes
found".
Two sites skipped autogroup:internet at the compile / steering layer:
compileViaForNode's *AutoGroup arm produced no FilterRule for the
via-tagged exit node, and ViaRoutesForPeer's *AutoGroup arm produced
no Include/Exclude. With pm.needsPerNodeFilter true, the exit node's
matchers were empty, BuildPeerMap could not link source to exit, and
RoutesForPeer's ReduceRoutes stripped 0.0.0.0/0 and ::/0 from
AllowedIPs.
The skip belongs at the wire-format layer (ReduceFilterRules), not at
the compile layer that also feeds internal matchers. Lift
autogroup:internet handling into both *AutoGroup arms with the same
shape used for *Prefix destinations: emit a TheInternet rule on
via-tagged exit advertisers; surface peer.ExitRoutes() in Include
when the peer carries the via tag, Exclude otherwise.
ReduceFilterRules continues to keep the rule on exit-route
advertisers' wire output and strip it elsewhere, preserving SaaS
PacketFilter encoding.
Also drop compileViaForNode's early len(SubnetRoutes)==0 return:
SubnetRoutes excludes exit routes, so the early return pre-empted the
autogroup:internet branch on nodes that only advertise exit routes.
Existing tests pinning the buggy behaviour (TestViaRoutesForPeer
subtests, TestCompileViaGrant case) flipped to the new contract.
Fixes#3233
Force-disconnect leaves stale routes in the container's network
namespace: libnetwork removes the host-side veth but the
namespace-internal route survives. The next ConnectNetwork on the
same network then fails with "cannot program address X/16 in sandbox
interface because it conflicts with existing route", and the route
never resolves on its own. Bounded retry around ConnectNetwork
exhausts MaxElapsedTime instead of recovering.
Without Force, libnetwork drains the namespace routes synchronously
during disconnect and ConnectNetwork sees a clean slate. Cable-pull
semantic is preserved: docker still tears down the endpoint at the
namespace level, leaving in-flight TCP half-open inside the
container's view, verified via paired probe-timeout pairs in HA
prober logs while both routers are physically disconnected.
Fixes#3234
`time.After(ProbeTimeout)` returned a single channel shared by every
probe goroutine in the cycle. Only the first goroutine to receive the
deadline tick drains the channel; any other goroutine still waiting on
its `responseCh` is then stuck forever, `wg.Wait()` never returns, and
the scheduler loop in `app.go` stalls on the next tick. The condition
fires whenever two or more nodes time out in the same cycle — common
under cable-pull where IsOnline lags reality and both routers stay in
the candidate set as half-open TCP.
Move the timer inside each goroutine so every probe has its own
deadline.
Updates #3234
The generator scans `integration/` recursively for `Test*` functions
and emits one CI job per match. Helper subpackages like
`dockertestutil` and `tsic` host plain unit tests that should run
under `go test`, not as Docker-based integration matrix entries.
Limit the scan to depth 1 so only top-level `integration/*_test.go`
files contribute job names.
Libnetwork endpoint cleanup is eventually consistent. A back-to-back
disconnect+connect on the same network can race teardown and return a
transient error. Wrap the daemon calls in bounded exponential backoff
so TestHASubnetRouterFailoverDockerDisconnect no longer flakes on
phase 4c reconnect.
Fixes#3234
electPrimaryRoutes' all-unhealthy fallback picked candidates[0]
(lowest NodeID) regardless of who was prev. Under cable-pull
semantics IsOnline lags reality (long-poll TCP half-open), so
both routers stay in candidates and both go Unhealthy via the
prober — the fallback then churned primary to a node that was
itself unreachable.
Prefer prev when still in candidates; fall through to
candidates[0] only when prev is gone. Anti-blackhole holds.
Update the property test reference model and split the unit
test into existence (KeepsAPrimary) and identity
(PreservesPrevious) cases.
Fixes#3203
Add DisconnectFromNetwork/ReconnectToNetwork on TailscaleClient
backed by pool.Client.DisconnectNetwork.
Exercise single-router fail+recover either side, sequential dual
failure, and simultaneous dual failure. The dual-failure legs
assert no flap to a known-bad primary; the single-router-return
legs check traffic only because docker network disconnect
transiently fails probes on sibling routers.
Fails on parent; passes after the fix.
Updates #3203
Three regression tests for the user scenario: an in-process
Disconnect/Reconnect, a tailscale-down/up integration test, and
an iptables -j DROP cable-pull integration test.
Updates #3203
Restore the legacy auto-clear at write boundaries that drop HA
candidacy: Disconnect, SetApprovedRoutes(empty), and
UpdateNodeFromMapRequest shrinking advertised routes to empty.
Plus a defensive guard in SetNodeUnhealthy.
Updates #3203
Replace routes.PrimaryRoutes reads with NodeStore. Connect bumps
SessionEpoch; Disconnect re-checks it inside UpdateNode so the
check and mutation are atomic against a concurrent Connect on
the same node.
The connect_race regression test is carried in its final
SessionEpoch form.
Updates #3203
- test: comment that the !regReq.Expiry.IsZero() gate also covers
the tags-only PreAuthKey path
- CHANGELOG: note pre-existing 0001-01-01 rows self-heal on
re-registration rather than being backfilled
When a user owned node registers or re registers with a PreAuthKey and the
client sends zero client expiry while node.expiry is set to 0, the expiry
column ends up stored as 0001-01-01 00:00:00 instead of NULL. Two sites in
HandleNodeFromPreAuthKey build a non nil pointer to regReq.Expiry even when
the value is zero time, and the needsDefaultExpiry guard only replaces it
when s.cfg.Node.Expiry > 0, so the pointer to zero time survives to the
database.
Convert an unset regReq.Expiry to nil before handing it off so the
needsDefaultExpiry path is the only place that assigns a non nil pointer.
This is a narrower sibling of #3170 on the user owned PreAuthKey path. The
regression was introduced alongside the fix for #3111 in 6337a3db.
TestEnablingExitRoutes runs without an ACL, so tailcfg.FilterAllowAll
hides any policy-path regression. Add a sibling that applies the literal
#3212 policy via hsic.WithACLPolicy after registration and approval,
then asserts each peer carries 0.0.0.0/0 + ::/0 in AllowedIPs and
ExitNodeOption is true — the daemon-derived bool that drives
`tailscale exit-node list`.
Updates #3212
compileFilterRules skipped autogroup:internet destinations to keep them
out of the wire-format PacketFilter, but those same compiled rules are
the source of pm.matchers — and Node.CanAccess relies on a matcher whose
DestsIsTheInternet covers the public internet to surface exit-node peers
to ACL sources. With the skip in place no such matcher existed, exit
nodes silently dropped out of the source's peer list, and the docs'
exit-node walkthrough stopped working: `tailscale exit-node list`
returned "no exit nodes found" and `tailscale set --exit-node=<ip>`
returned "no node found in netmap with IP".
Drop the compile-time skip so autogroup:internet flows through normal
matcher derivation, and teach ReduceFilterRules to keep the resulting
client packet-filter rule on exit-route advertisers — Tailscale SaaS
sends those rules to exit nodes so the kernel filter accepts traffic
forwarded by autogroup:internet sources.
Verified against a live tailnet on 2026-04-28 via tscap; the b17/b18
captures land under testdata/issue_3212/ as a regression guard. The
captures are isolated from testdata/routes_results/ because the broader
TestRoutesCompat machinery assumes a CIDR-prefix wire format that
differs from the IPSet-range form SaaS emits for autogroup:internet —
aligning that wire format is tracked separately.
Fixes#3212
Add an integration test that runs the tailscale-rs axum example
against headscale end-to-end. The test provisions one headscale
instance, one Go tailscale probe client (tsic), and one
tailscale-rs node (tsric) on the same tailnet, then verifies:
- the tailscale-rs node registers and is assigned both an IPv4
and an IPv6 address
- the Go probe sees the tailscale-rs node as a peer in its status
- GET /index.html and GET /assets/index.css from the axum server
return the expected content over the tailnet
- three sequential POST /count calls return distinct, incrementing
counter values, proving netstack state is maintained across
multiple TCP connections
This is the first integration test that exercises a non-Go
Tailscale client against headscale, giving end-to-end coverage of
the control protocol for alternate implementations.
Add TailscaleRustInContainer (tsric), a Rust-client counterpart to
integration/tsic. It runs the axum example from tailscale-rs in a
Docker container and exposes the same lifecycle hooks as tsic
(Shutdown, SaveLog, Execute, WriteFile) so integration tests can
treat it as any other Tailscale node.
Dockerfile.tailscale-rs clones tailscale-rs at build time, so no
local source checkout is required. The repo URL and ref are Docker
build arguments (TAILSCALE_RS_REPO, TAILSCALE_RS_REF) exposed as
tsric.WithRepo / tsric.WithRef options. The HEADSCALE_INTEGRATION_
TAILSCALE_RS_IMAGE environment variable provides an escape hatch
for using a pre-built image instead of building from source.
The Dockerfile patches the cloned root Cargo.toml to expose
ts_control's insecure-keyfetch feature through the tailscale crate
so the axum example can fetch the control key over plain HTTP.
The integration harness serves the control plane without TLS,
which is the only mode tailscale-rs can register against until it
grows a way to inject a custom CA bundle.
compileFilterRules, compileGrants, and updateLocked guarded the
"no rules so allow all" fallback with len(pol.Grants) == 0, which
matches both an absent grants field and an explicit empty array.
JSON {"grants": []} unmarshals to a non-nil empty slice; it should
compile to zero filter rules (deny all) to match Tailscale SaaS,
but the length check sent it down the FilterAllowAll path.
Distinguish absent (nil) from explicit-empty by switching the guard
to pol.Grants == nil, the same asymmetry already used for ACLs.
{} keeps allowing all; {"acls": []} and {"grants": []} now both
deny all.
Fixes#3211
The scheduled job has failed every night since 2026-04-13. Two
prior fixes (race-condition guards and splitting the combined
where predicate in #3200) did not address the actual cause:
`flatten` collapses nested records in a list-of-records pipeline,
so after `gh api ... | from json | flatten` the `label` and `user`
columns no longer exist - their fields are lifted to the top
level (with prefix only on naming collisions). `where label.name
== ...` and `where user.type != "Bot"` then both reference a
column that is not there.
`gh api --paginate` already returns a single concatenated JSON
array, so `from json` produces a list of records directly and no
flattening is needed. Drop both `| flatten` calls.
Verified locally with nu 0.108 against the events stream of issue
#3178: without flatten, `where event == "labeled" | where
label.name == "needs-more-info" | last` returns the labeled event
with its `label` record intact.
GitHub's /issues/:n/events endpoint returns a mixed-schema table.
Only labeled/unlabeled rows carry a `label` column. Nu's `where`
does not short-circuit on missing columns, so the combined
predicate `event == "labeled" and label.name == ...` dereferenced
`label.name` on every row and crashed on the first non-labeled
event with `nu::shell::column_not_found`.
The scheduled job has failed every night since 2026-04-13 (the
first run with a labeled issue), so no `needs-more-info` issue
has been auto-closed.
Split into two sequential `where` filters so `label.name` is only
accessed on rows that have the column.
Summarise the ingest rewrite, the SaaS-matching collision rule, and the
BREAKING change from random-suffix to numeric-suffix collision labels
and from "invalid-<rand>" to the literal "node" fallback.
Updates #3188
Ingest (registration and MapRequest updates) now calls
dnsname.SanitizeHostname directly and lets NodeStore auto-bump on
collision. Admin rename uses dnsname.ValidLabel + SetGivenName so
conflicts are surfaced to the caller instead of silently mutated.
Three duplicate invalidDNSRegex definitions, the old NormaliseHostname
and ValidateHostname helpers, EnsureHostname, InvalidString,
ApplyHostnameFromHostInfo, GivenNameHasBeenChanged, generateGivenName
and EnsureUniqueGivenName are removed along with their tests.
ValidateHostname's username half is retained as ValidateUsername for
users.go.
The SaaS-matching collision rule replaces the random "invalid-xxxxxx"
fallback and the 8-character hash suffix; the empty-input fallback is
the literal "node". TestUpdateHostnameFromClient now exercises the
rewrite end-to-end with awkward macOS/Windows names.
Fixes#3188Fixes#2926Fixes#2343Fixes#2762Fixes#2449
Updates #2177
Updates #2121
Updates #363
NodeStore's writer goroutine now resolves GivenName collisions inside
applyBatch: on PutNode/UpdateNode the landing label gets -N appended
until unique, matching Tailscale SaaS. Empty labels fall back to the
literal "node".
SetGivenName exposes the admin-rename path: validates via
dnsname.ValidLabel and rejects on collision with ErrGivenNameTaken,
so renames do not silently rewrite behind the caller.
Updates #3188
Updates #2926
Updates #2343
Updates #2762
Existing HA tests verify server-side primary election; these add
end-to-end assertions from a viewer client's perspective, that marking
the primary unhealthy or revoking its approved route propagates through
the netmap so the viewer sees the flip.
Updates #3157
Resolve by GivenName (unique per tailnet) before Hostname (client-
reported, may collide); within each pass, pick the lowest NodeID so
results are deterministic across NodeStore snapshot iterations.
Updates #3157
change.Merge keeps the first PingRequest seen when merging, which
means a later probe's callback URL is silently dropped if a stale merge
is in-flight. Document the contract at Merge and at doPing so callers
know to serialise probes.
Updates #3157
dev derives additional ports from --port + offsets (metrics, gRPC,
debug). A --port near uint16 max would overflow silently; add up-front
validation that rejects values that would push derived ports over 65535.
Updates #3157
DestsIsTheInternet now reports the internet when either family's /0
is contained (0.0.0.0/0 or ::/0), matching what operators expect when
they write the /0 directly. Also documents MatchFromStrings fail-open.
Updates #3157
The /debug/ping callback hits /machine/ping-response on the main
TLS router, not the noise chain, so URLIsNoise stays false. Document
this at the emit site to prevent accidental changes.
Updates #3157
Blocked callers waiting on a pingTracker response channel would
hang forever if the server Close()d mid-probe. Drain the pending map on
Close so those goroutines unblock and exit cleanly.
Updates #3157
X-Frame-Options: DENY and frame-ancestors 'none' stop clickjacking
of OIDC, register-confirm, and debug HTML pages. nosniff and no-referrer
are cheap defence-in-depth for the same surfaces.
Updates #3157
chi routes only HEAD to the handler, but assert explicitly so a
future router config change cannot silently accept GET/POST and leak
latency bytes or side-effects.
Updates #3157
elem-go does not escape attribute values, so the raw query reaches
the rendered HTML verbatim. Pre-escape with html.EscapeString to prevent
reflected XSS.
Updates #3157
Subnet routers that advertise routes must not accept peer routes.
With co-router visibility the HA primary's subnet appears in co-routers'
AllowedIPs, and --accept-routes installs a system route that conflicts
with local subnet forwarding.
Updates #3157
Data-driven tests for via grants combined with HA primary routes:
crossed via tags on same prefix, mixed via+regular across HA pairs,
four-way HA, and the kitchen-sink scenario. Each case uses an inline
topology captured from SaaS.
Updates #3157
End-to-end exercise of via-grant compilation against SaaS captures:
peer visibility, AllowedIPs, PrimaryRoutes, and per-rule src/dst
reachability from each viewer's perspective.
Updates #3157
Typed Capture/Input/Node/Topology structs for golden SaaS captures.
Schema drift between the tscap capture tool and headscale now becomes a
compile error instead of a silent test pass.
Updates #3157
Tests were dumping megabytes of zerolog output on failure; silence
at init and let individual tests opt in via SetGlobalLevel when they need
log-driven assertions.
Updates #3157
reduceCapGrantRule was dropping rules whose CapGrant IPs overlap a
subnet route; treat subnet routes as part of node identity so those rules
survive reduction. ReduceFilterRules now also reduces route-reachable
destinations.
Updates #3157
Threads PolicyManager into compiledGrant so via grants resolve
viewer identity at compile time instead of re-resolving per MapRequest.
Adds a matchersForNodeMap cache invalidated on policy reload and on node
add/remove.
Updates #3157
CanAccess now treats a node's advertised subnet routes as part of
its source identity, so an ACL granting the subnet-owner as source lets
traffic from the subnet through. Matches SaaS semantics.
Updates #3157
Golden captures of SaaS filter-rules and netmaps across the ACL,
grant, routes, and SSH corpora. These back the data-driven compat tests
that verify headscale's policy output against Tailscale SaaS verbatim.
Updates #3157
Block ping callbacks via iptables while keeping the Noise session alive
to simulate a zombie-connected router. Verify the prober detects it,
fails over, and does not flap on recovery.
Updates #2129
Updates #2902
Ping HA subnet routers each probe cycle and mark unresponsive nodes
unhealthy. Reconnecting a node clears its unhealthy state since the
fresh Noise session proves basic connectivity.
Updates #2129
Updates #2902
Track unhealthy nodes in PrimaryRoutes so primary election skips them.
When all nodes for a prefix are unhealthy, keep the first as a degraded
primary rather than dropping the route entirely.
Anti-flap is built in: a recovered node becomes standby, not primary,
because updatePrimaryLocked keeps the current primary when still
available and healthy.
Updates #2129
Updates #2902
- Remove redundant inline button/input styles that duplicate CSS
- Use CSS variables for input (dark mode support)
- Use A(), Ul(), Ol(), P() wrappers from general.go
- Add expandable explanation of what the ping tests
- Fix section spacing rhythm (spaceXL before results, space2XL
before connected nodes)
- Add flex-wrap for mobile responsiveness
Add a reusable <details>/<summary> component to the shared design
system. Styled to match the existing card/box component family
(border, radius, CSS variables for dark mode).
Collapsed by default with a clickable summary line.
Add a lightweight dev tool that starts a headscale server on localhost
with a pre-created user and pre-auth key, ready for connecting real
tailscale nodes via mts.
The tool builds the headscale binary, writes a minimal dev config
(SQLite, public DERP, debug logging), starts the server as a
subprocess, and prints a banner with the server URL, auth key, and
mts usage instructions.
Usage: go run ./cmd/dev
make dev-server
Unit tests for Change (IsEmpty, Merge, Type, PingNode constructor),
ping tracker (register/complete/cancel lifecycle, concurrency, latency),
and end-to-end servertests exercising the full round-trip with real
controlclient.Direct instances.
Updates #2902
Updates #2129
Implement tailcfg.PingRequest support so the control server can verify
whether a connected node is still reachable. This is the foundation for
faster offline detection (currently ~16min due to Go HTTP/2 TCP retransmit
behavior) and future C2N communication.
The server sends a PingRequest via MapResponse with a unique callback
URL. The Tailscale client responds with a HEAD request to that URL,
proving connectivity. Round-trip latency is measured.
Wire PingRequest through the Change → Batcher → MapResponse pipeline,
add a ping tracker on State for correlating requests with responses,
add ResolveNode for looking up nodes by ID/IP/hostname, and expose a
/debug/ping page (elem-go form UI) and /machine/ping-response endpoint.
Updates #2902
Updates #2129
The directory /usr/share/doc/headscale/examples may be used to install
arbitrary example files. This is useful to get a matching configuration
for the release which gets also overwritten automatically.
Replace the bullet list of device details with a two-column table
for cleaner visual hierarchy. Labels are bold and left-aligned,
values right-aligned with subtle row separators. The machine key
value uses an inline code style.
Updates juanfont/headscale#3182
Tighten the SVG viewBox to the actual content bounding box and
remove hardcoded width/height attributes so the browser no longer
adds horizontal padding via preserveAspectRatio. The "h" wordmark
now left-aligns with the page content below it.
Replace the error icon SVG path (which had an off-center X) with
a simple circle + two crossed lines drawn from a centered viewBox.
Both icons now use fill="currentColor" for dark mode adaptation.
Updates juanfont/headscale#3182
Replace hardcoded Go color constants with var(--hs-*) and
var(--md-*) CSS custom properties in externalLink, orDivider,
card, warningBox, downloadButton, and pageFooter. This ensures
all components follow the dark mode theme automatically.
Also switch pageFooter from div to semantic footer element and
simplify externalLink by letting CSS handle link styling.
Updates juanfont/headscale#3182
Bump base font size from 0.8rem to 1rem (16px) to meet mobile
accessibility guidelines and avoid iOS auto-zoom on inputs.
Add CSS custom properties for all theme colors with a
prefers-color-scheme: dark media query so pages adapt to OS dark
mode. Component inline styles reference var(--hs-*) tokens so they
follow the scheme automatically.
Accessibility improvements:
- role="status" + aria-live="polite" on success boxes
- role="alert" + aria-live="assertive" on error boxes
- role="note" on warning boxes
- Visible focus rings via :focus-visible
- Link underlines (don't rely on color alone)
- SVG icons use currentColor for theme adaptation
- prefers-reduced-motion media query
- <main> landmark element wrapping page content
- Button styling with 44px min-height touch target
- List item spacing
Updates juanfont/headscale#3182
Add httpUserError() alongside httpError() for browser-facing error
paths. It renders a styled HTML page using the AuthError template
instead of returning plain text. Technical error details stay in
server logs; the HTML page shows actionable messages derived from
the HTTP status code:
401/403 → "You are not authorized. Please contact your administrator."
410 → "Your session has expired. Please try again."
400-499 → "The request could not be processed. Please try again."
500+ → "Something went wrong. Please try again later."
Convert all httpError calls in oidc.go (OIDC callback, SSH check,
registration confirm) to httpUserError. Machine-facing endpoints
(noise, verify, key, health, debug) are unchanged.
Fixesjuanfont/headscale#3182
Add errorBox() and errorIcon() to the design system, mirroring the
existing successBox()/checkboxIcon() pattern with red error styling.
Extract error color constants from the inline values in statusMessage().
Add AuthError() template that renders a styled HTML error page using
the same HtmlStructure/mdTypesetBody/logo/footer as all other
browser-facing pages.
Updates juanfont/headscale#3182
Those were required to streamline new installs with updates before 0.27.
Since 0.29 will not allow direct upgrades from <0.27 to 0.29 we might as
well remove it.
The integration test generator scanned all files under integration/
with ripgrep, matching func Test* patterns in README.md code examples
(TestMyScenario, TestRouteAdvertisementBasic). Add --type go to limit
the search to Go source files.
Render an interstitial showing device hostname, OS, and machine-key
fingerprint before finalising OIDC registration. The user must POST
to /register/confirm/{auth_id} with a CSRF double-submit cookie.
Removes the TODO at oidc.go:201.
Build the statsviz Server directly and wrap its Index/Ws handlers in
tsweb.Protected instead of calling statsviz.Register on the raw mux
which bypasses AllowDebugAccess.
Add json:"-" to PostgresConfig.Pass, OIDCConfig.ClientSecret, and
CLIConfig.APIKey so they are excluded from json.Marshal output
(e.g. the /debug/config endpoint).
Add a row-level check so concurrent registrations with the same
single-use key cannot both succeed. Skip the call on
re-registration where the key is already marked used (#2830).
Reflection is a streaming RPC and bypasses the unary auth
interceptor on the remote (TCP) gRPC server. Remove it there;
the unix-socket server retains it for local debugging.
getCookieName sliced value[:6] unconditionally; a short state query
parameter caused a panic recovered by chi middleware. Reject states
shorter than cookieNamePrefixLen with 400.
The #2862 restart path returned nodeToRegisterResponse after a
NodeKey-only lookup without verifying MachineKey. Add the same
check handleLogout already performs.
SSHActionHandler now verifies that the Noise session's machine key
matches the dst node before proceeding. The (src, dst) pair is
captured at hold-and-delegate time via a new SSHCheckBinding on
AuthRequest so sshActionFollowUp can verify the follow-up URL
matches. The OIDC non-registration callback requires the
authenticated user to own the src node before approving.
Replace zcache with golang-lru/v2/expirable for both the state auth
cache and the OIDC state cache. Add tuning.register_cache_max_entries
(default 1024) to cap the number of pending registration entries.
Introduce types.RegistrationData to replace caching a full *Node;
only the fields the registration callback path reads are retained.
Remove the dead HSDatabase.regCache field. Drop zgo.at/zcache/v2
from go.mod.
When send() is called on a node with zero active connections
(disconnected but kept for rapid reconnection), it returns nil
(success). handleNodeChange then calls updateSentPeers, recording
peers as delivered when no client received the data.
This corrupts lastSentPeers: future computePeerDiff calculations
produce wrong results because they compare against phantom state.
After reconnection, the node's initial map resets lastSentPeers,
but any changes processed during the disconnect window leave
stale entries that cause asymmetric peer visibility.
Return errNoActiveConnections from send() when there are no
connections. handleNodeChange treats this as a no-op (the change
was generated but not deliverable) and skips updateSentPeers,
keeping lastSentPeers consistent with what clients actually
received.
When a FullUpdate produces zero visible peers (e.g., a restrictive
policy isolates a node), the MapResponse has Peers: [] (empty
non-nil). The Tailscale client only processes Peers as a full
replacement when len(Peers) > 0 (controlclient/map.go:462), so an
empty list is silently ignored and stale peers persist.
This triggers when a FullUpdate() replaces a pending PolicyChange()
in the batcher. The PolicyChange would have used computePeerDiff to
send explicit PeersRemoved, but the FullUpdate goes through
buildFromChange which sets Peers: [] that the client ignores.
When a full update produces zero peers, compute the peer diff
against lastSentPeers and add explicit PeersRemoved entries so the
client correctly clears its stale peer state.
Merging two changes targeted at different nodes is not supported
because the result can only carry one TargetNode. The second
target's content would be silently misrouted.
Add a panic guard that catches this at the Merge call site rather
than allowing silent data loss. In production, Merge is only called
with broadcast changes (TargetNode=0) so the guard acts as
insurance against future misuse.
url.JoinPath resolves path-traversal segments like '..' and '.',
which silently drops the OIDC subject from the identifier. For
example, Iss='https://example.com' with Sub='..' produces
'https://example.com' — the subject is lost entirely. This causes
distinct OIDC users to receive colliding identifiers.
Replace url.JoinPath with simple string concatenation using a slash
separator. This preserves the subject literally regardless of its
content. url.PathEscape does not help because dots are valid URL
path characters and are not escaped.
Owner() on a non-tagged node with nil User returns an invalid
UserView that panics when Name() is called. Add a guard to return
an empty UserView{} when the user is not valid.
TailscaleUserID() calls UserID().Get() without checking Valid()
first, which panics on orphaned nodes (no tags, no UserID). Add a
validity check to return 0 for this invalid state.
Callers should check Owner().Valid() before accessing fields.
The migration at db.go:680 appends validated tags to existing tags
using append(existingTags, validatedTags...) where existingTags
aliases node.Tags. When node.Tags has spare capacity, append writes
into the shared backing array, and the subsequent slices.Sort
corrupts the original.
Clone existingTags before appending to prevent aliasing.
routesChanged aliases newHI.RoutableIPs into a local variable then
sorts it in place, which mutates the caller's Hostinfo data. The
Hostinfo is subsequently stored on the node, so the mutation
propagates but the input contract is violated.
Clone the slice before sorting to avoid mutating the input.
Merge copies the receiver by value, but the slice headers share the
backing array with the original. When append has spare capacity, it
writes through to the original's memory, and uniqueNodeIDs then
sorts that shared data in place.
Replace append with slices.Concat which always allocates a fresh
backing array, preventing mutation of the receiver's slices.
Procedural content moves to cmd/hi/README.md and integration/README.md.
Stale references (poll.go:420, mapper/tail.go, notifier/,
quality-control-enforcer, validateAndNormalizeTags) are corrected or
removed.
The nix dev shell refresh in 758fef9b pulled in protoc-gen-go-grpc
v1.6.1 and newer tailscale.com/cmd/{viewer,cloner}, so rerunning
`make generate` updates the version header comments in the three
affected generated files. No semantic changes.
Refresh flake.lock (nixpkgs 2026-03-08 -> 2026-04-09) and bump the
tool pins that live directly in flake.nix:
* golangci-lint 2.9.0 -> 2.11.4
* protoc-gen-grpc-gateway 2.27.7 -> 2.28.0 (keeps the dev-shell
code-gen tool in sync with the grpc-gateway Go module)
* protobuf-language-server pinned commit bumped to ab4c128
Also replace nodePackages.prettier with the top-level prettier
attribute. nodePackages was removed from nixpkgs in the update and
the dev shell would otherwise fail to evaluate with:
error: nodePackages has been removed because it was unmaintainable
within nixpkgs
`nix flake check --all-systems` and `nix build .#headscale` both
pass, and `golangci-lint 2.11.4` reports no new issues on the tree.
Routine bump of direct Go dependencies. Notable updates:
* tailscale.com v1.94.1 -> v1.96.5 (gvisor bumped in lockstep to
match upstream tailscale go.mod)
* modernc.org/sqlite v1.44.3 -> v1.48.2, modernc.org/libc v1.67.6
-> v1.70.0 (updated together as required by the fragile libc
dependency noted in #2188)
* google.golang.org/grpc v1.78.0 -> v1.80.0
* grpc-ecosystem/grpc-gateway/v2 v2.27.7 -> v2.28.0
* tailscale/hujson, tailscale/squibble, tailscale/tailsql
* golang.org/x/{crypto,net,sync,oauth2,exp,sys,text,time,term,mod,tools}
* rs/zerolog, samber/lo, sasha-s/go-deadlock, coreos/go-oidc/v3,
creachadair/command, go-json-experiment/json, pterm/pterm
Update the nix vendorHash to match the new go.sum. Regenerating capver
against tailscale v1.96.5 produces no diff: v1.96.0 was already
captured in 442fcdbd and the capability version has not changed in
the patch series.
All unit tests and `golangci-lint run --new-from-rev=main` are clean.
Tailscale main now requires go >= 1.26.2, so building the HEAD derper
image against golang:1.26.1-alpine fails with:
go: go.mod requires go >= 1.26.2 (running go 1.26.1; GOTOOLCHAIN=local)
Bump Dockerfile.derper to match the earlier fix for Dockerfile.tailscale-HEAD
in 6390fcee so TestDERPVerifyEndpoint can build the derper container
again. This test is the only consumer of Dockerfile.derper, which is why
the failure was scoped to that single integration job.
DestroyUser called ListPreAuthKeys(tx) which returns ALL pre-auth keys
across all users, then deleted every one of them. This caused deleting
any single user to wipe out pre-auth keys for every other user.
Extract a ListPreAuthKeysByUser function (consistent with the existing
ListNodesByUser pattern) and use it in DestroyUser to scope key deletion
to the user being destroyed.
Add unit test (table-driven in TestDestroyUserErrors) and integration
test to prevent regression.
Fixes#3154
Co-authored-by: Kristoffer Dalby <kristoffer@dalby.cc>
When UpdateNodeFromMapRequest and SetNodeTags race on persistNodeToDB,
the first caller to run updatePolicyManagerNodes detects the tag change
and returns a PolicyChange. The second caller finds no change and falls
back to NodeAdded.
If UpdateNodeFromMapRequest wins the race, it checked
policyChange.IsFull() which is always false for PolicyChange (only sets
IncludePolicy and RequiresRuntimePeerComputation). This caused the
PolicyChange to be dropped, so affected clients never received
PeersRemoved and the stale peer remained in their NetMap indefinitely.
Fix: check !policyChange.IsEmpty() instead, which correctly detects
any non-trivial policy change including PolicyChange().
This fixes the root cause of TestACLTagPropagation/multiple-tags-partial-
removal flaking at ~20% on CI.
Updates #3125
GORM's struct-based Updates() silently skips nil pointer fields.
When SetNodeTags sets node.UserID = nil to transfer ownership to tags,
the in-memory NodeStore is correct but the database retains the old
user_id value. This causes tagged nodes to remain associated with the
original user in the database, preventing user deletion and risking
ON DELETE CASCADE destroying tagged nodes.
Add Select("*") before Omit() on all three node persistence paths
to force GORM to include all fields in the UPDATE statement, including
nil pointers. This is the same pattern already used in db/ip.go for
IPv4/IPv6 nil handling, and is documented GORM behavior:
db.Select("*").Omit("excluded").Updates(struct)
The three affected paths are:
- persistNodeToDB: used by SetNodeTags and MapRequest updates
- applyAuthNodeUpdate: used by re-authentication with --advertise-tags
- HandleNodeFromPreAuthKey: used by PAK re-registration
Fixes#3161
Add four tests that verify the tags-as-identity ownership transition
correctly persists to the database when converting a user-owned node
to a tagged node via SetTags:
- TestSetTags_ClearsUserIDInDatabase: verifies user_id is NULL in DB
- TestSetTags_NodeDisappearsFromUserListing: verifies ListNodes by user
- TestSetTags_NodeStoreAndDBConsistency: verifies in-memory and DB agree
- TestSetTags_UserDeletionDoesNotCascadeToTaggedNode: verifies user
deletion does not cascade-delete tagged nodes
Three of these tests currently fail because GORM's struct-based
Updates() silently skips nil pointer fields, so user_id is never
written as NULL to the database after SetNodeTags clears it in memory.
Updates #3161
go generate ./hscontrol/capver/...
Adds v1.96 (capVer 133) to tailscaleToCapVer and capVerToTailscaleVer,
rolls the 10-version support window forward so MinSupportedCapabilityVersion
is now 109 (v1.78), and refreshes the test fixture accordingly.
Connect and Disconnect previously only appended a PolicyChange when
the affected node was a subnet router (routeChange) or the database
persist returned a full change. For every other node the peers just
received a small PeerChangedPatch{Online: ...} and no filter rules
were recomputed. That was too narrow: a node going offline or coming
online can affect policy compilation in ways beyond subnet routes.
TestGrantCapRelay Phase 4 exposed this. When the cap/relay target node
went down with `tailscale down`, headscale only sent an Online=false
patch, peers never got a recomputed netmap, and their cached
PeerRelay allocation stayed populated until the 120s assertion
timeout. With a PolicyChange queued on Disconnect, peers immediately
receive a full netmap on relay loss and clear PeerRelay as expected;
the symmetric change on Connect lets Phase 5 re-publish the policy
when the relay comes back.
Drop the now-unused routeChange return from the Disconnect gate.
Updates #2180
Address-based aliases (Prefix, Host) now resolve to exactly the literal
prefix and do not expand to include the matching node's other IP
addresses. This means an IPv4-only host definition only produces IPv4
filter rules, and an IPv6-only definition only produces IPv6 rules.
Update TestACLDevice1CanAccessDevice2 and TestACLNamedHostsCanReach to
track which addresses each test case covers via test1Addr/test2Addr/
test3Addr fields and only assert connectivity for that family.
Previously the tests assumed all address families would work regardless
of how the policy aliases were defined, which was true only when
address-based aliases auto-expanded to include all of a node's IPs.
The group test case (identity-based) keeps using IPv4 since tags, users,
groups, autogroups and the wildcard still resolve to both IPv4 and IPv6.
Updates #2180
When WithPingUntilDirect(false) is set, the Ping helper should accept
any indirect path, but the substring check only matched "via DERP" and
"via relay". Tailscale peer relay pings output
pong from ... via peer-relay(ip:port:vni:N) in Nms
which does not contain the "via relay" substring and was therefore
rejected as errTailscalePingNotDERP. TestGrantCapRelay Phase 4 never
passed because of this: even when the data plane was healthy the
helper returned an error.
Commit abe1a3e7 attempted to fix this by adding "via relay" alongside
"via DERP" but missed the "peer-" prefix used by peer relay output.
Add an explicit "via peer-relay" substring check so peer relay pongs
are accepted alongside DERP and plain relay pongs.
Updates #2180
Tailscale main now requires go >= 1.26.2, so building the HEAD image
against golang:1.26.1-alpine fails with:
go: go.mod requires go >= 1.26.2 (running go 1.26.1; GOTOOLCHAIN=local)
Bump the base image to golang:1.26.2-alpine so `go run ./cmd/hi run`
can build the HEAD container locally again.
Replace httptest (real TCP sockets) with tailscale.com/net/memnet
so all connections stay in-process. Wire the client's tsdial.Dialer
to the server's memnet.Network via SetSystemDialerForTest,
preserving the full Noise protocol path.
Also update servertest to use the new Node.Ephemeral.InactivityTimeout
config path introduced in the types refactor, and add WithNodeExpiry
server option for testing default node key expiry behaviour.
Updates #1711
Use the node.expiry config to apply a default expiry to non-tagged
nodes when the client does not request a specific expiry. This covers
all registration paths: new node creation, re-authentication, and
pre-auth key re-registration.
Tagged nodes remain exempt and never expire.
Fixes#1711
Introduce a structured NodeConfig that replaces the flat
EphemeralNodeInactivityTimeout field with a nested Node section.
Add node.expiry config (default: no expiry) as the unified default key
expiry for all non-tagged nodes regardless of registration method.
Remove oidc.expiry entirely — node.expiry now applies to OIDC nodes
the same as all other registration methods. Using oidc.expiry in the
config is a hard error. determineNodeExpiry() returns nil (no expiry)
unless use_expiry_from_token is enabled, letting state.go apply the
node.expiry default uniformly.
The old ephemeral_node_inactivity_timeout key is preserved for
backwards compatibility.
Updates #1711
Rename all 594 test data files from .json to .hujson and add
descriptive header comments to each file documenting what policy
rules are under test and what outcome is expected.
Update test loaders in all 5 _test.go files to parse HuJSON via
hujson.Parse/Standardize/Pack before json.Unmarshal.
Add cross-dependency warning to via_compat_test.go documenting
that GRANT-V29/V30/V31/V36 are shared with TestGrantsCompat.
Add .gitignore exemption for testdata HuJSON files.
Remove unused error variables (ErrGrantViaNotSupported, ErrGrantEmptySources, ErrGrantEmptyDestinations, ErrGrantViaOnlyTag) and the stale TODO for via implementation. Update compat test skip reasons to reflect that user:*@passkey wildcard is a known unsupported feature, not a pending implementation.
Updates #2180
Remove TestGrantViaExitNodeSteering and TestGrantViaMixedSteering.
Exit node traffic forwarding through via grants cannot be validated
with curl/traceroute in Docker containers because Tailscale exit nodes
strip locally-connected subnets from their forwarding filter.
The correctness of via exit steering is validated by:
- Golden MapResponse comparison (TestViaGrantMapCompat with GRANT-V31
and GRANT-V36) comparing full netmap output against Tailscale SaaS
- Filter rule compatibility (TestGrantsCompat with GRANT-V14 through
GRANT-V36) comparing per-node PacketFilter rules against Tailscale SaaS
- TestGrantViaSubnetSteering (kept) validates via subnet steering with
actual curl/traceroute through Docker, which works for subnet routes
Updates #2180
Use per-node compilation path for via grants in BuildPeerMap and MatchersForNode to ensure via-granted nodes appear in peer maps. Fix ViaRoutesForPeer golden test route inference to correctly resolve via grant effects.
Updates #2180
Add golden test data for via exit route steering and fix via exit grant compilation to match Tailscale SaaS behavior. Includes MapResponse golden tests for via grant route steering verification.
Updates #2180
Add NetworkSpec struct with optional Subnet field to ScenarioSpec.Networks.
When Subnet is set, the Docker network is created with that specific CIDR
instead of Docker's auto-assigned RFC1918 range.
Fix all exit node integration tests to use curl + traceroute. Tailscale
exit nodes strip locally-connected subnets from their forwarding filter
(shrinkDefaultRoute + localInterfaceRoutes), so exit nodes cannot
forward to IPs on their Docker network via the default route alone.
This is by design: exit nodes provide internet access, not LAN access.
To also get LAN access, the subnet must be explicitly advertised as a
route — matching real-world Tailscale deployment requirements.
- TestSubnetRouterMultiNetworkExitNode: advertise usernet1 subnet
alongside exit route, upgraded from ping to curl + traceroute
- TestGrantViaExitNodeSteering: usernet1 subnet in via grants and
auto-approvers alongside autogroup:internet
- TestGrantViaMixedSteering: externet subnet in auto-approvers and
route advertisement for exit traffic
Updates #2180
Add three tests that verify control plane behavior for grant policies:
- TestGrantViaSubnetFilterRules: verifies the router's PacketFilter
contains destination rules for via-steered subnets. Without per-node
filter compilation for via grants, these rules were missing and the
router would drop forwarded traffic.
- TestGrantViaExitNodeFilterRules: same verification for exit nodes
with via grants steering autogroup:internet traffic.
- TestGrantIPv6OnlyPrefixACL: verifies that address-based aliases
(Prefix, Host) resolve to exactly the literal prefix and do not
expand to include the matching node's other IP addresses. An
IPv6-only host definition produces only IPv6 filter rules.
Updates #2180
Via grants compile filter rules that depend on the node's route state
(SubnetRoutes, ExitRoutes). Without per-node compilation, these rules
were only included in the global filter path which explicitly skips via
grants (compileFilterRules skips grants with non-empty Via fields).
Add a needsPerNodeFilter flag that is true when the policy uses either
autogroup:self or via grants. filterForNodeLocked now uses this flag
instead of usesAutogroupSelf alone, ensuring via grant rules are
compiled per-node through compileFilterRulesForNode/compileViaGrant.
The filter cache also needs to account for route-dependent compilation:
- nodesHavePolicyAffectingChanges now treats route changes as
policy-affecting when needsPerNodeFilter is true, so SetNodes
triggers updateLocked and clears caches through the normal flow.
- invalidateGlobalPolicyCache now clears compiledFilterRulesMap
(the unreduced per-node cache) alongside filterRulesMap when
needsPerNodeFilter is true and routes changed.
Updates #2180
Action.UnmarshalJSON produces the format
'action="unknown-action" is not supported: invalid ACL action',
not the reversed format the test expected.
Updates #2180
Add servertest grant policy control plane tests covering basic grants, via grants, and cap grants. Fix ReduceRoutes in State to apply route reduction to non-via routes first, then append via-included routes, preventing via grant routes from being incorrectly filtered.
Updates #2180
Test via route computation for viewer-peer pairs: self-steering returns
empty, viewer not in source returns empty, peer without advertised
destination returns empty, peer with/without via tag populates
Include/Exclude respectively, mixed prefix and autogroup:internet
destinations, and exit route steering.
7 subtests covering all code paths in ViaRoutesForPeer.
Updates #2180
Test companionCapGrantRules, sourcesHaveWildcard, sourcesHaveDangerAll,
srcIPsWithRoutes, the FilterAllowAll fix for grant-only policies,
compileViaGrant, compileGrantWithAutogroupSelf grant paths, and
destinationsToNetPortRange autogroup:internet skipping.
51 subtests across 8 test functions covering all grant-specific code
paths in filter.go that previously had no test coverage.
Updates #2180
Add NodeAttrsTaildriveShare and NodeAttrsTaildriveAccess to the node capability map, enabling Taildrive file sharing when granted via policy. Add integration test verifying the full cap/drive grant lifecycle.
Updates #2180
Add ConnectToNetwork to the TailscaleClient interface for multi-network test scenarios and implement peer relay ping support. Use these to test that cap/relay grants correctly enable peer-to-peer relay connections between tagged nodes.
Updates #2180
compileFilterRules checked only pol.ACLs == nil to decide whether
to return FilterAllowAll (permit-any). Policies that use only Grants
(no ACLs) had nil ACLs, so the function short-circuited before
compiling any CapGrant rules. This meant cap/relay, cap/drive, and
any other App-based grant capabilities were silently ignored.
Check both ACLs and Grants are empty before returning FilterAllowAll.
Updates #2180
Add integration tests validating that via grants correctly steer
routes to designated nodes per client group:
- TestGrantViaSubnetSteering: two routers advertise the same
subnet, via grants steer each client group to a specific router.
Verifies per-client route visibility, curl reachability, and
traceroute path.
- TestGrantViaExitNodeSteering: two exit nodes, via grants steer
each client group to a designated exit node. Verifies exit
routes are withdrawn from non-designated nodes and the client
rejects setting a non-designated exit node.
- TestGrantViaMixedSteering: cross-steering where subnet routes
and exit routes go to different servers per client group.
Verifies subnet traffic uses the subnet-designated server while
exit traffic uses the exit-designated server.
Also add autogroupp helper for constructing AutoGroup aliases in
grant policy configurations.
Updates #2180
Via grants steer routes to specific nodes per viewer. Until now,
all clients saw the same routes for each peer because route
assembly was viewer-independent. This implements per-viewer route
visibility so that via-designated peers serve routes only to
matching viewers, while non-designated peers have those routes
withdrawn.
Add ViaRouteResult type (Include/Exclude prefix lists) and
ViaRoutesForPeer to the PolicyManager interface. The v2
implementation iterates via grants, resolves sources against the
viewer, matches destinations against the peer's advertised routes
(both subnet and exit), and categorizes prefixes by whether the
peer has the via tag.
Add RoutesForPeer to State which composes global primary election,
via Include/Exclude filtering, exit routes, and ACL reduction.
When no via grants exist, it falls back to existing behavior.
Update the mapper to call RoutesForPeer per-peer instead of using
a single route function for all peers. The route function now
returns all routes (subnet + exit), and TailNode filters exit
routes out of the PrimaryRoutes field for HA tracking.
Updates #2180
compileViaGrant only handled *Prefix destinations, skipping
*AutoGroup entirely. This meant via grants with
dst=[autogroup:internet] produced no filter rules even when the
node was an exit node with approved exit routes.
Switch the destination loop from a type assertion to a type switch
that handles both *Prefix (subnet routes) and *AutoGroup (exit
routes via autogroup:internet). Also check ExitRoutes() in
addition to SubnetRoutes() so the function doesn't bail early
when a node only has exit routes.
Updates #2180
Add autogroup:danger-all as a valid source alias that matches ALL IP
addresses including non-Tailscale addresses. When used as a source,
it resolves to 0.0.0.0/0 + ::/0 internally but produces SrcIPs: ["*"]
in filter rules. When used as a destination, it is rejected with an
error matching Tailscale SaaS behavior.
Key changes:
- Add AutoGroupDangerAll constant and validation
- Add sourcesHaveDangerAll() helper and hasDangerAll parameter to
srcIPsWithRoutes() across all compilation paths
- Add ErrAutogroupDangerAllDst for destination rejection
- Remove 3 AUTOGROUP_DANGER_ALL skip entries (K6, K7, K8)
Updates #2180
Implement comprehensive grant validation including: accept empty sources/destinations (they produce no rules), validate grant ip/app field requirements, capability name format, autogroup constraints, via tag existence, and default route CIDR restrictions.
Updates #2180
Compile grants with "via" field into FilterRules that are placed only
on nodes matching the via tag and actually advertising the destination
subnets. Key behavior:
- Filter rules go exclusively to via-nodes with matching approved routes
- Destination subnets not advertised by the via node are silently dropped
- App-only via grants (no ip field) produce no packet filter rules
- Via grants are skipped in the global compileFilterRules since they
are node-specific
Reduces grant compat test skips from 41 to 30 (11 newly passing).
Updates #2180
Compile grant app fields into CapGrant FilterRules matching Tailscale
SaaS behavior. Key changes:
- Generate CapGrant rules in compileFilterRules and
compileGrantWithAutogroupSelf, with node-specific /32 and /128
Dsts for autogroup:self grants
- Add reversed companion rules for drive→drive-sharer and
relay→relay-target capabilities, ordered by original cap name
- Narrow broad CapGrant Dsts to node-specific prefixes in
ReduceFilterRules via new reduceCapGrantRule helper
- Skip merging CapGrant rules in mergeFilterRules to preserve
per-capability structure
- Remove ip+app mutual exclusivity validation (Tailscale accepts both)
- Add semantic JSON comparison for RawMessage types and netip.Prefix
comparators in test infrastructure
Reduces grant compat test skips from 99 to 41 (58 newly passing).
Updates #2180
When an ACL source list contains a wildcard (*) alongside explicit
sources (tags, groups, hosts, etc.), Tailscale preserves the individual
IPs from non-wildcard sources in SrcIPs alongside the merged wildcard
CGNAT ranges. Previously, headscale's IPSetBuilder would merge all
sources into a single set, absorbing the explicit IPs into the wildcard
range.
Track non-wildcard resolved addresses separately during source
resolution, then append their individual IP strings to the output
when a wildcard is also present. This fixes the remaining 5 ACL
compat test failures (K01 and M06 subtests).
Updates #2180
When an ACL has non-autogroup destinations (groups, users, tags, hosts)
alongside autogroup:self, emit non-self grants before self grants to
match Tailscale's filter rule ordering. ACLs with only autogroup
destinations (self + member) preserve the policy-defined order.
This fixes ACL-A17, ACL-SF07, and ACL-SF11 compat test failures.
Updates #2180
Add exit route check in ReduceFilterRules to prevent exit nodes from receiving packet filter rules for destinations that only overlap via exit routes. Remove resolved SUBNET_ROUTE_FILTER_RULES grant skip entries and update error message formatting for grant validation.
Updates #2180
Per Tailscale documentation, the wildcard (*) source includes "any
approved subnets" — the actually-advertised-and-approved routes from
nodes, not the autoApprover policy prefixes.
Change Asterix.resolve() to return just the base CGNAT+ULA set, and
add approved subnet routes as separate SrcIPs entries in the filter
compilation path. This preserves individual route prefixes that would
otherwise be merged by IPSet (e.g., 10.0.0.0/8 absorbing 10.33.0.0/16).
Also swap rule ordering in compileGrantWithAutogroupSelf() to emit
non-self destination rules before autogroup:self rules, matching the
Tailscale FilterRule wire format ordering.
Remove the unused AutoApproverPolicy.prefixes() method.
Updates #2180
Add routable_ips and approved_routes fields to the node topology
definitions in all golden test files. These represent the subnet
routes actually advertised by nodes on the Tailscale SaaS network
during data capture:
Routes topology (92 files, 6 router nodes):
big-router: 10.0.0.0/8
subnet-router: 10.33.0.0/16
ha-router1: 192.168.1.0/24
ha-router2: 192.168.1.0/24
multi-router: 172.16.0.0/24
exit-node: 0.0.0.0/0, ::/0
ACL topology (199 files, 1 router node):
subnet-router: 10.33.0.0/16
Grants topology (203 files, 1 router node):
subnet-router: 10.33.0.0/16
The route assignments were deduced from the golden data by analyzing
which router nodes receive FilterRules for which destination CIDRs
across all test files, and cross-referenced with the MTS setup
script (setup_grant_nodes.sh).
Updates #2180
Use ip.String() instead of netip.PrefixFrom(ip, ip.BitLen()).String()
when building DstPorts for autogroup:self destinations. This produces
bare IPs like "100.90.199.68" instead of CIDR notation like
"100.90.199.68/32", matching the Tailscale FilterRule wire format.
Updates #2180
AppendToIPSet now adds both IPv4 and IPv6 addresses for nodes, matching Tailscale's FilterRule wire format where identity-based aliases (tags, users, groups, autogroups) resolve to both address families. Update ReduceFilterRules test expectations to include IPv6 entries.
Updates #2180
Replace 8,286 lines of inline Go test expectations with 92 JSON golden files captured from Tailscale SaaS. The data-driven test driver validates route filtering, auto-approval, HA routing, and exit node behavior against real Tailscale output.
Updates #2180
Replace 9,937 lines of inline Go test expectations with 215 JSON golden files captured from Tailscale SaaS. The new data-driven test driver compares headscale's filter compilation output against real Tailscale behavior for each node in an 8-node topology.
Updates #2180
Introduce ResolvedAddresses type for structured IP set results. Refactor all Alias.Resolve() methods to return ResolvedAddresses instead of raw IPSets. Restrict identity-based aliases to matching address families, fix nil dereferences in partial resolution paths, and update test expectations for the new IP format (bare IPs, IP ranges instead of CIDR prefixes).
Updates #2180
Rename tailscale_compat_test.go to tailscale_acl_compat_test.go to make room for the grants compat test. Add 237 GRANT-*.json golden test files captured from Tailscale SaaS and a data-driven test driver that compares headscale's grant filter compilation against real Tailscale behavior.
Updates #2180
Add support for the Grant policy format as an alternative to ACL format,
following Tailscale's policy v2 specification. Grants provide a more
structured way to define network access rules with explicit separation
of IP-based and capability-based permissions.
Key changes:
- Add Grant struct with Sources, Destinations, InternetProtocols (ip),
and App (capabilities) fields
- Add ProtocolPort type for unmarshaling protocol:port strings
- Add Grant validation in Policy.validate() to enforce:
- Mutual exclusivity of ip and app fields
- Required ip or app field presence
- Non-empty sources and destinations
- Refactor compileFilterRules to support both ACLs and Grants
- Convert ACLs to Grants internally via aclToGrants() for unified
processing
- Extract destinationsToNetPortRange() helper for cleaner code
- Rename parseProtocol() to toIANAProtocolNumbers() for clarity
- Add ProtocolNumberToName mapping for reverse lookups
The Grant format allows policies to be written using either the legacy
ACL format or the new Grant format. ACLs are converted to Grants
internally, ensuring backward compatibility while enabling the new
format's benefits.
Updates #2180
WithTags was defined but never passed through to CreatePreAuthKey.
Fix NewClient to use CreateTaggedPreAuthKey when tags are specified,
enabling tests that need tagged nodes (e.g. via grant steering).
Updates #2180
When exit routes are approved, SubnetRoutes remains empty because exit
routes (0.0.0.0/0, ::/0) are classified separately. Without checking
ExitRoutes, the PolicyManager cache is not invalidated on exit route
approval, causing stale filter rules that lack via grant entries for
autogroup:internet destinations.
Updates #2180
Switch all integration test jobs (build, build-postgres, test
template) from ubuntu-latest (x86_64) to ubuntu-24.04-arm (aarch64).
ARM runners on GitHub Actions are free for public repos and tend
to have more consistent performance characteristics than the
shared x86_64 pool. This should reduce flakiness caused by
resource contention on congested runners.
Updates #3125
Apply CI-aware scaling to all remaining hardcoded timeouts:
- requireAllClientsOfflineStaged: scale the three internal stage
timeouts (15s/20s/60s) with ScaledTimeout.
- validateReloginComplete: scale requireAllClientsOnline (120s)
and requireAllClientsNetInfoAndDERP (3min) calls.
- WaitForTailscaleSyncPerUser callers in acl_test.go (3 sites, 60s).
- WaitForRunning callers in tags_test.go (10 sites): switch to
PeerSyncTimeout() to match convention.
- WaitForRunning/WaitForPeers direct callers in route_test.go.
- requireAllClientsOnline callers in general_test.go and
auth_key_test.go.
Replace pingAllHelper with assertPingAll/assertPingAllWithCollect:
- Wraps pings in EventuallyWithT so transient docker exec timeouts
are retried instead of immediately failing the test.
- Timeout scales with the ping matrix size (2s per ping budget for
2 full sweeps) so large tests get proportionally more time.
- Uses CollectT correctly, fixing the broken EventuallyWithT usage
in TestEphemeral where the old t.Errorf bypassed CollectT.
- Follows the established assert*/assertWithCollect naming.
Updates #3125
The default docker execute timeout (10s) is the root cause of
"dockertest command timed out" errors across many integration tests
on CI. On congested GitHub Actions runners, docker exec latency
alone can consume 2-5 seconds of this budget before the command
even starts inside the container.
Replace the hardcoded 10s constant with a function that returns
20s on CI, doubling the budget for all container commands
(tailscale status, headscale CLI, curl, etc.).
Similarly, scale the default tailscale ping timeout from 200ms to
400ms on CI. This doubles the per-attempt budget and the docker
exec timeout for pings (from 200ms*5=1s to 400ms*5=2s), giving
more headroom for docker exec overhead.
Updates #3125
TestACLTagPropagationPortSpecific failed twice on CI because it jumped
from SetNodeTags directly to checking curl, without first verifying the
tag change was applied on the server. This races against server-side
processing.
Add a tag verification step (matching TestACLTagPropagation's pattern)
and bump the Step 4 timeout from 60s to 90s since port-specific filter
changes require both endpoints to process the new PacketFilter from
the MapResponse while the WireGuard tunnel stays up.
Updates #3125
Wrap all 329 hardcoded EventuallyWithT timeouts across 12 test files
with integrationutil.ScaledTimeout(), which applies a 2x multiplier
on CI runners. This addresses the systemic issue where hardcoded
timeouts that work locally are insufficient under CI resource
contention.
Variable-based timeouts (propagationTime, assertTimeout in
route_test.go and totalWaitTime in auth_oidc_test.go) are wrapped
at their definition site so all downstream usages benefit.
The retry intervals (second duration parameter) are intentionally
NOT scaled, as they control polling frequency, not total wait time.
Updates #3125
Replace Curl() with CurlFailFast() in all negative curl assertions
(where the test expects the connection to fail). CurlFailFast uses
1 retry and 2s max time instead of 3 retries and 5s max, which
avoids wasting time on unnecessary retries when we expect the
connection to be blocked.
This affects 21 call sites across 7 test functions:
- TestACLAllowUser80Dst
- TestACLDenyAllPort80
- TestACLAllowUserDst
- TestACLAllowStarDst
- TestACLNamedHostsCanReach
- TestACLDevice1CanAccessDevice2
- TestPolicyUpdateWhileRunningWithCLIInDatabase
- TestACLAutogroupSelf
- TestACLPolicyPropagationOverTime
Where possible, the inline Curl+Error pattern is replaced with the
assertCurlFailWithCollect helper introduced in the previous commit.
Updates #3125
Add ScaledTimeout to scale EventuallyWithT timeouts by 2x on CI,
consistent with the existing PeerSyncTimeout (60s/120s) and
dockertestMaxWait (300s/600s) conventions.
Add assertCurlSuccessWithCollect and assertCurlFailWithCollect helpers
following the existing *WithCollect naming convention.
assertCurlFailWithCollect uses CurlFailFast internally for aggressive
timeouts, avoiding wasted retries when expecting blocked connections.
Apply these to the three flakiest ACL tests:
- TestACLTagPropagation: swap NetMap and curl verification order so
the fast NetMap check (confirms MapResponse arrived) runs before
the slower curl check. Use curl helpers and scaled timeouts.
- TestACLTagPropagationPortSpecific: use curl helpers and scaled
timeouts.
- TestACLHostsInNetMapTable: scale the 10s EventuallyWithT timeout.
Updates #3125
Move the container image and binary download details from the README
into a dedicated documentation page at setup/install/main. This gives
development builds a proper home in the docs site alongside the other
install methods. The README now links to the docs page instead.
Build and push multi-arch container images (linux/amd64, linux/arm64)
to GHCR and Docker Hub on every push to main that changes Go or Nix
files. Images are tagged as main-<short-sha> using ko with the same
distroless base image as release builds.
Cross-compiled binaries for linux and darwin (amd64, arm64) are
uploaded as workflow artifacts. The README links to these via
nightly.link for stable download URLs.
The project mkdocs-material is in maintenance-only mode and their
successor is not ready yet.
Use the modern, refreshed theme and drop the pymdownx.magiclink
extension.
Add a highly visible ASCII-art warning banner that is printed at
startup when the configured IP prefixes fall outside the standard
Tailscale CGNAT (100.64.0.0/10) or ULA (fd7a:115c:a1e0::/48) ranges.
The warning fires once even if both v4 and v6 are non-standard, and
the warnBanner() function is reusable for other critical configuration
warnings in the future.
Also updates config-example.yaml to clarify that subsets of the
default ranges are fine, but ranges outside CGNAT/ULA are not.
Closes#3055
Three corrections to issue tests that had wrong assumptions about
when data becomes available:
1. initial_map_should_include_peer_online_status: use WaitForCondition
instead of checking the initial netmap. Online status is set by
Connect() which sends a PeerChange patch after the initial
RegisterResponse, so it may not be present immediately.
2. disco_key_should_propagate_to_peers: use WaitForCondition. The
DiscoKey is sent in the first MapRequest (not RegisterRequest),
so peers may not see it until a subsequent map update.
3. approved_route_without_announcement: invert the test expectation.
Tailscale uses a strict advertise-then-approve model -- routes are
only distributed when the node advertises them (Hostinfo.RoutableIPs)
AND they are approved. An approval without advertisement is a dormant
pre-approval. The test now asserts the route does NOT appear in
AllowedIPs, matching upstream Tailscale semantics.
Also fix TestClient.Reconnect to clear the cached netmap and drain
pending updates before re-registering. Without this, WaitForPeers
returned immediately based on the old session's stale data.
Two fixes to how online status is handled during registration:
1. Re-registration (applyAuthNodeUpdate, HandleNodeFromPreAuthKey) no
longer resets IsOnline to false. Online status is managed exclusively
by Connect()/Disconnect() in the poll session lifecycle. The reset
caused a false offline blip: the auth handler's change notification
triggered a map regeneration showing the node as offline to peers,
even though Connect() would set it back to true moments later.
2. New node creation (createAndSaveNewNode) now explicitly sets
IsOnline=false instead of leaving it nil. This ensures peers always
receive a known online status rather than an ambiguous nil/unknown.
When a node disconnects, serveLongPoll defers a cleanup that starts a
grace period goroutine. This goroutine polls batcher.IsConnected() and,
if the node has not reconnected within ~10 seconds, calls
state.Disconnect() to mark it offline. A TOCTOU race exists: the node
can reconnect (calling Connect()) between the IsConnected check and
the Disconnect() call, causing the stale Disconnect() to overwrite
the new session's online status.
Fix with a monotonic per-node generation counter:
- State.Connect() increments the counter and returns the current
generation alongside the change list.
- State.Disconnect() accepts the generation from the caller and
rejects the call if a newer generation exists, making stale
disconnects from old sessions a no-op.
- serveLongPoll captures the generation at Connect() time and passes
it to Disconnect() in the deferred cleanup.
- RemoveNode's return value is now checked: if another session already
owns the batcher slot (reconnect happened), the old session skips
the grace period entirely.
Update batcher_test.go to track per-node connect generations and
pass them through to Disconnect(), matching production behavior.
Fixes the following test failures:
- server_state_online_after_reconnect_within_grace
- update_history_no_false_offline
- nodestore_correct_after_rapid_reconnect
- rapid_reconnect_peer_never_sees_offline
Add three test files designed to stress the control plane under
concurrent and adversarial conditions:
- race_test.go: 14 tests exercising concurrent mutations, session
replacement, batcher contention, NodeStore access, and map response
delivery during disconnect. All pass the Go race detector.
- poll_race_test.go: 8 tests targeting the poll.go grace period
interleaving. These confirm a logical TOCTOU race: when a node
disconnects and reconnects within the grace period, the old
session's deferred Disconnect() can overwrite the new session's
Connect(), leaving IsOnline=false despite an active poll session.
- stress_test.go: sustained churn, rapid mutations, rolling
replacement, data integrity checks under load, and verification
that rapid reconnects do not leak false-offline notifications.
Known failing tests (grace period TOCTOU race):
- server_state_online_after_reconnect_within_grace
- update_history_no_false_offline
- rapid_reconnect_peer_never_sees_offline
Split TestIssues into 7 focused test functions to stay under cyclomatic
complexity limits while testing more aggressively.
Issues surfaced (4 failing tests):
1. initial_map_should_include_peer_online_status: Initial MapResponse
has Online=nil for peers. Online status only arrives later via
PeersChangedPatch.
2. disco_key_should_propagate_to_peers: DiscoPublicKey set by client
is not visible to peers. Peers see zero disco key.
3. approved_route_without_announcement_is_visible: Server-side route
approval without client-side announcement silently produces empty
SubnetRoutes (intersection of empty announced + approved = empty).
4. nodestore_correct_after_rapid_reconnect: After 5 rapid reconnect
cycles, NodeStore reports node as offline despite having an active
poll session. The connect/disconnect grace period interleaving
leaves IsOnline in an incorrect state.
Passing tests (20) verify:
- IP uniqueness across 10 nodes
- IP stability across reconnect
- New peers have addresses immediately
- Node rename propagates to peers
- Node delete removes from all peer lists
- Hostinfo changes (OS field) propagate
- NodeStore/DB consistency after route mutations
- Grace period timing (8-20s window)
- Ephemeral node deletion (not just offline)
- 10-node simultaneous connect convergence
- Rapid sequential node additions
- Reconnect produces complete map
- Cross-user visibility with default policy
- Same-user multiple nodes get distinct IDs
- Same-hostname nodes get unique GivenNames
- Policy change during connect still converges
- DERP region references are valid
- User profiles present for self and peers
- Self-update arrives after route approval
- Route advertisement stored as AnnouncedRoutes
Extend the servertest harness with:
- TestClient.Direct() accessor for advanced operations
- TestClient.WaitForPeerCount and WaitForCondition helpers
- TestHarness.ChangePolicy for ACL policy testing
- AssertDERPMapPresent and AssertSelfHasAddresses
New test suites:
- content_test.go: self node, DERP map, peer properties, user profiles,
update history monotonicity, and endpoint update propagation
- policy_test.go: default allow-all, explicit policy, policy triggers
updates on all nodes, multiple policy changes, multi-user mesh
- ephemeral_test.go: ephemeral connect, cleanup after disconnect,
mixed ephemeral/regular, reconnect prevents cleanup
- routes_test.go: addresses in AllowedIPs, route advertise and approve,
advertised routes via hostinfo, CGNAT range validation
Also fix node_departs test to use WaitForCondition instead of
assert.Eventually, and convert concurrent_join_and_leave to
interleaved_join_and_leave with grace-period-tolerant assertions.
Add three test files exercising the servertest harness:
- lifecycle_test.go: connection, disconnection, reconnection, session
replacement, and mesh formation at various sizes.
- consistency_test.go: symmetric visibility, consistent peer state,
address presence, concurrent join/leave convergence.
- weather_test.go: rapid reconnects, flapping stability, reconnect
with various delays, concurrent reconnects, and scale tests.
All tests use table-driven patterns with subtests.
Add a new hscontrol/servertest package that provides a test harness
for exercising the full Headscale control protocol in-process, using
Tailscale's controlclient.Direct as the client.
The harness consists of:
- TestServer: wraps a Headscale instance with an httptest.Server
- TestClient: wraps controlclient.Direct with NetworkMap tracking
- TestHarness: orchestrates N clients against a single server
- Assertion helpers for mesh completeness, visibility, and consistency
Export minimal accessor methods on Headscale (HTTPHandler, NoisePublicKey,
GetState, SetServerURL, StartBatcher, StartEphemeralGC) so the servertest
package can construct a working server from outside the hscontrol package.
This enables fast, deterministic tests of connection lifecycle, update
propagation, and network weather scenarios without Docker.
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:
1. MapResponses delivered out of order — a later change could finish
generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
updateSentPeers does Clear() + Store() which is not atomic relative
to a concurrent Range() in computePeerDiff.
Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.
Fixes#3140
Replace inline WithDockerEntrypoint shell scripts in
TestACLTagPropagation and TestACLTagPropagationPortSpecific with
the standard WithPackages and WithWebserver options.
The custom entrypoints used fragile fixed sleeps and lacked the
robust network/cert readiness waits that buildEntrypoint provides.
Updates #3139
Make embedded DERP server and TLS the default configuration for all
integration tests, replacing the per-test opt-in model that led to
inconsistent and flaky test behavior.
Infrastructure changes:
- DefaultConfigEnv() includes embedded DERP server settings
- New() auto-generates a proper CA + server TLS certificate pair
- CA cert is installed into container trust stores and returned by
GetCert() so clients and internal tools (curl) trust the server
- CreateCertificate() now returns (caCert, cert, key) instead of
discarding the CA certificate
- Add WithPublicDERP() and WithoutTLS() opt-out options
- Remove WithTLS(), WithEmbeddedDERPServerOnly(), and WithDERPAsIP()
since all their behavior is now the default or unnecessary
Test cleanup:
- Remove all redundant WithTLS/WithEmbeddedDERPServerOnly/WithDERPAsIP
calls from test files
- Give every test a unique WithTestName by parameterizing aclScenario,
sshScenario, and derpServerScenario helpers
- Add WithTestName to tests that were missing it
- Document all non-standard options with inline comments explaining
why each is needed
Updates #3139
The Batcher's connected field (*xsync.Map[types.NodeID, *time.Time])
encoded three states via pointer semantics:
- nil value: node is connected
- non-nil time: node disconnected at that timestamp
- key missing: node was never seen
This was error-prone (nil meaning 'connected' inverts Go idioms),
redundant with b.nodes + hasActiveConnections(), and required keeping
two parallel maps in sync. It also contained a bug in RemoveNode where
new(time.Now()) was used instead of &now, producing a zero time.
Replace the separate connected map with a disconnectedAt field on
multiChannelNodeConn (atomic.Pointer[time.Time]), tracked directly
on the object that already manages the node's connections.
Changes:
- Add disconnectedAt field and helpers (markConnected, markDisconnected,
isConnected, offlineDuration) to multiChannelNodeConn
- Remove the connected field from Batcher
- Simplify IsConnected from two map lookups to one
- Simplify ConnectedMap and Debug from two-map iteration to one
- Rewrite cleanupOfflineNodes to scan b.nodes directly
- Remove the markDisconnectedIfNoConns helper
- Update all tests and benchmarks
Fixes#3141
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:
1. MapResponses delivered out of order — a later change could finish
generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
updateSentPeers does Clear() + Store() which is not atomic relative
to a concurrent Range() in computePeerDiff.
Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.
Fixes#3140
The Noise handshake accepts any machine key without checking
registration, so all endpoints behind the Noise router are reachable
without credentials. Three handlers used io.ReadAll without size
limits, allowing an attacker to OOM-kill the server.
Fix:
- Add http.MaxBytesReader middleware (1 MiB) on the Noise router.
- Replace io.ReadAll + json.Unmarshal with json.NewDecoder in
PollNetMapHandler and RegistrationHandler.
- Stop reading the body in NotImplementedHandler entirely.
Remove XTestBatcherChannelClosingRace (~95 lines) and
XTestBatcherScalability (~515 lines). These were disabled by
prefixing with X (making them invisible to go test) and served
as dead code. The functionality they covered is exercised by the
active test suite.
Updates #2545
L8: Rename SCREAMING_SNAKE_CASE test constants to idiomatic Go
camelCase. Remove highLoad* and extremeLoad* constants that were
only referenced by disabled (X-prefixed) tests.
L10: Fix misleading assert message that said "1337" while checking
for region ID 999.
L12: Remove emoji from test log output to avoid encoding issues
in CI environments.
Updates #2545
L1: Replace crypto/rand with an atomic counter for generating
connection IDs. These identifiers are process-local and do not need
cryptographic randomness; a monotonic counter is cheaper and
produces shorter, sortable IDs.
L5: Use getActiveConnectionCount() in Debug() instead of directly
locking the mutex and reading the connections slice. This avoids
bypassing the accessor that already exists for this purpose.
L6: Extract the hardcoded 15*time.Minute cleanup threshold into
the named constant offlineNodeCleanupThreshold.
L7: Inline the trivial addWork wrapper; AddWork now calls addToBatch
directly.
Updates #2545
Move connectionEntry, multiChannelNodeConn, generateConnectionID, and
all their methods from batcher.go into a dedicated file. This reduces
batcher.go from ~1170 lines to ~800 and separates per-node connection
management from batcher orchestration.
Pure move — no logic changes.
Updates #2545
- TestBatcher_CloseBeforeStart_DoesNotHang: verifies Close() before
Start() returns promptly now that done is initialized in NewBatcher.
- TestBatcher_QueueWorkAfterClose_DoesNotHang: verifies queueWork
returns via the done channel after Close(), even without Start().
- TestIsConnected_FalseAfterAddNodeFailure: verifies IsConnected
returns false after AddNode fails and removes the last connection.
- TestRemoveConnectionAtIndex_NilsTrailingSlot: verifies the backing
array slot is nil-ed after removal to avoid retaining pointers.
Updates #2545
M7: Nil out trailing *connectionEntry pointers in the backing array
after slice removal in removeConnectionAtIndexLocked and send().
Without this, the GC cannot collect removed entries until the slice
is reallocated.
M1: Initialize the done channel in NewBatcher instead of Start().
Previously, calling Close() or queueWork before Start() would select
on a nil channel, blocking forever. Moving the make() to the
constructor ensures the channel is always usable.
M2: Move b.connected.Delete and b.totalNodes decrement inside the
Compute callback in cleanupOfflineNodes. Previously these ran after
the Compute returned, allowing a concurrent AddNode to reconnect
between the delete and the bookkeeping update, which would wipe the
fresh connected state.
M3: Call markDisconnectedIfNoConns on AddNode error paths. Previously,
when initial map generation or send timed out, the connection was
removed but b.connected retained its old nil (= connected) value,
making IsConnected return true for a node with zero connections.
Updates #2545
Add four unit tests guarding fixes introduced in recent commits:
- TestConnectionEntry_SendFastPath_TimerStopped: verifies the
time.NewTimer fix (H1) does not leak goroutines after many
fast-path sends on a buffered channel.
- TestBatcher_CloseWaitsForWorkers: verifies Close() blocks until all
worker goroutines exit (H3), preventing sends on torn-down channels.
- TestBatcher_CloseThenStartIsNoop: verifies the one-shot lifecycle
contract; Start() after Close() must not spawn new goroutines.
- TestBatcher_CloseStopsTicker: verifies Close() stops the internal
ticker to prevent resource leaks.
Updates #2545
Remove Caller(), channel pointer formatting (fmt.Sprintf("%p",...)),
and mutex timing from send(), addConnection(), and
removeConnectionByChannel(). Move per-broadcast summary and
no-connection logs from Debug to Trace. Remove per-connection
"attempting"/"succeeded" logs entirely; keep Warn for failures.
These methods run on every MapResponse delivery, so the savings
compound quickly under load.
Updates #2545
Close() previously closed the done channel and returned immediately,
without waiting for worker goroutines to exit. This caused goroutine
leaks in tests and allowed workers to race with connection teardown.
The ticker was also never stopped, leaking its internal goroutine.
Add a sync.WaitGroup to track the doWork goroutine and every worker
it spawns. Close() now calls wg.Wait() after signalling shutdown,
ensuring all goroutines have exited before tearing down connections.
Also stop the ticker to prevent resource leaks.
Document that a Batcher must not be reused after Close().
connectionEntry.send() is on the hot path: called once per connection
per broadcast tick. time.After allocates a timer that sits in the
runtime timer heap until it fires (50 ms), even when the channel send
succeeds immediately. At 1000 connected nodes, every tick leaks 1000
timers into the heap, creating continuous GC pressure.
Replace with time.NewTimer + defer timer.Stop() so the timer is
removed from the heap as soon as the fast-path send completes.
Add embedded DERP server, TLS, and netfilter=off to match the
infrastructure configuration used by all other ACL integration tests.
Without these options, the test fails intermittently because traffic
routes through external DERP relays and iptables initialization fails
in Docker containers.
Updates #3139
Remove the Batcher interface since there is only one implementation.
Rename LockFreeBatcher to Batcher and merge batcher_lockfree.go into
batcher.go.
Drop type assertions in debug.go now that mapBatcher is a concrete
*mapper.Batcher pointer.
Rewrite multiChannelNodeConn.send() to use a two-phase approach:
1. RLock: snapshot connections slice (cheap pointer copy)
2. Unlock: send to all connections (50ms timeouts happen here)
3. Lock: remove failed connections by pointer identity
Previously, send() held the write lock for the entire duration of
sending to all connections. With N stale connections each timing out
at 50ms, this blocked addConnection/removeConnection for N*50ms.
The two-phase approach holds the lock only for O(N) pointer
operations, not for N*50ms I/O waits.
Replace the two-phase Load-check-Delete in cleanupOfflineNodes with
xsync.Map.Compute() for atomic check-and-delete. This prevents the
TOCTOU race where a node reconnects between the hasActiveConnections
check and the Delete call.
Add nil guards on all b.nodes.Load() and b.nodes.Range() call sites
to prevent nil pointer panics from concurrent cleanup races.
Move per-node pending changes from a shared xsync.Map on the batcher
into multiChannelNodeConn, protected by a dedicated mutex. The new
appendPending/drainPending methods provide atomic append and drain
operations, eliminating data races in addToBatch and
processBatchedChanges.
Add sync.Once to multiChannelNodeConn.close() to make it idempotent,
preventing panics from concurrent close calls on the same channel.
Add started atomic.Bool to guard Start() against being called
multiple times, preventing orphaned goroutines.
Add comprehensive concurrency tests validating these changes.
Add comprehensive unit tests for the LockFreeBatcher covering
AddNode/RemoveNode lifecycle, addToBatch routing (broadcast, targeted,
full update), processBatchedChanges deduplication, cleanup of offline
nodes, close/shutdown behavior, IsConnected state tracking, and
connected map consistency.
Add benchmarks for connection entry send, multi-channel send and
broadcast, peer diff computation, sentPeers updates, addToBatch at
various scales (10/100/1000 nodes), processBatchedChanges, broadcast
delivery, IsConnected lookups, connected map enumeration, connection
churn, and concurrent send+churn scenarios.
Widen setupBatcherWithTestData to accept testing.TB so benchmarks can
reuse the same database-backed test setup as unit tests.
Add golang.org/x/tools/cmd/stress as a tool dependency for running
tests under repeated stress to surface flaky failures.
Update flake vendorHash for the new go.mod dependencies.
Buffer the AuthRequest verdict channel to prevent a race where the
sender blocks indefinitely if the receiver has already timed out, and
increase the auth followup test timeout from 100ms to 5s to prevent
spurious failures under load.
Skip postgres-backed tests when the postgres server is unavailable
instead of calling t.Fatal, which was preventing the rest of the test
suite from running.
Add TestMain to db, types, and policy/v2 packages to chdir to the
source directory before running tests. This ensures relative testdata/
paths resolve correctly when the test binary is executed from an
arbitrary working directory (e.g., via "go tool stress").
When stale-send cleanup prunes a connection from the batcher, the old serveLongPoll session needs an explicit stop signal. Pass a stop hook into AddNode and trigger it when that connection is removed, so the session exits through its normal cancel path instead of relying on channel closure from the batcher side.
When the batcher timed out sending to a node, it removed the channel from multiChannelNodeConn but left the old serveLongPoll goroutine running on that channel. That left a live stale session behind: it no longer received new updates, but it could still keep the stream open and block shutdown.
Close the pruned channel when stale-send cleanup removes it so the old map session exits after draining any buffered update.
A connection can already be removed from multiChannelNodeConn by the stale-send cleanup path before serveLongPoll reaches its deferred RemoveNode call. In that case RemoveNode used to return early on "channel not found" and never updated the node's connected state.
Drop that early return so RemoveNode still checks whether any active connections remain and marks the node disconnected when the last one is gone.
Update mdformat and related packages from python313Packages to
python314Packages. All four packages (mdformat, mdformat-footnote,
mdformat-frontmatter, mdformat-mkdocs) are available in the updated
nixpkgs.
Updates #1261
Explicitly set derp.urls to an empty list in the NixOS VM test,
matching the upstream nixpkgs test. The VMs have no internet
access, so fetching the default Tailscale DERP map would silently
fail and add unnecessary timeout delay to the test run.
Add missing typed options from the upstream nixpkgs module:
- configFile: read-only option exposing the generated config path
for composability with other NixOS modules
- dns.split: split DNS configuration with proper type checking
- dns.extra_records: typed submodule with name/type/value validation
Sync descriptions and assertions with upstream:
- Use Tailscale doc link for override_local_dns description
- Remove redundant requirement note from nameservers.global
- Match upstream assertion message wording and expression style
Update systemd script to reference cfg.configFile instead of a
local let-binding, matching the upstream pattern.
Add end-to-end integration test that validates localpart:*@domain
SSH user mapping with real Tailscale clients. The test sets up an
SSH policy with localpart entries and verifies that users can SSH
into tagged servers using their email local-part as the username.
Updates #3049
Add support for localpart:*@<domain> entries in SSH policy users.
When a user SSHes into a target, their email local-part becomes the
OS username (e.g. alice@example.com → OS user alice).
Type system (types.go):
- SSHUser.IsLocalpart() and ParseLocalpart() for validation
- SSHUsers.LocalpartEntries(), NormalUsers(), ContainsLocalpart()
- Enforces format: localpart:*@<domain> (wildcard-only)
- UserWildcard.Resolve for user:*@domain SSH source aliases
- acceptEnv passthrough for SSH rules
Compilation (filter.go):
- resolveLocalparts: pure function mapping users to local-parts
by email domain. No node walking, easy to test.
- groupSourcesByUser: single walk producing per-user principals
with sorted user IDs, and tagged principals separately.
- ipSetToPrincipals: shared helper replacing 6 inline copies.
- selfPrincipalsForNode: self-access using pre-computed byUser.
The approach separates data gathering from rule assembly. Localpart
rules are interleaved per source user to match Tailscale SaaS
first-match-wins ordering.
Updates #3049
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
NodeView.CanAccess called node2.AsStruct() on every check. In peer-map construction we run CanAccess in O(n^2) pair scans (often twice per pair), so that per-call clone multiplied into large heap churn
Add ReadLog method to headscale integration container for log
inspection. Split SSH check mode tests into CLI and OIDC variants
and add comprehensive test coverage:
- TestSSHOneUserToOneCheckModeCLI: basic check mode with CLI approval
- TestSSHOneUserToOneCheckModeOIDC: check mode with OIDC approval
- TestSSHCheckModeUnapprovedTimeout: rejection on cache expiry
- TestSSHCheckModeCheckPeriodCLI: session expiry and re-auth
- TestSSHCheckModeAutoApprove: auto-approval within check period
- TestSSHCheckModeNegativeCLI: explicit rejection via CLI
Update existing integration tests to use headscale auth register.
Updates #1850
Add SSH check period tracking so that recently authenticated users
are auto-approved without requiring manual intervention each time.
Introduce SSHCheckPeriod type with validation (min 1m, max 168h,
"always" for every request) and encode the compiled check period
as URL query parameters in the HoldAndDelegate URL.
The SSHActionHandler checks recorded auth times before creating a
new HoldAndDelegate flow. Auth timestamps are stored in-memory:
- Default period (no explicit checkPeriod): auth covers any
destination, keyed by source node with Dst=0 sentinel
- Explicit period: auth covers only that specific destination,
keyed by (source, destination) pair
Auth times are cleared on policy changes.
Updates #1850
Add gRPC service definitions for managing auth requests:
AuthRegister to register interactive auth sessions and
AuthApprove/AuthReject to approve or deny pending requests
(used for SSH check mode).
Updates #1850
Implement the SSH "check" action which requires additional
verification before allowing SSH access. The policy compiler generates
a HoldAndDelegate URL that the Tailscale client calls back to
headscale. The SSHActionHandler creates an auth session and waits for
approval via the generalised auth flow.
Sort check (HoldAndDelegate) rules before accept rules to match
Tailscale's first-match-wins evaluation order.
Updates #1850
Extract shared HTML/CSS design into a common template and create
generalised auth success and web auth templates that work for both
node registration and SSH check authentication flows.
Updates #1850
Generalise the registration pipeline to a more general auth pipeline
supporting both node registrations and SSH check auth requests.
Rename RegistrationID to AuthID, unexport AuthRequest fields, and
introduce AuthVerdict to unify the auth finish API.
Add the urlParam generic helper for extracting typed URL parameters
from chi routes, used by the new auth request handler.
Updates #1850
Replace gorilla/mux with go-chi/chi as the HTTP router and add a
custom zerolog-based request logger to replace chi's default
stdlib-based middleware.Logger, consistent with the rest of the
application.
Updates #1850
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Needs More Info - Timer / remove-label-on-response (push) Has been cancelled
Needs More Info - Timer / close-stale (push) Has been cancelled
update-flake-lock / lockfile (push) Has been cancelled
GitHub Actions Version Updater / build (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Extract the existing-node lookup logic from HandleNodeFromPreAuthKey
into a separate method. This reduces the cyclomatic complexity from
32 to 28, below the gocyclo limit of 30.
Updates #3077
Tagged nodes no longer have user_id set, so ListNodes(user) cannot
find them. Update integration tests to use ListNodes() (all nodes)
when looking up tagged nodes.
Add a findNode helper to locate nodes by predicate from an
unfiltered list, used in ACL tests that have multiple nodes per
scenario.
Updates #3077
Test the tagged-node-survives-user-deletion scenario at two layers:
DB layer (users_test.go):
- success_user_only_has_tagged_nodes: tagged nodes with nil
user_id do not block user deletion and survive it
- error_user_has_tagged_and_owned_nodes: user-owned nodes
still block deletion even when tagged nodes coexist
App layer (grpcv1_test.go):
- TestDeleteUser_TaggedNodeSurvives: full registration flow
with tagged PreAuthKey verifies nil UserID after registration,
absence from nodesByUser index, user deletion succeeds, and
tagged node remains in global node list
Also update auth_tags_test.go assertions to expect nil UserID
on tagged nodes, consistent with the new invariant.
Updates #3077
Tagged nodes are owned by their tags, not a user. Enforce this
invariant at every write path:
- createAndSaveNewNode: do not set UserID for tagged PreAuthKey
registration; clear UserID when advertise-tags are applied
during OIDC/CLI registration
- SetNodeTags: clear UserID/User when tags are assigned
- processReauthTags: clear UserID/User when tags are applied
during re-authentication
- validateNodeOwnership: reject tagged nodes with non-nil UserID
- NodeStore: skip nodesByUser indexing for tagged nodes since
they have no owning user
- HandleNodeFromPreAuthKey: add fallback lookup for tagged PAK
re-registration (tagged nodes indexed under UserID(0)); guard
against nil User deref for tagged nodes in different-user check
Since tagged nodes now have user_id = NULL, ListNodesByUser
will not return them and DestroyUser naturally allows deleting
users whose nodes have all been tagged. The ON DELETE CASCADE
FK cannot reach tagged nodes through a NULL foreign key.
Also tone down shouty comments throughout state.go.
Fixes#3077
Tagged nodes are owned by their tags, not a user. Previously
user_id was kept as "created by" tracking, but this prevents
deleting users whose nodes have all been tagged, and the
ON DELETE CASCADE FK would destroy the tagged nodes.
Add a migration that sets user_id = NULL on all existing tagged
nodes. Subsequent commits enforce this invariant at write time.
Updates #3077
Add --disable flag to "headscale nodes expire" CLI command and
disable_expiry field handling in the gRPC API to allow disabling
key expiry for nodes. When disabled, the node's expiry is set to
NULL and IsExpired() returns false.
The CLI follows the new grpcRunE/RunE/printOutput patterns
introduced in the recent CLI refactor.
Also fix NodeSetExpiry to persist directly to the database instead
of going through persistNodeToDB which omits the expiry field.
Fixes#2681
Co-authored-by: Marco Santos <me@marcopsantos.com>
Add bool disable_expiry field (field 3) to ExpireNodeRequest proto
and regenerate all protobuf, gRPC gateway, and OpenAPI files.
Fixes#2681
Co-authored-by: Marco Santos <me@marcopsantos.com>
Add end-to-end test cases to TestUnmarshalPolicy that verify bracketed
IPv6 addresses are correctly parsed through the full policy pipeline
(JSON unmarshal -> splitDestinationAndPort -> parseAlias -> parsePortRange)
and survive JSON round-trips.
Cover single port, multiple ports, wildcard port, CIDR prefix, port
range, bracketed IPv4, and hostname rejection.
Updates #2754
Headscale rejects IPv6 addresses with square brackets in ACL policy
destinations (e.g. "[fd7a:115c:a1e0::87e1]:80,443"), while Tailscale
SaaS accepts them. The root cause is that splitDestinationAndPort uses
strings.LastIndex(":") which leaves brackets on the destination string,
and netip.ParseAddr does not accept brackets.
Add a bracket-handling branch at the top of splitDestinationAndPort that
uses net.SplitHostPort for RFC 3986 parsing when input starts with "[".
The extracted host is validated with netip.ParseAddr/ParsePrefix to
ensure brackets are only accepted around IP addresses and CIDR prefixes,
not hostnames or other alias types like tags and groups.
Fixes#2754
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Needs More Info - Timer / remove-label-on-response (push) Has been cancelled
Needs More Info - Timer / close-stale (push) Has been cancelled
Deploy docs / deploy (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
This just fixes a small issue I noticed reading the docs: the two 'scenarios' listed in the scaling section end up showing up as a numbered list of five items, instead of the desired two items + their descriptions.
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Remove dead if-resp-nil checks in tagCmd and approveRoutesCmd; gRPC
returns either an error or a valid response, never (nil, nil).
Rename HasMachineOutputFlag to hasMachineOutputFlag since it has a
single internal caller in root.go.
Add bypassDatabase() to consolidate the repeated LoadServerConfig +
NewHeadscaleDatabase pattern in getPolicy and setPolicy.
Replace os.Open + io.ReadAll with os.ReadFile in setPolicy and
checkPolicy, removing the manual file-handle management.
Move errMissingParameter from users.go to utils.go alongside the
other shared sentinel errors; the variable is referenced by
api_key.go and preauthkeys.go.
Move the Error constant-error type from debug.go to mockoidc.go,
its only consumer.
Replace five hardcoded "2006-01-02 15:04:05" strings with the
HeadscaleDateTimeFormat constant already defined in utils.go.
Replace two literal 10 base arguments to strconv.FormatUint with
util.Base10 to match the convention in api_key.go and nodes.go.
Flags registered on a cobra.Command cannot fail to read at runtime;
GetString/GetUint64/GetStringSlice only error when the flag name is
unknown. The error-handling blocks for these calls are unreachable
dead code.
Adopt the value, _ := pattern already used in api_key.go,
preauthkeys.go and users.go, removing ~40 lines of dead code from
nodes.go and debug.go.
The --namespace flag on nodes list/register and debug create-node was
never wired to the --user flag, so its value was silently ignored.
Remove it along with the deprecateNamespaceMessage constant.
Also remove the namespace/ns command aliases on users and
machine/machines aliases on nodes, which have been deprecated since
the naming changes in 0.23.0.
Add expirationFromFlag helper that parses the --expiration flag into a
timestamppb.Timestamp, replacing identical duration-parsing blocks in
api_key.go and preauthkeys.go.
Add apiKeyIDOrPrefix helper to validate the mutually-exclusive --id and
--prefix flags, replacing the duplicated switch block in expireAPIKeyCmd
and deleteAPIKeyCmd.
Centralise the repeated force-flag-check + YesNo-prompt logic into a
single confirmAction(cmd, prompt) helper. Callers still decide what
to return on decline (error, message, or nil).
Replace three inconsistent MarkFlagRequired error-handling styles
(stdlib log.Fatal, zerolog log.Fatal, silently discarded) with a
single mustMarkRequired helper that panics on programmer error.
Also fixes a bug where renameNodeCmd.MarkFlagRequired("new-name")
targeted the wrong command (should be renameUserCmd), making the
--new-name flag effectively never required on "headscale users rename".
Add a helper that checks the --output flag and either serialises as
JSON/YAML or invokes a table-rendering callback. This removes the
repeated format,_ := cmd.Flags().GetString("output") + if-branch from
the five list commands.
Convert the 10 commands that were still using Run with
ErrorOutput/SuccessOutput or log.Fatal/os.Exit:
- backfillNodeIPsCmd: use grpcRunE-style manual connection with
error returns; simplify the confirm/force logic
- getPolicy, setPolicy, checkPolicy: replace ErrorOutput with
fmt.Errorf returns in both the bypass-gRPC and gRPC paths
- serveCmd, configTestCmd: replace log.Fatal with error returns
- mockOidcCmd: replace log.Error+os.Exit with error return
- versionCmd, generatePrivateKeyCmd: replace SuccessOutput with
printOutput
- dumpConfigCmd: return the error instead of swallowing it
Rename grpcRun to grpcRunE: the inner closure now returns error
and the wrapper returns a cobra RunE-compatible function.
Change newHeadscaleCLIWithConfig to return an error instead of
calling log.Fatal/os.Exit, making connection failures propagate
through the normal error path.
Add formatOutput (returns error) and printOutput (writes to stdout)
as non-exiting replacements for the old output/SuccessOutput pair.
Extract output format string literals into package-level constants.
Mark the old ErrorOutput, SuccessOutput and output helpers as
deprecated; they remain temporarily for the unconverted commands.
Convert all 22 grpcRunE commands from Run+ErrorOutput+SuccessOutput
to RunE+fmt.Errorf+printOutput. Change usernameAndIDFromFlag to
return an error instead of calling ErrorOutput directly.
Update backfillNodeIPsCmd and policy.go callers of
newHeadscaleCLIWithConfig for the new 5-return signature while
keeping their Run-based pattern for now.
Set SilenceErrors and SilenceUsage on the root command so that
cobra never prints usage text for runtime errors. A SetFlagErrorFunc
callback re-enables usage output specifically for flag-parsing
errors (the kubectl pattern).
Add printError to utils.go and switch Execute() to ExecuteC() so
the returned error is formatted as JSON/YAML when --output requests
machine-readable output.
Add a grpcRun helper that wraps cobra RunFuncs, injecting a ready
gRPC client and context. The connection lifecycle (cancel, close)
is managed by the wrapper, eliminating the duplicated 3-line
boilerplate (newHeadscaleCLIWithConfig + defer cancel + defer
conn.Close) from 22 command handlers across 7 files.
Three call sites are intentionally left unconverted:
- backfillNodeIPsCmd: creates the client only after user confirmation
- getPolicy/setPolicy: conditionally use gRPC vs direct DB access
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Docker 29 (shipped with runner-images 20260209.23.1) breaks docker
build via Go client libraries (broken pipe writing build context)
and docker load/save with certain tarball formats. Add Docker's
official apt repository and install docker-ce 28.5.x in all CI
jobs that interact with Docker.
See https://github.com/actions/runner-images/issues/13474
Updates #3058
Use new(users["name"]) instead of extracting to intermediate
variables that staticcheck does not recognise as used with
Go 1.26 new(value) syntax.
Updates #3058
Fix issues found by the upgraded golangci-lint:
- wsl_v5: add required whitespace in CLI files
- staticcheck SA4006: replace new(var.Field) with &localVar
pattern since staticcheck does not recognize Go 1.26
new(value) as a use of the variable
- staticcheck SA5011: use t.Fatal instead of t.Error for
nil guard checks so execution stops
- unused: remove dead ptrTo helper function
Use GORM AutoMigrate instead of raw SQL to create the
database_versions table, since PostgreSQL does not support the
datetime type used in the raw SQL (it requires timestamp).
Add a version check that runs before database migrations to ensure
users do not skip minor versions or downgrade. This protects database
migrations and allows future cleanup of old migration code.
Rules enforced:
- Same minor version: always allowed (patch changes either way)
- Single minor upgrade (e.g. 0.27 -> 0.28): allowed
- Multi-minor upgrade (e.g. 0.25 -> 0.28): blocked with guidance
- Any minor downgrade: blocked
- Major version change: blocked
- Dev builds: warn but allow, preserve stored version
The version is stored in a purpose-built database_versions table
after migrations succeed. The table is created with raw SQL before
gormigrate runs to avoid circular dependencies.
Updates #3058
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Replace tiangolo/issue-manager with custom logic that distinguishes
bot comments from human responses. The issue-manager action treated
all comments equally, so the bot's own instruction comment would
trigger label removal on the next scheduled run.
Split into two jobs:
- remove-label-on-response: triggers on issue_comment from non-bot
users, removes the needs-more-info label immediately
- close-stale: runs on daily schedule, uses nushell to iterate open
needs-more-info issues, checks for human comments after the label
was added, and closes after 3 days with no response
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Needs More Info - Timer / manage-needs-more-info (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
update-flake-lock / lockfile (push) Has been cancelled
GitHub Actions Version Updater / build (push) Has been cancelled
Remove the `issues: labeled` trigger from the timer workflow.
When both workflows triggered on label addition, the comment workflow
would post the bot comment, and by the time the timer workflow ran,
issue-manager would see "a comment was added after the label" and
immediately remove the label due to `remove_label_on_comment: true`.
The timer workflow now only runs on:
- Daily cron (to close stale issues)
- issue_comment (to remove label when humans respond)
- workflow_dispatch (for manual testing)
Add workflow that automatically closes issues labeled as
support-request with a message directing users to Discord
for configuration and support questions.
The workflow:
- Triggers when support-request label is added
- Posts a comment explaining this tracker is for bugs/features
- Links to documentation and Discord
- Closes the issue as "not planned"
Add GitHub Actions automation that helps manage issues requiring
additional information from reporters:
- Post an instruction comment when 'needs-more-info' label is added,
requesting environment details, debug logs from multiple nodes,
configuration files, and proper formatting
- Automatically remove the label when anyone comments
- Close the issue after 3 days if no response is provided
- Exempt needs-more-info labeled issues from the stale bot
The instruction comment includes guidance on:
- Required environment and debug information
- Collecting logs from both connecting and connected-to nodes
- Proper redaction rules (replace consistently, never remove IPs)
- Formatting requirements for attachments and Markdown
- Encouragement to discuss on Discord before filing issues
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
This commit upgrades the codebase from Go 1.25.5 to Go 1.26rc2 and
adopts new language features.
Toolchain updates:
- go.mod: go 1.25.5 → go 1.26rc2
- flake.nix: buildGo125Module → buildGo126Module, go_1_25 → go_1_26
- flake.nix: build golangci-lint from source with Go 1.26
- Dockerfile.integration: golang:1.25-trixie → golang:1.26rc2-trixie
- Dockerfile.tailscale-HEAD: golang:1.25-alpine → golang:1.26rc2-alpine
- Dockerfile.derper: golang:alpine → golang:1.26rc2-alpine
- .goreleaser.yml: go mod tidy -compat=1.25 → -compat=1.26
- cmd/hi/run.go: fallback Go version 1.25 → 1.26rc2
- .pre-commit-config.yaml: simplify golangci-lint hook entry
Code modernization using Go 1.26 features:
- Replace tsaddr.SortPrefixes with slices.SortFunc + netip.Prefix.Compare
- Replace ptr.To(x) with new(x) syntax
- Replace errors.As with errors.AsType[T]
Lint rule updates:
- Add forbidigo rules to prevent regression to old patterns
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
update-flake-lock / lockfile (push) Has been cancelled
GitHub Actions Version Updater / build (push) Has been cancelled
Update the 0.29.0 changelog entry to document the minimum
supported Tailscale client version (v1.76.0), which corresponds
to capability version 106 based on the 10-version support window.
Errors should not start capitalised and they should not contain the word error
or state that they "failed" as we already know it is an error
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
Go style recommends that log messages and error strings should not be
capitalized (unless beginning with proper nouns or acronyms) and should
not end with punctuation.
This change normalizes all zerolog .Msg() and .Msgf() calls to start
with lowercase letters, following Go conventions and making logs more
consistent across the codebase.
Replace raw string field names with zf constants in state.go and
db/node.go for consistent, type-safe logging.
state.go changes:
- User creation, hostinfo validation, node registration
- Tag processing during reauth (processReauthTags)
- Auth path and PreAuthKey handling
- Route auto-approval and MapRequest processing
db/node.go changes:
- RegisterNodeForTest logging
- Invalid hostname replacement logging
Add sub-logger patterns to worker(), AddNode(), RemoveNode() and
multiChannelNodeConn to eliminate repeated field calls. Use zf.*
constants for consistent field naming.
Changes in batcher_lockfree.go:
- Add wlog sub-logger in worker() with worker.id context
- Add log field to multiChannelNodeConn struct
- Initialize mc.log with node.id in newMultiChannelNodeConn()
- Add nlog sub-loggers in AddNode() and RemoveNode()
- Update all connection methods to use mc.log
Changes in batcher.go:
- Use zf.NodeID and zf.Reason in handleNodeChange()
Replace manual field extraction with EmbedObject for node logging
in gRPC handlers. Use zf.* constants for consistent field naming.
Changes:
- RegisterNode: use EmbedObject(node), zf.RegistrationKey, etc.
- SetTags: use EmbedObject(node)
- ExpireNode: use EmbedObject(node), zf.ExpiresAt
- RenameNode: use EmbedObject(node), zf.NewName
- SetApprovedRoutes: use zf.NodeID
Add sub-logger pattern to SetRoutes() to eliminate repeated node.id
field calls. Replace raw strings with zf.* constants throughout
the primary routes code for consistent field naming.
Changes:
- Add nlog sub-logger in SetRoutes() with node.id context
- Replace "prefix" with zf.Prefix
- Replace "changed" with zf.Changes
- Replace "newState" with zf.NewState
- Replace "finalState" with zf.FinalState
Replace the helper functions (logf, infof, tracef, errf) with a
zerolog sub-logger initialized in newMapSession(). The sub-logger
is pre-populated with session context (component, node, omitPeers,
stream) eliminating repeated field calls throughout the code.
Changes:
- Add log field to mapSession struct
- Initialize sub-logger with EmbedObject(node) and request context
- Remove logf/infof/tracef/errf helper functions
- Update all callers to use m.log.Level().Caller()... pattern
- Update noise.go to use sess.log instead of sess.tracef
This reduces code by ~20 lines and eliminates ~15 repeated field
calls per log statement.
Add a lint rule to enforce use of zf.* constants for zerolog field
names instead of inline string literals. This catches at lint time
any new code that doesn't follow the convention.
The rule matches common zerolog field methods (Str, Int, Bool, etc.)
and flags any usage with a string literal first argument.
Add hscontrol/util/zlog package with:
- zf subpackage: field name constants for compile-time safety
- SafeHostinfo: wrapper that redacts device fingerprinting data
- SafeMapRequest: wrapper that redacts client endpoints
The zf (zerolog fields) subpackage provides short constant names
(e.g., zf.NodeID instead of inline "node.id" strings) ensuring
consistent field naming across all log statements.
Security considerations:
- SafeHostinfo never logs: OSVersion, DeviceModel, DistroName
- SafeMapRequest only logs endpoint counts, not actual IPs
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Update integration test expectations to match current policy behavior:
1. IPProto defaults include all four protocols (TCP, UDP, ICMPv4,
ICMPv6) for port-range ACL rules, not just TCP and UDP.
2. Filter rules with identical SrcIPs and IPProto are now merged
into a single rule with combined DstPorts, so the subnet router
receives one filter rule instead of two.
Updates #3036
According to Tailscale SaaS behavior, autogroup:internet is handled
by exit node routing via AllowedIPs, not by packet filtering. ACL
rules with autogroup:internet as destination should produce no
filter rules for any node.
Previously, Headscale expanded autogroup:internet to public CIDR
ranges and distributed filters to exit nodes (because 0.0.0.0/0
"covers" internet destinations). This was incorrect.
Add detection for AutoGroupInternet in filter compilation to skip
filter generation for this autogroup. Update test expectations
accordingly.
Fix two compatibility issues discovered in Tailscale SaaS testing:
1. Wildcard DstPorts format: Headscale was expanding wildcard
destinations to CGNAT ranges (100.64.0.0/10, fd7a:115c:a1e0::/48)
while Tailscale uses {IP: "*"} directly. Add detection for
wildcard (Asterix) alias type in filter compilation to use the
correct format.
2. proto:icmp handling: The "icmp" protocol name was returning both
ICMPv4 (1) and ICMPv6 (58), but Tailscale only returns ICMPv4.
Users should use "ipv6-icmp" or protocol number 58 explicitly
for IPv6 ICMP.
Update all test expectations accordingly. This significantly reduces
test file line count by replacing duplicated CGNAT range patterns
with single wildcard entries.
Update test expectations across policy tests to expect merged
FilterRule entries instead of separate ones. Tests now expect:
- Single FilterRule with combined DstPorts for same source
- Reduced matcher counts for exit node tests
Updates #3036
Tailscale merges multiple ACL rules into fewer FilterRule entries
when they have identical SrcIPs and IPProto, combining their DstPorts
arrays. This change implements the same behavior in Headscale.
Add mergeFilterRules() which uses O(n) hash map lookup to merge rules
with identical keys. DstPorts are NOT deduplicated to match Tailscale
behavior.
Also fix DestsIsTheInternet() to handle merged filter rules where
TheInternet is combined with other destinations - now uses superset
check instead of equality check.
Updates #3036
Change Asterix.Resolve() to use Tailscale's CGNAT range (100.64.0.0/10)
and ULA range (fd7a:115c:a1e0::/48) instead of all IPs (0.0.0.0/0 and
::/0).
This better matches Tailscale's security model where wildcard (*) means
"any node in the tailnet" rather than literally "any IP address on the
internet".
Updates #3036
Updates #3036
Tailscale validates that autogroup:self destinations in ACL rules can
only be used when ALL sources are users, groups, autogroup:member, or
wildcard (*). Previously, Headscale only performed this validation for
SSH rules.
Add validateACLSrcDstCombination() to enforce that tags, autogroup:tagged,
hosts, and raw IPs cannot be used as sources with autogroup:self
destinations. Invalid policies like `tag:client → autogroup:self:*` are
now rejected at validation time, matching Tailscale behavior.
Wildcard (*) is allowed because autogroup:self evaluation narrows it
per-node to only the node's own IPs.
Updates #3036
When ACL rules don't specify a protocol, Headscale now defaults to
[TCP, UDP, ICMP, ICMPv6] instead of just [TCP, UDP], matching
Tailscale's behavior.
Also export protocol number constants (ProtocolTCP, ProtocolUDP, etc.)
for use in external test packages, renaming the string protocol
constants to ProtoNameTCP, ProtoNameUDP, etc. to avoid conflicts.
This resolves 78 ICMP-related TODOs in the Tailscale compatibility
tests, reducing the total from 165 to 87.
Updates #3036
Add extensive test coverage verifying Headscale's ACL policy behavior
matches Tailscale's coordination server. Tests cover:
- Source/destination resolution for users, groups, tags, hosts, IPs
- autogroup:member, autogroup:tagged, autogroup:self behavior
- Filter rule deduplication and merging semantics
- Multi-rule interaction patterns
- Error case validation
Key behavioral differences documented:
- Headscale creates separate filter entries per ACL rule; Tailscale
merges rules with identical sources
- Headscale deduplicates Dsts within a rule; Tailscale does not
- Headscale does not validate autogroup:self source restrictions for
ACL rules (only SSH rules); Tailscale rejects invalid sources
Tests are based on real Tailscale coordination server responses
captured from a test environment with 5 nodes (1 user-owned, 4 tagged).
Updates #3036
Skip autogroup:self destination processing for tagged nodes since they
can never match autogroup:self (which only applies to user-owned nodes).
Also reorder the IsTagged() check to short-circuit before accessing
User() to avoid potential nil pointer access on tagged nodes.
Updates #3036
Add TestTagsAuthKeyConvertToUserViaCLIRegister that reproduces the
exact panic from #3038: register a node with a tags-only PreAuthKey
(no user), force reauth with empty tags, then register via CLI with
a user. The mapper panics on node.Owner().Model().ID when User is nil.
The critical detail is using a tags-only PreAuthKey (User: nil). When
the key is created under a user, the node inherits the User pointer
from createAndSaveNewNode and the bug is masked.
Also add Owner() validity assertions to the existing unit test
TestTaggedNodeWithoutUserToDifferentUser to catch the nil pointer
at the unit test level.
Updates #3038
processReauthTags sets UserID when converting a tagged node to
user-owned, but does not set the User pointer. When the node was
registered with a tags-only PreAuthKey (User: nil), the in-memory
NodeStore cache holds a node with User=nil. The mapper's
generateUserProfiles then calls node.Owner().Model().ID, which
dereferences the nil pointer and panics.
Set node.User alongside node.UserID in processReauthTags. Also add
defensive nil checks in generateUserProfiles to gracefully handle
nodes with invalid owners rather than panicking.
Fixes#3038
These were thin wrappers around applyAuthNodeUpdate that only added
logging. Move the logging into applyAuthNodeUpdate and call it directly
from HandleNodeFromAuthPath.
This simplifies the code structure without changing behavior.
Updates #3038
Move tag validation before the UpdateNode callback in applyAuthNodeUpdate.
Previously, tag validation happened inside the callback, and the error
check occurred after UpdateNode had already committed changes to the
NodeStore. This left the NodeStore in an inconsistent state when tags
were rejected.
Now validation happens first, and UpdateNode is only called when we know
the operation will succeed. This follows the principle that UpdateNode
should only be called when we have all information and are ready to commit.
Also extract validateRequestTags as a reusable function and use it in
createAndSaveNewNode to deduplicate the tag validation logic.
Updates #3038
Updates #3048
Previously, expiry handling ran BEFORE processReauthTags(), using the
old tagged status to determine whether to set/clear expiry. This caused:
- Personal → Tagged: Expiry remained set (should be cleared to nil)
- Tagged → Personal: Expiry remained nil (should be set from client)
Move expiry handling after tag processing and handle all four transition
cases based on the new tagged status:
- Tagged → Personal: Set expiry from client request
- Personal → Tagged: Clear expiry (tagged nodes don't expire)
- Personal → Personal: Update expiry from client
- Tagged → Tagged: Keep existing nil expiry
Fixes#3048
Reorganize HandleNodeFromAuthPath (~300 lines) into a cleaner structure
with named conditions and extracted helper functions.
Changes:
- Add authNodeUpdateParams struct for shared update logic
- Extract applyAuthNodeUpdate for common reauth/convert operations
- Extract reauthExistingNode and convertTaggedNodeToUser handlers
- Extract createNewNodeFromAuth for new node creation
- Use named boolean conditions (nodeExistsForSameUser, existingNodeIsTagged,
existingNodeOwnedByOtherUser) instead of compound if conditions
- Create logger with common fields (registration_id, user.name, machine.key,
method) to reduce log statement verbosity
Updates #3038
When a node was registered with a tags-only PreAuthKey (no user
associated), the node had User=nil and UserID=nil. When attempting to
re-register this node to a different user via HandleNodeFromAuthPath,
two issues occurred:
1. The code called oldUser.Name() without checking if oldUser was valid,
causing a nil pointer dereference panic.
2. The existing node lookup logic didn't find the tagged node because it
searched by (machineKey, userID), but tagged nodes have no userID.
This caused a new node to be created instead of updating the existing
tagged node.
Fix this by restructuring HandleNodeFromAuthPath to:
1. First check if a node exists for the same user (existing behavior)
2. If not found, check if an existing TAGGED node exists with the same
machine key (regardless of userID)
3. If a tagged node exists, UPDATE it to convert from tagged to
user-owned (preserving the node ID)
4. Only create a new node if the existing node is user-owned by a
different user
This ensures consistent behavior between:
- personal → tagged → personal (same node, same owner)
- tagged (no user) → personal (same node, new owner)
Add a test that reproduces the panic and conversion scenario by:
1. Creating a tags-only PreAuthKey (no user)
2. Registering a node with that key
3. Re-registering the same machine to a different user
4. Verifying the node ID stays the same (conversion, not creation)
Fixes#3038
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Both compileFilterRules and compileSSHPolicy include .Caller() on
their resolution error log statements, but compileACLWithAutogroupSelf
does not. Add .Caller() to the three log sites (source resolution
error, destination resolution error, nil destination) for consistent
debuggability across all compilation paths.
Updates #2990
In compileSSHPolicy, when resolving other (non-autogroup:self)
destinations, the code discards the entire result on error via
`continue`. If a destination alias (e.g., a tag owned by a group
with a non-existent user) returns a partial IPSet alongside an
error, valid IPs are lost.
Both ACL compilation paths (compileFilterRules and
compileACLWithAutogroupSelf) already handle this correctly by
logging the error and using the IPSet if non-nil.
Remove the `continue` so the SSH path is consistent with the
ACL paths.
Fixes#2990
Three related issues where User().ID() is called on potentially tagged
nodes without first checking IsTagged():
1. compileACLWithAutogroupSelf: the autogroup:self block at line 166
lacks the !node.IsTagged() guard that compileSSHPolicy already has.
If a tagged node is the compilation target, node.User().ID() may
panic. Tagged nodes should never participate in autogroup:self.
2. compileSSHPolicy: the IsTagged() check is on the right side of &&,
so n.User().ID() evaluates first and may panic before short-circuit
can prevent it. Swap to !n.IsTagged() && n.User().ID() == ... to
match the already-correct order in compileACLWithAutogroupSelf.
3. invalidateAutogroupSelfCache: calls User().ID() at ~10 sites
without IsTagged() guards. Tagged nodes don't participate in
autogroup:self, so they should be skipped when collecting affected
users and during cache lookup. Tag status transitions are handled
by using the non-tagged version's user ID.
Fixes#2990
In compileACLWithAutogroupSelf, when a group contains a non-existent
user, Group.Resolve() returns a partial IPSet (with IPs from valid
users) alongside an error. The code was discarding the entire result
via `continue`, losing valid IPs. The non-autogroup-self path
(compileFilterRules) already handles this correctly by logging the
error and using the IPSet if non-empty.
Remove the `continue` on error for both source and destination
resolution, matching the existing behavior in compileFilterRules.
Also reorder the IsTagged check before User().ID() comparison
in the same-user node filter to prevent nil dereference on tagged
nodes that have no User set.
Fixes#2990
Add test reproducing the exact scenario from issue #2990 where:
- One user (user1) in group:admin
- node1: user device (not tagged)
- node2: tagged with tag:admin, same user
The test verifies that peer visibility and packet filters are correct.
Updates #2990
Previously, nodes with empty filter rules (e.g., tagged servers that are
only destinations, never sources) were skipped entirely in BuildPeerMap.
This could cause visibility issues when using autogroup:self with
multiple user groups.
Remove the len(filter) == 0 skip condition so all nodes are included in
nodeMatchers. Empty filters result in empty matchers where CanAccess()
returns false, but the node still needs to be in the map so symmetric
visibility works correctly: if node A can access node B, both should see
each other regardless of B's filter rules.
Add comprehensive tests for:
- Multi-group scenarios where autogroup:self is used by privileged users
- Nodes with empty filters remaining visible to authorized peers
- Combined access rules (autogroup:self + tags in same rule)
Updates #2990
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
When a PreAuthKey is deleted, the database correctly sets auth_key_id
to NULL on referencing nodes via ON DELETE SET NULL. However, the
NodeStore (in-memory cache) retains the old AuthKeyID value.
When nodes send MapRequests (e.g., after tailscaled restart), GORM's
Updates() tries to persist the stale AuthKeyID, causing a foreign key
constraint error when trying to reference a deleted PreAuthKey.
Fix this by adding AuthKeyID and AuthKey to the Omit() call in all
three places where nodes are updated via GORM's Updates():
- persistNodeToDB (MapRequest processing)
- HandleNodeFromAuthPath (re-auth via web/OIDC)
- HandleNodeFromPreAuthKey (re-registration with preauth key)
This tells GORM to never touch the auth_key_id column or AuthKey
association during node updates, letting the database handle the
foreign key relationship correctly.
Added TestDeletedPreAuthKeyNotRecreatedOnNodeUpdate to verify that
deleted PreAuthKeys are not recreated when nodes send MapRequests.
Update integration tests to use valid SSH patterns:
- TestSSHOneUserToAll: use autogroup:member and autogroup:tagged
instead of wildcard destination
- TestSSHMultipleUsersAllToAll: use autogroup:self instead of
username destinations for group-to-user SSH access
Updates #3009
Updates #3010
Update unit tests to use valid SSH patterns that conform to Tailscale's
security model:
- Change group->user destinations to group->tag
- Change tag->user destinations to tag->tag
- Update expected error messages for new validation format
- Add proper tagged/untagged node setup in filter tests
Updates #3009
Updates #3010
Add validation for SSH source/destination combinations that enforces
Tailscale's security model:
- Tags/autogroup:tagged cannot SSH to user-owned devices
- autogroup:self destination requires source to contain only users/groups
- Username destinations require source to be that same single user only
- Wildcard (*) is no longer supported as SSH destination; use
autogroup:member or autogroup:tagged instead
The validateSSHSrcDstCombination() function is called during policy
validation to reject invalid configurations at load time.
Fixes#3009Fixes#3010
Refactor the RequestTags migration (202601121700-migrate-hostinfo-request-tags)
to use PolicyManager.NodeCanHaveTag() instead of reimplementing tag validation.
Changes:
- NewHeadscaleDatabase now accepts *types.Config to allow migrations
access to policy configuration
- Add loadPolicyBytes helper to load policy from file or DB based on config
- Add standalone GetPolicy(tx *gorm.DB) for use during migrations
- Replace custom tag validation logic with PolicyManager
Benefits:
- Full HuJSON parsing support (not just JSON)
- Proper group expansion via PolicyManager
- Support for nested tags and autogroups
- Works with both file and database policy modes
- Single source of truth for tag validation
Co-Authored-By: Shourya Gautam <shouryamgautam@gmail.com>
When autogroup:self was combined with other ACL rules (e.g., group:admin
-> *:*), tagged nodes became invisible to users who should have access.
The BuildPeerMap function had two code paths:
- Global filter path: used symmetric OR logic (if either can access, both
see each other)
- Autogroup:self path: used asymmetric logic (only add peer if that
specific direction has access)
This caused problems with one-way rules like admin -> tagged-server. The
admin could access the server, but since the server couldn't access the
admin, neither was added to the other's peer list.
Fix by using symmetric visibility in the autogroup:self path, matching
the global filter path behavior: if either node can access the other,
both should see each other as peers.
Credit: vdovhanych <vdovhanych@users.noreply.github.com>
Fixes#2990
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Extend TestApiKeyCommand to test the new --id flag for expire and
delete commands, verifying that API keys can be managed by their
database ID in addition to the existing --prefix method.
Updates #2986
Add --id flag as an alternative to --prefix for expiring and
deleting API keys. This allows users to use the ID shown in
'headscale apikeys list' output, which is more convenient than
the prefix.
Either --id or --prefix must be provided; both flags are optional
but at least one is required.
Updates #2986
Update ExpireApiKey and DeleteApiKey handlers to accept either ID or
prefix for identifying the API key. Returns InvalidArgument error if
neither or both are provided.
Add tests for:
- Expire by ID
- Expire by prefix (backwards compatibility)
- Delete by ID
- Delete by prefix (backwards compatibility)
- Error when neither ID nor prefix provided
- Error when both ID and prefix provided
Updates #2986
Add id field to ExpireApiKeyRequest and DeleteApiKeyRequest messages.
This allows API keys to be expired or deleted by their database ID
in addition to the existing prefix-based lookup.
Updates #2986
Add GetAPIKeyByID method to the state layer, delegating to the existing
database layer function. This enables API key lookup by ID in addition
to the existing prefix-based lookup.
Updates #2986
Convert config loading tests from gopkg.in/check.v1 Suite-based testing
to standard Go tests with testify assert/require.
Changes:
- Remove Suite boilerplate (Test, Suite type, SetUpSuite, TearDownSuite)
- Convert TestConfigFileLoading and TestConfigLoading to standalone tests
- Replace check assertions with testify equivalents
Migrate all database tests from gopkg.in/check.v1 Suite-based testing
to standard Go tests with testify assert/require.
Changes:
- Remove empty Suite files (hscontrol/suite_test.go, hscontrol/mapper/suite_test.go)
- Convert hscontrol/db/suite_test.go to modern helpers only
- Convert 6 Suite test methods in node_test.go to standalone tests
- Convert 5 Suite test methods in api_key_test.go to standalone tests
- Fix stale global variable reference in db_test.go
The legacy TestListPeers Suite method was renamed to TestListPeersManyNodes
to avoid conflict with the existing modern TestListPeers function, as they
test different aspects (basic peer listing vs ID filtering).
Make DeleteUser call updatePolicyManagerUsers() to refresh the policy
manager's cached user list after user deletion. This ensures consistency
with CreateUser, UpdateUser, and RenameUser which all update the policy
manager.
Previously, DeleteUser only removed the user from the database without
updating the policy manager. This could leave stale user references in
the cached user list, potentially causing issues when policy is
re-evaluated.
The gRPC handler now uses the change returned from DeleteUser instead of
manually constructing change.UserRemoved().
Fixes#2967
Add DeleteUser method to ControlServer interface and implement it in
HeadscaleInContainer to enable testing user deletion scenarios.
Add two integration tests for issue #2967:
- TestACLGroupWithUnknownUser: tests that valid users can communicate
when a group references a non-existent user
- TestACLGroupAfterUserDeletion: tests connectivity after deleting a
user that was referenced in an ACL group
These tests currently pass but don't fully reproduce the reported issue
where deleted users break connectivity for the entire group.
Updates #2967
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Replace the Tags column with an Owner column that displays:
- Tags (newline-separated) if the key has ACL tags
- User name if the key is associated with a user
- Dash (-) if neither is present
This aligns the CLI output with the tags-as-identity model where
preauthkeys can be created with either tags or user ownership.
- Rename TestTagsAuthKeyWithoutUserIgnoresAdvertisedTags to
TestTagsAuthKeyWithoutUserRejectsAdvertisedTags to reflect actual
behavior (PreAuthKey registrations reject advertised tags)
- Fix TestTagsAuthKeyWithoutUserInheritsTags to use ListNodes() without
user filter since tags-only nodes don't have a user association
Updates #2977
HandleNodeFromPreAuthKey assumed pak.User was always set, but
tags-only PreAuthKeys have nil User. This caused nil pointer
dereference when registering nodes with tags-only keys.
Also updates integration tests to use GetTags() instead of the
removed GetValidTags() method.
Updates #2977
When SetNodeTags changed a node's tags, the node's self view wasn't
updated. The bug manifested as: the first SetNodeTags call updates
the server but the client's self view doesn't update until a second
call with the same tag.
Root cause: Three issues combined to prevent self-updates:
1. SetNodeTags returned PolicyChange which doesn't set OriginNode,
so the mapper's self-update check failed.
2. The Change.Merge function didn't preserve OriginNode, so when
changes were batched together, OriginNode was lost.
3. generateMapResponse checked OriginNode only in buildFromChange(),
but PolicyChange uses RequiresRuntimePeerComputation which
bypasses that code path entirely and calls policyChangeResponse()
instead.
The fix addresses all three:
- state.go: Set OriginNode on the returned change
- change.go: Preserve OriginNode (and TargetNode) during merge
- batcher.go: Pass isSelfUpdate to policyChangeResponse so the
origin node gets both self info AND packet filters
- mapper.go: Add includeSelf parameter to policyChangeResponse
Fixes#2978
Update 8 tests that involve admin tag assignment via SetNodeTags()
to verify both server-side state and node self view updates:
- TestTagsAuthKeyWithTagAdminOverrideReauthPreserves
- TestTagsAuthKeyWithTagCLICannotModifyAdminTags
- TestTagsAuthKeyWithoutTagCLINoOpAfterAdminWithReset
- TestTagsAuthKeyWithoutTagCLINoOpAfterAdminWithEmptyAdvertise
- TestTagsAuthKeyWithoutTagCLICannotReduceAdminMultiTag
- TestTagsUserLoginCLINoOpAfterAdminAssignment
- TestTagsUserLoginCLICannotRemoveAdminTags
- TestTagsAdminAPICanSetUnownedTag
Each test now validates that tag updates propagate to the node's
own self view using assertNodeSelfHasTagsWithCollect, addressing
the issue #2978 scenario where tag changes were observed to
propagate to peers but not to the node itself.
Updates #2978
Add TestTagsIssue2978ReproTagReplacement that specifically tests the
scenario from issue #2978:
- Register node with tag:foo via web auth with --advertise-tags
- Admin changes tag to tag:bar via SetNodeTags
- Verify client's self view updates (not just server-side)
The test performs multiple tag replacements with timing checks to
verify whether tag updates propagate to the node's self view after
the first call (fixed behavior) or only after a redundant second
call (bug behavior).
Add helper functions for test validation:
- assertNodeSelfHasTagsWithCollect: validates client's status.Self.Tags
- assertNetmapSelfHasTagsWithCollect: validates client's netmap.SelfNode.Tags
Updates #2978
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
update-flake-lock / lockfile (push) Has been cancelled
GitHub Actions Version Updater / build (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Add TestTagsUserLoginReauthWithEmptyTagsRemovesAllTags to validate that
nodes can be untagged via `tailscale up --advertise-tags= --force-reauth`.
The test verifies:
- Node starts with tags and is owned by tagged-devices
- After reauth with empty tags, all tags are removed
- Node ownership returns to the authenticating user
Updates #2979
When a node re-authenticates via OIDC/web auth with empty RequestTags
(from `tailscale up --advertise-tags= --force-reauth`), remove all tags
and return ownership to the authenticating user.
This allows nodes to transition from any tagged state (including nodes
originally registered with a tagged pre-auth key) back to user-owned.
Fixes#2979
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Deploy docs / deploy (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
Extends #2971 fix to also cover nodes that authenticate as users but
become tagged immediately via --advertise-tags. When RequestTags are
approved by policy, the node's expiry is now disabled, consistent with
nodes registered via tagged PreAuthKeys.
Nodes registered with tagged PreAuthKeys now have key expiry disabled,
matching Tailscale's behavior. User-owned nodes continue to use the
client-requested expiry.
On re-authentication, tagged nodes preserve their disabled expiry while
user-owned nodes can update their expiry from the client request.
Fixes#2971
Just the tags tag:router and tag:exit are owned by alice. Upon join,
those nodes will have their ownership transferred from alice to the
system user "tagged-devices".
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
Deploy docs / deploy (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
User.Proto() was returning u.Name directly, which is empty for OIDC
users who have their identifier in the Email field instead. This caused
"headscale nodes list" to show empty user names for OIDC-authenticated
nodes.
Only fall back to Username() when Name is empty, which provides a
display-friendly identifier (Email > ProviderIdentifier > ID). This
ensures OIDC users display their email while CLI users retain their
original Name.
Fixes#2972
Simplifies the API by exposing a single tags field instead of three
separate fields for forced, invalid, and valid tags. The distinction
between these was an internal implementation detail that should not
be exposed in the public API.
Marks fields 18-20 as reserved to prevent field number reuse.
Update documentation to reflect the new concurrent test execution
capabilities and add guidance on run ID isolation.
AGENTS.md:
- Add examples for running multiple tests concurrently
- Document run ID format and container naming conventions
- Update "Critical Notes" to explain isolation mechanisms
.claude/agents/headscale-integration-tester.md:
- Add "Concurrent Execution and Run ID Isolation" section
- Document forbidden and safe operations for cleanup
- Add "Agent Session Isolation Rules" for multi-agent environments
- Add 6th core responsibility about concurrent execution awareness
- Add ISOLATION PRINCIPLE to critical principles
- Update pre-test cleanup documentation
Remove the concurrent test prevention logic and update cleanup to use
run ID-based isolation, allowing multiple tests to run simultaneously.
Changes:
- cleanup: Add killTestContainersByRunID() to clean only containers
belonging to a specific run, add cleanupStaleTestContainers() to
remove only stopped/exited containers without affecting running tests
- docker: Remove RunningTestInfo, checkForRunningTests(), and related
error types, update cleanupAfterTest() to use run ID-based cleanup
- run: Remove Force flag and concurrent test prevention check
The test runner now:
- Allows multiple concurrent test runs on the same Docker daemon
- Cleans only stale containers before tests (not running ones)
- Cleans only containers with matching run ID after tests
- Prints run ID and monitoring info for operator visibility
Add run ID-based isolation to container naming and network setup to
enable multiple integration tests to run concurrently on the same
Docker daemon without conflicts.
Changes:
- hsic: Add run ID prefix to headscale container names and use dynamic
port allocation for metrics endpoint (port 0 lets kernel assign)
- tsic: Add run ID prefix to tailscale container names
- dsic: Add run ID prefix to DERP container names
- scenario: Use run ID-aware test suite container name for network setup
Container naming now follows: {type}-{runIDShort}-{identifier}-{hash}
Example: ts-mdjtzx-1-74-fgdyls, hs-mdjtzx-pingallbyip-abc123
The run ID is obtained from HEADSCALE_INTEGRATION_RUN_ID environment
variable via dockertestutil.GetIntegrationRunID().
This commit replaces the ChangeSet with a simpler bool based
change model that can be directly used in the map builder to
build the appropriate map response based on the change that
has occured. Previously, we fell back to sending full maps
for a lot of changes as that was consider "the safe" thing to
do to ensure no updates were missed.
This was slightly problematic as a node that already has a list
of peers will only do full replacement of the peers if the list
is non-empty, meaning that it was not possible to remove all
nodes (if for example policy changed).
Now we will keep track of last seen nodes, so we can send remove
ids, but also we are much smarter on how we send smaller, partial
maps when needed.
Fixes#2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit adds tests to validate that there are
issues with how we propagate tag changes in the system.
This replicates #2389
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
This commit changes so that node changes to the policy is
calculated if any of the nodes has changed in a way that might
affect the policy.
Previously we just checked if the number of nodes had changed,
which meant that if a node was added and removed, we would be
in a bad state.
Signed-off-by: Kristoffer Dalby <kristoffer@dalby.cc>
Close inactive issues / close-issues (push) Has been cancelled
Build / build-nix (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
update-flake-lock / lockfile (push) Has been cancelled
GitHub Actions Version Updater / build (push) Has been cancelled
This PR restructures the integration tests and prebuilds all common assets used in all tests:
Headscale and Tailscale HEAD image
hi binary that is used to run tests
go cache is warmed up for compilation of the test
This essentially means we spend 6-10 minutes building assets before any tests starts, when that is done, all tests can just sprint through.
It looks like we are saving 3-9 minutes per test, and since we are limited to running max 20 concurrent tests across the repo, that means we had a lot of double work.
There is currently 113 checks, so we have to do five runs of 20, and the saving should be quite noticeable! I think the "worst case" saving would be 20+min and "best case" probably towards an hour.
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Fixes#2927
In v0.27.0, the list-routes command with -i flag and -o json output
was returning all nodes instead of just the specified node.
The issue was that JSON output was happening before the identifier
filtering logic. This change moves the JSON output to after both
the identifier filter and route existence filter are applied,
ensuring the correct filtered results are returned.
This restores the v0.26.1 behavior where:
headscale nodes list-routes -i 12 -o json
correctly returns only node 12's route information.
Close inactive issues / close-issues (push) Has been cancelled
Build / build-nix (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
This PR investigates, adds tests and aims to correctly implement Tailscale's model for how Tags should be accepted, assigned and used to identify nodes in the Tailscale access and ownership model.
When evaluating in Headscale's policy, Tags are now only checked against a nodes "tags" list, which defines the source of truth for all tags for a given node. This simplifies the code for dealing with tags greatly, and should help us have less access bugs related to nodes belonging to tags or users.
A node can either be owned by a user, or a tag.
Next, to ensure the tags list on the node is correctly implemented, we first add tests for every registration scenario and combination of user, pre auth key and pre auth key with tags with the same registration expectation as observed by trying them all with the Tailscale control server. This should ensure that we implement the correct behaviour and that it does not change or break over time.
Lastly, the missing parts of the auth has been added, or changed in the cases where it was wrong. This has in large parts allowed us to delete and simplify a lot of code.
Now, tags can only be changed when a node authenticates or if set via the CLI/API. Tags can only be fully overwritten/replaced and any use of either auth or CLI will replace the current set if different.
A user owned device can be converted to a tagged device, but it cannot be changed back. A tagged device can never remove the last tag either, it has to have a minimum of one.
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
This PR changes tags to be something that exists on nodes in addition to users, to being its own thing. It is part of moving our tags support towards the correct tailscale compatible implementation.
There are probably rough edges in this PR, but the intention is to get it in, and then start fixing bugs from 0.28.0 milestone (long standing tags issue) to discover what works and what doesnt.
Updates #2417Closes#2619
Build / build-cross (GOARCH=amd64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=amd64 GOOS=linux) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=darwin) (push) Has been cancelled
Build / build-cross (GOARCH=arm64 GOOS=linux) (push) Has been cancelled
Check Generated Files / check-generated (push) Has been cancelled
Deploy docs / deploy (push) Has been cancelled
NixOS Module Tests / nix-module-check (push) Has been cancelled
Tests / test (push) Has been cancelled
Close inactive issues / close-issues (push) Has been cancelled
This improves security and explicitly fails on startup when a user picks
the wrong directory to store its data.
- Run in read-only mode
- Make /var/run/headscale a read-write tmpfs
- Mount the config volume read-only
- Use the /health endpoint to check if Headscale is up
Previously we tested migrations on schemas and dumps
of old databases.
The problems with testing migrations against the schemas
is that the migration table is empty, so we try to run
migrations that are already ran on that schema, which might
blow up.
This commit removes the schema approach and just leaves all
the dumps, which include the migration table.
Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>
Document the API endpoint and the built-in swagger docs at /swagger. The
remote control docs are just a use case for gRPC - move it in the API
docs and update links to it.
Move favicon.png, style.css, and headscale.svg to hscontrol/assets/
and create a single assets.go file with all embed directives.
Update hscontrol/handlers.go and hscontrol/templates/general.go to
use the centralized assets package.
Add test to validate HTML template output consistency across all
templates (OIDC callback, registration, Windows, Apple).
Verifies all templates produce valid HTML5 with:
- Proper DOCTYPE declaration
- HTML5 lang attribute
- UTF-8 charset
- Viewport meta tag
- Semantic HTML structure
Ensures template refactoring maintains standards compliance.
Refactor template system to use go:embed for external assets and
CSS classes for styling instead of inline styles:
- general.go: Add go:embed directives for style.css and headscale.svg,
replace inline styles with CSS classes (H1, H2, H3, P, etc.),
add mdTypesetBody wrapper with Material for MkDocs styling
- apple.go, oidc_callback.go, register_web.go, windows.go:
Update to use new CSS-based helper functions (H1, H2, P, etc.)
and mdTypesetBody for consistent layout
This separates content from presentation, making templates easier
to maintain and update. All styling is now centralized in style.css
with Material for MkDocs design system.
Add design system assets for HTML templates:
- headscale.svg: Logo with optimized viewBox for proper alignment
- style.css: Material for MkDocs CSS variables and typography
- design.go: Design system constants for consistent styling
The logo viewBox is adjusted to 32.92 0 1247.08 640 to eliminate
whitespace from the original export and ensure left alignment with
text content.
Replace html/template with type-safe elem-go templating for OIDC
callback page. Improves consistency with other templates and provides
compile-time safety. All UI elements and styling preserved.
description: Use this agent when you need to execute, analyze, or troubleshoot Headscale integration tests. This includes running specific test scenarios, investigating test failures, interpreting test artifacts, validating end-to-end functionality, or ensuring integration test quality before releases. Examples: <example>Context: User has made changes to the route management code and wants to validate the changes work correctly. user: 'I've updated the route advertisement logic in poll.go. Can you run the relevant integration tests to make sure everything still works?' assistant: 'I'll use the headscale-integration-tester agent to run the subnet routing integration tests and analyze the results.' <commentary>Since the user wants to validate route-related changes with integration tests, use the headscale-integration-tester agent to execute the appropriate tests and analyze results.</commentary></example> <example>Context: A CI pipeline integration test is failing and the user needs help understanding why. user: 'The TestSubnetRouterMultiNetwork test is failing in CI. The logs show some timing issues but I can't figure out what's wrong.' assistant: 'Let me use the headscale-integration-tester agent to analyze the test failure and examine the artifacts.' <commentary>Since this involves analyzing integration test failures and interpreting test artifacts, use the headscale-integration-tester agent to investigate the issue.</commentary></example>
color: green
---
You are a specialist Quality Assurance Engineer with deep expertise in Headscale's integration testing system. You understand the Docker-based test infrastructure, real Tailscale client interactions, and the complex timing considerations involved in end-to-end network testing.
## Integration Test System Overview
The Headscale integration test system uses Docker containers running real Tailscale clients against a Headscale server. Tests validate end-to-end functionality including routing, ACLs, node lifecycle, and network coordination. The system is built around the `hi` (Headscale Integration) test runner in `cmd/hi/`.
## Critical Test Execution Knowledge
### System Requirements and Setup
```bash
# ALWAYS run this first to verify system readiness
go run ./cmd/hi doctor
```
This command verifies:
- Docker installation and daemon status
- Go environment setup
- Required container images availability
- Sufficient disk space (critical - tests generate ~100MB logs per run)
- Network configuration
### Test Execution Patterns
**CRITICAL TIMEOUT REQUIREMENTS**:
- **NEVER use bash `timeout` command** - this can cause test failures and incomplete cleanup
- **ALWAYS use the built-in `--timeout` flag** with generous timeouts (minimum 15 minutes)
- **Increase timeout if tests ever time out** - infrastructure issues require longer timeouts
```bash
# Single test execution (recommended for development)
# ALWAYS use --timeout flag with minimum 15 minutes (900s)
go run ./cmd/hi run "TestSubnetRouterMultiNetwork" --timeout=900s
# Database-heavy tests require PostgreSQL backend and longer timeouts
go run ./cmd/hi run "TestExpireNode" --postgres --timeout=1800s
# Pattern matching for related tests - use longer timeout for multiple tests
go run ./cmd/hi run "TestSubnet*" --timeout=1800s
# Long-running individual tests need extended timeouts
go run ./cmd/hi run "TestNodeOnlineStatus" --timeout=2100s # Runs for 12+ minutes
# Full test suite (CI/validation only) - very long timeout required
3.**MapResponse JSON files**: Protocol-level debugging for network map generation issues
4.**Client status dumps** (`*_status.json`): Network state and peer connectivity information
5.**Database snapshots** (`.db` files): For data consistency and state persistence issues
## Common Failure Patterns and Root Cause Analysis
### CRITICAL MINDSET: Code Issues vs Infrastructure Issues
**⚠️ IMPORTANT**: When tests fail, it is ALMOST ALWAYS a code issue with Headscale, NOT infrastructure problems. Do not immediately blame disk space, Docker issues, or timing unless you have thoroughly investigated the actual error logs first.
### Systematic Debugging Process
1.**Read the actual error message**: Don't assume - read the stderr logs completely
2.**Check Headscale server logs first**: Most issues originate from server-side logic
3.**Verify client connectivity**: Only after ruling out server issues
4.**Check timing patterns**: Use proper `EventuallyWithT` patterns
5.**Infrastructure as last resort**: Only blame infrastructure after code analysis
### Real Failure Patterns
#### 1. Timing Issues (Common but fixable)
```go
// ❌ Wrong: Immediate assertions after async operations
},10*time.Second,100*time.Millisecond,"route should be advertised")
```
**Timeout Guidelines**:
- Route operations: 3-5 seconds
- Node state changes: 5-10 seconds
- Complex scenarios: 10-15 seconds
- Policy recalculation: 5-10 seconds
#### 2. NodeStore Synchronization Issues
Route advertisements must propagate through poll requests (`poll.go:420`). NodeStore updates happen at specific synchronization points after Hostinfo changes.
**Why focus on WHY**: Helps maintainers understand architectural decisions and security requirements.
## EventuallyWithT Pattern for External Calls
### Overview
EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
### External Calls That Must Be Wrapped
The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
- `headscale.ListNodes()` - Queries server state
- `client.Status()` - Gets client network status
- `client.Curl()` - Makes HTTP requests through the network
}, 10*time.Second, 100*time.Millisecond, "route should be approved")
```
## Your Core Responsibilities
1. **Test Execution Strategy**: Execute integration tests with appropriate configurations, understanding when to use `--postgres` and timing requirements for different test categories. Follow phase-based testing approach prioritizing route tests.
- **Why this priority**: Route tests are less infrastructure-sensitive and validate core security logic
2. **Systematic Test Analysis**: When tests fail, systematically examine artifacts starting with Headscale server logs, then client logs, then protocol data. Focus on CODE ISSUES first (99% of cases), not infrastructure. Use real-world failure patterns to guide investigation.
- **Why this approach**: Most failures are logic bugs, not environment issues - efficient debugging saves time
3. **Timing & Synchronization Expertise**: Understand asynchronous Headscale operations, particularly route advertisements, NodeStore synchronization at `poll.go:420`, and policy propagation. Fix timing with `EventuallyWithT` while preserving original test expectations.
- **Why preserve expectations**: Test assertions encode business requirements and security policies
- **Key Pattern**: Apply the EventuallyWithT pattern correctly for all external calls as documented above
4. **Root Cause Analysis**: Distinguish between actual code regressions (route approval logic, HA failover architecture), timing issues requiring `EventuallyWithT` patterns, and genuine infrastructure problems (DNS, Docker, container issues).
- **Why this distinction matters**: Different problem types require completely different solution approaches
- **EventuallyWithT Issues**: Often manifest as flaky tests or immediate assertion failures after async operations
5.**Security-Aware Quality Validation**: Ensure tests properly validate end-to-end functionality with realistic timing expectations and proper error handling. Never suggest security bypasses or test expectation changes. Add comprehensive godoc when you understand test business logic.
- **Why security focus**: Integration tests are the last line of defense against security regressions
- **EventuallyWithT Usage**: Proper use prevents race conditions without weakening security assertions
**CRITICAL PRINCIPLE**: Test expectations are sacred contracts that define correct system behavior. When tests fail, fix the code to match the test, never change the test to match broken code. Only timing and observability improvements are allowed - business logic expectations are immutable.
**EventuallyWithT PRINCIPLE**: Every external call to headscale server or tailscale client must be wrapped in EventuallyWithT. Follow the five key rules strictly: one external call per block, proper variable scoping, no nesting, use CollectT for assertions, and provide descriptive messages.
**Remember**: Test failures are usually code issues in Headscale that need to be fixed, not infrastructure problems to be ignored. Use the specific debugging workflows and failure patterns documented above to efficiently identify root causes. Infrastructure issues have very specific signatures - everything else is code-related.
Thank you for taking the time to report this issue.
To help us investigate and resolve this, we need more information. Please provide the following:
> [!TIP]
> Most issues turn out to be configuration errors rather than bugs. We encourage you to discuss your problem in our [Discord community](https://discord.gg/c84AZQhmpx) **before** opening an issue. The community can often help identify misconfigurations quickly, saving everyone time.
- **Attach long files** - Do not paste large logs or configurations inline. Use GitHub file attachments or GitHub Gists.
- **Use proper Markdown** - Format code blocks, logs, and configurations with appropriate syntax highlighting.
- **Structure your response** - Use the headings above to organize your information clearly.
## Redaction Rules
> [!CAUTION]
> **Replace, do not remove.** Removing information makes debugging impossible.
When redacting sensitive information:
- ✅ **Replace consistently** - If you change `alice@company.com` to `user1@example.com`, use `user1@example.com` everywhere (logs, config, policy, etc.)
- ❌ **Never remove information** - Gaps in data prevent us from correlating events across logs
- ❌ **Never redact IP addresses** - We need the actual IPs to trace network paths and identify issues
**If redaction rules are not followed, we will be unable to debug the issue and will have to close it.**
---
**Note:** This issue will be automatically closed in 3 days if no additional information is provided. Once you reply with the requested information, the `needs-more-info` label will be removed automatically.
If you need help gathering this information, please visit our [Discord community](https://discord.gg/c84AZQhmpx).
This issue tracker is used for **bug reports and feature requests** only. Your question appears to be a support or configuration question rather than a bug report.
For help with setup, configuration, or general questions, please visit our [Discord community](https://discord.gg/c84AZQhmpx) where the community and maintainers can assist you in real-time.
**Before posting in Discord, please check:**
- [Documentation](https://headscale.net/)
- [FAQ](https://headscale.net/stable/faq/)
- [Debugging and Troubleshooting Guide](https://headscale.net/stable/ref/debug/)
If after troubleshooting you determine this is actually a bug, please open a new issue with the required debug information from the troubleshooting guide.
body: 'Nix build failed with wrong gosum, please update "vendorSha256" (${{ steps.build.outputs.OLD_HASH }}) for the "headscale" package in flake.nix with the new SHA: ${{ steps.build.outputs.NEW_HASH }}'
body: 'Vendor hash in `flakehashes.json` is stale (was `${{ steps.vendorhash.outputs.expected_sri }}`, should be `${{ steps.vendorhash.outputs.actual_sri }}`). Run `go run ./cmd/vendorhash update` and commit the result.'
Please follow the steps outlined in the [upgrade guide](https://headscale.net/stable/setup/upgrade/) to update your existing Headscale installation.
**It's best to update from one stable version to the next** (e.g., 0.24.0 → 0.25.1 → 0.26.1) in case you are multiple releases behind. You should always pick the latest available patch release.
Be sure to check the changelog above for version-specific upgrade instructions and breaking changes.
### Backup Your Database
**Always backup your database before upgrading.** Here's how to backup a SQLite database:
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Overview
Headscale is an open-source implementation of the Tailscale control server written in Go. It provides self-hosted coordination for Tailscale networks (tailnets), managing node registration, IP allocation, policy enforcement, and DERP routing.
- **Database Tests**: Extensive migration and data integrity validation
- **Policy Tests**: ACL rule evaluation and edge cases
- **Performance Tests**: NodeStore and high-frequency operation validation
## Integration Testing System
### Overview
Headscale uses Docker-based integration tests with real Tailscale clients to validate end-to-end functionality. The integration test system is complex and requires specialized knowledge for effective execution and debugging.
### **MANDATORY: Use the headscale-integration-tester Agent**
**CRITICAL REQUIREMENT**: For ANY integration test execution, analysis, troubleshooting, or validation, you MUST use the `headscale-integration-tester` agent. This agent contains specialized knowledge about:
- Test execution strategies and timing requirements
- Infrastructure vs code issue distinction (99% vs 1% failure patterns)
- Security-critical debugging rules and forbidden practices
- Comprehensive artifact analysis workflows
- Real-world failure patterns from HA debugging experiences
### Quick Reference Commands
```bash
# Check system requirements (always run first)
go run ./cmd/hi doctor
# Run single test (recommended for development)
go run ./cmd/hi run "TestName"
# Use PostgreSQL for database-heavy tests
go run ./cmd/hi run "TestName" --postgres
# Pattern matching for related tests
go run ./cmd/hi run "TestPattern*"
```
**Critical Notes**:
- Only ONE test can run at a time (Docker port conflicts)
- Tests generate ~100MB of logs per run in `control_logs/`
- Clean environment before each test: `rm -rf control_logs/202507* && docker system prune -f`
### Test Artifacts Location
All test runs save comprehensive debugging artifacts to `control_logs/TIMESTAMP-ID/` including server logs, client logs, database dumps, MapResponse protocol data, and Prometheus metrics.
**For all integration test work, use the headscale-integration-tester agent - it contains the complete knowledge needed for effective testing and debugging.**
## NodeStore Implementation Details
**Key Insight from Recent Work**: The NodeStore is a critical performance optimization that caches node data in memory while ensuring consistency with the database. When working with route advertisements or node state changes:
1.**Timing Considerations**: Route advertisements need time to propagate from clients to server. Use `require.EventuallyWithT()` patterns in tests instead of immediate assertions.
2.**Synchronization Points**: NodeStore updates happen at specific points like `poll.go:420` after Hostinfo changes. Ensure these are maintained when modifying the polling logic.
3.**Peer Visibility**: The NodeStore's `peersFunc` determines which nodes are visible to each other. Policy-based filtering is separate from monitoring visibility - expired nodes should remain visible for debugging but marked as expired.
## Testing Guidelines
### Integration Test Patterns
#### **CRITICAL: EventuallyWithT Pattern for External Calls**
**All external calls in integration tests MUST be wrapped in EventuallyWithT blocks** to handle eventual consistency in distributed systems. External calls include:
-`client.Status()` - Getting Tailscale client status
-`client.Curl()` - Making HTTP requests through clients
- Some tests require significant time (e.g., `TestNodeOnlineStatus` runs for 12 minutes)
- Infrastructure issues like disk space can cause test failures unrelated to code changes
- Use `--postgres` flag when testing database-heavy scenarios
## Quality Assurance and Testing Requirements
### **MANDATORY: Always Use Specialized Testing Agents**
**CRITICAL REQUIREMENT**: For ANY task involving testing, quality assurance, review, or validation, you MUST use the appropriate specialized agent at the END of your task list. This ensures comprehensive quality validation and prevents regressions.
**Required Agents for Different Task Types**:
1.**Integration Testing**: Use `headscale-integration-tester` agent for:
- Running integration tests with `cmd/hi`
- Analyzing test failures and artifacts
- Troubleshooting Docker-based test infrastructure
- Validating end-to-end functionality changes
2.**Quality Control**: Use `quality-control-enforcer` agent for:
- Code review and validation
- Ensuring best practices compliance
- Preventing common pitfalls and anti-patterns
- Validating architectural decisions
**Agent Usage Pattern**: Always add the appropriate agent as the FINAL step in any task list to ensure quality validation occurs after all work is complete.
### Integration Test Debugging Reference
Test artifacts are preserved in `control_logs/TIMESTAMP-ID/` including:
- Headscale server logs (stderr/stdout)
- Tailscale client logs and status
- Database dumps and network captures
- MapResponse JSON files for protocol debugging
**For integration test issues, ALWAYS use the headscale-integration-tester agent - do not attempt manual debugging.**
## EventuallyWithT Pattern for Integration Tests
### Overview
EventuallyWithT is a testing pattern used to handle eventual consistency in distributed systems. In Headscale integration tests, many operations are asynchronous - clients advertise routes, the server processes them, updates propagate through the network. EventuallyWithT allows tests to wait for these operations to complete while making assertions.
### External Calls That Must Be Wrapped
The following operations are **external calls** that interact with the headscale server or tailscale clients and MUST be wrapped in EventuallyWithT:
-`headscale.ListNodes()` - Queries server state
-`client.Status()` - Gets client network status
-`client.Curl()` - Makes HTTP requests through the network
@@ -30,8 +30,8 @@ nodes in the Tailscale network. It assigns the IP addresses of the clients,
creates the boundaries between each user, enables sharing machines between users,
and exposes the advertised routes of your nodes.
A [Tailscale network (tailnet)](https://tailscale.com/kb/1136/tailnet/) is private
network which Tailscale assigns to a user in terms of private users or an
A [Tailscale network (tailnet)](https://tailscale.com/docs/concepts/tailnet) is
private network which Tailscale assigns to a user in terms of private users or an
organisation.
## Design goal
@@ -63,8 +63,18 @@ and container to run Headscale.**
Please have a look at the [`documentation`](https://headscale.net/stable/).
For NixOS users, a module is available in [`nix/`](./nix/).
## Builds from `main`
Development builds from the `main` branch are available as container images and
binaries. See the [development builds](https://headscale.net/stable/setup/install/main/)
documentation for details.
## Talks
- Fosdem 2026 (video): [Headscale & Tailscale: The complementary open source clone](https://fosdem.org/2026/schedule/event/KYQ3LL-headscale-the-complementary-open-source-clone/)
- presented by Kristoffer Dalby
- Fosdem 2023 (video): [Headscale: How we are using integration testing to reimplement Tailscale](https://fosdem.org/2023/schedule/event/goheadscale/)
- presented by Juan Font Alonso and Kristoffer Dalby
@@ -103,6 +113,8 @@ run `make lint` and `make fmt` before committing any code.
The **Proto** code is linted with [`buf`](https://docs.buf.build/lint/overview) and
formatted with [`clang-format`](https://clang.llvm.org/docs/ClangFormat.html).
The **docs** are formatted with [`mdformat`](https://mdformat.readthedocs.io).
The **rest** (Markdown, YAML, etc) is formatted with [`prettier`](https://prettier.io).
Check out the `.golangci.yaml` and `Makefile` to see the specific configuration.
@@ -147,6 +159,7 @@ make build
We recommend using Nix for dependency management to ensure you have all required tools. If you prefer to manage dependencies yourself, you can use Make directly:
// oauthTailnet is the single Headscale tailnet the v2 API addresses as "-".
constoauthTailnet="-"
funcinit(){
rootCmd.AddCommand(oauthClientsCmd)
oauthClientsCmd.AddCommand(listOAuthClientsCmd)
createOAuthClientCmd.Flags().
StringArrayP("scope","s",nil,"Scope the client's tokens are granted (repeatable): auth_keys, oauth_keys, devices:core, devices:routes, policy_file, feature_settings (each with a :read variant), or all/all:read")
createOAuthClientCmd.Flags().
StringArrayP("tag","t",nil,"Tag the client's tokens may assign to devices (repeatable), e.g. tag:k8s-operator")
if!confirmAction(cmd,"DO NOT run this command if an instance of headscale is running, are you sure headscale is not running?"){
returnnil,errAborted
}
returnbypassDatabase()
}
funcinit(){
rootCmd.AddCommand(policyCmd)
getPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
getPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing the API and does not require the server to be running")
policyCmd.AddCommand(getPolicy)
setPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
setPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing gRPC and does not require the server to be running")
setPolicy.Flags().BoolP(bypassFlag,"",false,"Uses the headscale config to directly access the database, bypassing the API and does not require the server to be running")
mustMarkRequired(setPolicy,"file")
policyCmd.AddCommand(setPolicy)
checkPolicy.Flags().StringP("file","f","","Path to a policy file in HuJSON format")
checkPolicy.Flags().BoolP(bypassFlag,"",false,"Open the database directly (no running server required) to resolve user references and to evaluate the policy's tests and sshTests blocks. Required when those checks are needed.")
mustMarkRequired(checkPolicy,"file")
policyCmd.AddCommand(checkPolicy)
}
@@ -47,61 +72,48 @@ var policyCmd = &cobra.Command{
@@ -24,9 +24,12 @@ We are more than happy to exchange emails, or to have dedicated calls before a P
## When/Why is Feature X going to be implemented?
We don't know. We might be working on it. If you're interested in contributing, please post a feature request about it.
We use [GitHub Milestones to plan for upcoming Headscale releases](https://github.com/juanfont/headscale/milestones).
Have a look at [our current plan](https://github.com/juanfont/headscale/milestones) to get an idea when a specific
feature is about to be implemented. The release plan is subject to change at any time.
Please be aware that there are a number of reasons why we might not accept specific contributions:
If you're interested in contributing, please post a feature request about it. Please be aware that there are a number of
reasons why we might not accept specific contributions:
- It is not possible to implement the feature in a way that makes sense in a self-hosted environment.
- Given that we are reverse-engineering Tailscale to satisfy our own curiosity, we might be interested in implementing the feature ourselves.
@@ -47,8 +50,8 @@ we have a "docker-issues" channel where you can ask for Docker-specific help to
## What is the recommended update path? Can I skip multiple versions while updating?
Please follow the steps outlined in the [upgrade guide](../setup/upgrade.md) to update your existing Headscale
installation. Its best to update from one stable version to the next (e.g. 0.24.0 → 0.25.1 → 0.26.1) in case
you are multiple releases behind. You should always pick the latest available patch release.
installation. Its required to update from one stable version to the next (e.g. 0.26.0 → 0.27.1 → 0.28.0) without
skipping minor versions in between. You should always pick the latest available patch release.
Be sure to check the [changelog](https://github.com/juanfont/headscale/blob/main/CHANGELOG.md) for version specific
upgrade instructions and breaking changes.
@@ -70,12 +73,12 @@ of Headscale:
1. An environment with 1000 servers
- they rarely "move" (change their endpoints)
- new nodes are added rarely
- they rarely "move" (change their endpoints)
- new nodes are added rarely
2. An environment with 80 laptops/phones (end user devices)
1. An environment with 80 laptops/phones (end user devices)
- nodes move often, e.g. switching from home to office
- nodes move often, e.g. switching from home to office
Headscale calculates a map of all nodes that need to talk to each other,
creating this "world map" requires a lot of CPU time. When an event that
@@ -131,7 +134,7 @@ help to the community.
Running headscale on a machine that is also in the tailnet can cause problems with subnet routers, traffic relay nodes, and MagicDNS. It might work, but it is not supported.
## Why do two nodes see each other in their status, even if an ACL allows traffic only in one direction?
## Why do two nodes see each other in their status, even if a policy rule allows traffic only in one direction?
A frequent use case is to allow traffic only from one node to another, but not the other way around. For example, the
workstation of an administrator should be able to connect to all nodes but the nodes themselves shouldn't be able to
@@ -139,31 +142,64 @@ connect back to the administrator's node. Why do all nodes see the administrator
`tailscale status`?
This is essentially how Tailscale works. If traffic is allowed to flow in one direction, then both nodes see each other
in their output of `tailscale status`. Traffic is still filtered according to the ACL, with the exception of`tailscale
ping` which is always allowed in either direction.
in their output of `tailscale status`. Traffic is still filtered according to the policy, with the exception of
`tailscale ping` which is always allowed in either direction.
See also <https://tailscale.com/kb/1087/device-visibility>.
See also <https://tailscale.com/docs/concepts/device-visibility>.
## My policy is stored in the database and Headscale refuses to start due to an invalid policy. How can I recover?
Headscale checks if the policy is valid during startup and refuses to start if it detects an error. The error message
indicates which part of the policy is invalid. Follow these steps to fix your policy:
- Dump the policy to a file: `headscale policy get --bypass-grpc-and-access-database-directly > policy.json`
- Dump the policy to a file: `headscale policy get --bypass-server-and-access-database-directly > policy.json`
- Edit and fixup `policy.json`. Use the command `headscale policy check --file policy.json` to validate the policy.
- Load the modified policy: `headscale policy set --bypass-grpc-and-access-database-directly --file policy.json`
- Load the modified policy: `headscale policy set --bypass-server-and-access-database-directly --file policy.json`
- Start Headscale as usual.
!!! warning "Full server configuration required"
The above commands to get/set the policy require a complete server configuration file including database settings. A
minimal config to [control Headscale via remote CLI](../ref/remote-cli.md) is not sufficient. You may use`headscale
-c /path/to/config.yaml` to specify the path to an alternative configuration file.
minimal config to [control Headscale via remote CLI](../ref/api.md#remote-control) is not sufficient. You may use
`headscale -c /path/to/config.yaml` to specify the path to an alternative configuration file.
## How can I migrate back to the recommended IP prefixes?
Tailscale only supports the IP prefixes `100.64.0.0/10` and `fd7a:115c:a1e0::/48` or smaller subnets thereof. The
following steps can be used to migrate from unsupported IP prefixes back to the supported and recommended ones.
!!! warning "Backup and test in a demo environment required"
The commands below update the IP addresses of all nodes in your tailnet and this might have a severe impact in your
specific environment. At a minimum:
- [Create a backup of your database](../setup/upgrade.md#backup)
- Test the commands below in a representive demo environment. This allows to catch subsequent connectivity errors
early and see how the tailnet behaves in your specific environment.
- Stop Headscale
- Restore the default prefixes in the [configuration file](../ref/configuration.md):
```yaml
prefixes:
v4: 100.64.0.0/10
v6: fd7a:115c:a1e0::/48
```
- Update the `nodes.ipv4` and `nodes.ipv6` columns in the database and assign each node a unique IPv4 and IPv6 address.
The following SQL statement assigns IP addresses based on the node ID:
Headscale implements the same policy ACLs as Tailscale.com, adapted to the self-hosted environment.
For instance, instead of referring to users when defining groups you must
use users (which are the equivalent to user/logins in Tailscale.com).
Please check https://tailscale.com/kb/1018/acls/ for further information.
When using ACL's the User borders are no longer applied. All machines
whichever the User have the ability to communicate with other hosts as
long as the ACL's permits this exchange.
## ACL Setup
To enable and configure ACLs in Headscale, you need to specify the path to your ACL policy file in the `policy.path` key in `config.yaml`.
Your ACL policy file must be formatted using [huJSON](https://github.com/tailscale/hujson).
Info on how these policies are written can be found
[here](https://tailscale.com/kb/1018/acls/).
Please reload or restart Headscale after updating the ACL file. Headscale may be reloaded either via its systemd service
(`sudo systemctl reload headscale`) or by sending a SIGHUP signal (`sudo kill -HUP $(pidof headscale)`) to the main
process. Headscale logs the result of ACL policy processing after each reload.
## Simple Examples
- [**Allow All**](https://tailscale.com/kb/1192/acl-samples#allow-all-default-acl): If you define an ACL file but completely omit the `"acls"` field from its content, Headscale will default to an "allow all" policy. This means all devices connected to your tailnet will be able to communicate freely with each other.
```json
{}
```
- [**Deny All**](https://tailscale.com/kb/1192/acl-samples#deny-all): To prevent all communication within your tailnet, you can include an empty array for the `"acls"` field in your policy file.
```json
{
"acls": []
}
```
## Complex Example
Let's build a more complex example use case for a small business (It may be the place where
ACL's are the most useful).
We have a small company with a boss, an admin, two developers and an intern.
The boss should have access to all servers but not to the user's hosts. Admin
should also have access to all hosts except that their permissions should be
limited to maintaining the hosts (for example purposes). The developers can do
anything they want on dev hosts but only watch on productions hosts. Intern
can only interact with the development servers.
There's an additional server that acts as a router, connecting the VPN users
to an internal network `10.20.0.0/16`. Developers must have access to those
internal resources.
Each user have at least a device connected to the network and we have some
// hosts should be defined using its IP addresses and a subnet mask.
// to define a single host, use a /32 mask. You cannot use DNS entries here,
// as they're prone to be hijacked by replacing their IP addresses.
// see https://github.com/tailscale/tailscale/issues/3800 for more information.
"hosts": {
"postgresql.internal": "10.20.0.2/32",
"webservers.internal": "10.20.10.1/29"
},
"acls": [
// boss have access to all servers
{
"action": "accept",
"src": ["group:boss"],
"dst": [
"tag:prod-databases:*",
"tag:prod-app-servers:*",
"tag:internal:*",
"tag:dev-databases:*",
"tag:dev-app-servers:*"
]
},
// admin have only access to administrative ports of the servers, in tcp/22
{
"action": "accept",
"src": ["group:admin"],
"proto": "tcp",
"dst": [
"tag:prod-databases:22",
"tag:prod-app-servers:22",
"tag:internal:22",
"tag:dev-databases:22",
"tag:dev-app-servers:22"
]
},
// we also allow admin to ping the servers
{
"action": "accept",
"src": ["group:admin"],
"proto": "icmp",
"dst": [
"tag:prod-databases:*",
"tag:prod-app-servers:*",
"tag:internal:*",
"tag:dev-databases:*",
"tag:dev-app-servers:*"
]
},
// developers have access to databases servers and application servers on all ports
// they can only view the applications servers in prod and have no access to databases servers in production
{
"action": "accept",
"src": ["group:dev"],
"dst": [
"tag:dev-databases:*",
"tag:dev-app-servers:*",
"tag:prod-app-servers:80,443"
]
},
// developers have access to the internal network through the router.
// the internal network is composed of HTTPS endpoints and Postgresql
// database servers.
{
"action": "accept",
"src": ["group:dev"],
"dst": ["10.20.0.0/16:443,5432"]
},
// servers should be able to talk to database in tcp/5432. Database should not be able to initiate connections to
// applications servers
{
"action": "accept",
"src": ["tag:dev-app-servers"],
"proto": "tcp",
"dst": ["tag:dev-databases:5432"]
},
{
"action": "accept",
"src": ["tag:prod-app-servers"],
"dst": ["tag:prod-databases:5432"]
},
// interns have access to dev-app-servers only in reading mode
{
"action": "accept",
"src": ["group:intern"],
"dst": ["tag:dev-app-servers:80,443"]
},
// Allow users to access their own devices using autogroup:self (see below for more details about performance impact)
{
"action": "accept",
"src": ["autogroup:member"],
"dst": ["autogroup:self:*"]
}
]
}
```
## Autogroups
Headscale supports several autogroups that automatically include users, destinations, or devices with specific properties. Autogroups provide a convenient way to write ACL rules without manually listing individual users or devices.
### `autogroup:internet`
Allows access to the internet through [exit nodes](routes.md#exit-node). Can only be used in ACL destinations.
```json
{
"action": "accept",
"src": ["group:users"],
"dst": ["autogroup:internet:*"]
}
```
### `autogroup:member`
Includes all users who are direct members of the tailnet. Does not include users from shared devices.
```json
{
"action": "accept",
"src": ["autogroup:member"],
"dst": ["tag:prod-app-servers:80,443"]
}
```
### `autogroup:tagged`
Includes all devices that have at least one tag.
```json
{
"action": "accept",
"src": ["autogroup:tagged"],
"dst": ["tag:monitoring:9090"]
}
```
### `autogroup:self`
**(EXPERIMENTAL)**
!!! warning "The current implementation of `autogroup:self` is inefficient"
Includes devices where the same user is authenticated on both the source and destination. Does not include tagged devices. Can only be used in ACL destinations.
```json
{
"action": "accept",
"src": ["autogroup:member"],
"dst": ["autogroup:self:*"]
}
```
*Using `autogroup:self` may cause performance degradation on the Headscale coordinator server in large deployments, as filter rules must be compiled per-node rather than globally and the current implementation is not very efficient.*
If you experience performance issues, consider using more specific ACL rules or limiting the use of `autogroup:self`.
```json
{
// The following rules allow internal users to communicate with their
// own nodes in case autogroup:self is causing performance issues.
Used in Tailscale SSH rules to allow access to any user except root. Can only be used in the `users` field of SSH rules.
```json
{
"action": "accept",
"src": ["autogroup:member"],
"dst": ["autogroup:self"],
"users": ["autogroup:nonroot"]
}
```
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
alaningtrump
Kristoffer Dalby
Florian Preinstorfer
Fredrik Ekre
Baptiste Fontaine
Kristoffer Dalby
kloba
Shourya Gautam
Lealem Amedie
Möhsün Babayev
MunMunMiao
SAY-5
primewildy
Akhilesh Arora
QEDeD
Jacky
Tanayk07
Juan Font
DM
Stefan Bethke
Brandon Sprague
chen
github-actions[bot]
Justin Angel
Rogan Lynch
Dusty Mabe
adinhodovic
Vitalij Dovhanyc
pwuersch
János Benjamin Antal
Acha
Tianon Gravi
Teej
Andrey Bobelev
