ListPeers with explicit IDs filtered every node, not every peer, so a change
batch naming the recipient returned it as its own peer. db.ListPeers keeps
this out with `id <> nodeID`; the NodeStore rewrite dropped it.
(cherry picked from commit d9aebf472d)
With node.ephemeral.inactivity_timeout set, ephemeral nodes are
usually deleted after they go offline, but under reconnect churn some
departed nodes stayed in the node list as disconnected indefinitely
until removed manually or until Headscale restarted.
Ephemeral cleanup is timer-based via EphemeralGarbageCollector, not a
periodic LastSeen scan. serveLongPoll cancelled any pending GC timer
at the very start of a long-poll attempt and only rescheduled on a
clean disconnect after Connect. If a reconnect cancelled the timer and
then failed before Connect (for example an UpdateNodeFromMapRequest
error), the deferred cleanup saw connectGen == 0 and returned without
Schedule. The node remained offline with no deletion timer and no
reconciler to recover it.
Cancel the ephemeral GC timer only after a successful Connect, so a
failed reconnect leaves an already-armed inactivity timer intact.
Successful reconnects still cancel GC once the node is online, and a
later disconnect reschedules as before.
Add TestFailedReconnectDoesNotCancelEphemeralGC to lock in the
ordering, plus IsScheduled and DeleteNodeFromStoreForTest helpers for
the test.
Fixes#3382
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit cfd845cb53)
Re-authenticating a tagged node with --advertise-tags checked the tag-owned
node, not the authenticating user, so every tag was rejected.
Fixes#3374
(cherry picked from commit 6275e3a356)
Re-registering a tagged node with a different key discarded the new key's
tags and left a stale auth-key reference; retag on key change and persist it.
Fixes#3370
(cherry picked from commit fc16cc6905)
Tagged nodes never expire, but handleLogout stamped a past expiry on them,
leaving them stuck expired and unable to re-authenticate.
Fixes#3371
(cherry picked from commit 1ed5693fa4)
Scan a node-health check registry at boot and log each node whose name
can't form a valid FQDN, with the rename fix. Log-only, no mutation.
Updates #3346
(cherry picked from commit 4946d1c88d)
A valid label can still overflow 255 chars under a long base_domain; gate
RenameNode with the new types.ValidateGivenName.
Updates #3346
(cherry picked from commit c497612c99)
Collapse the single-pick machine-key lookups onto GetNodesByMachineKeyAllUsers
so callers see every node sharing a machine key and reject the ambiguous or
impossible cases (tagged and user-owned coexistence; a tagged key with several
user-owned candidates) instead of mutating an arbitrarily-picked node.
Updates #3312
Concurrent registrations of one machine key each saw no existing node and
created their own, duplicating nodes and IPs. Hold a per-machine lock across
the find-then-create section.
Updates #3312
Re-registration mutated the node store before the database write and did not
revert on failure, so a restart could drop the client's current node key.
Snapshot the node and restore it if the write fails.
Updates #3312
A reusable key on a converted node, or a tagged key on a user-owned node,
fell through to new-node creation, leaving two nodes per machine. Match by
machine key and update or convert in place.
Updates #3312
The pre-auth-key path wrote the client node key without the collision check
the auth path applies, so a re-registration could claim another node's key
and poison the node-key index. Reject keys bound to a different machine.
Updates #3312
The re-registration fast-path skipped validation for a matching node key
without checking expiry, so an expired node could re-auth with a spent key.
Gate it on the node not being expired.
Updates #3312
A node converted to tagged is re-indexed under no user, so re-registration
keyed on the key's owner missed it and rejected the spent one-shot key.
Match the existing tagged node by machine key.
Fixes#3312
A cancelled map request whose handler ran late could Connect after the
live session, steal the newest SessionEpoch, then exit without
disconnecting (stillConnected path); the live session's final
Disconnect was rejected as stale and the node stayed online forever
(relogin flake). Counted releases are order-independent, so overlapping
sessions cannot strand a node in either direction.
Sentinel ErrNodeKeyInUse (err113); key the visible-peer set by tailcfg.NodeID
to drop an int64->uint64 cast (gosec G115); NewRequestWithContext (noctx); wsl.
filterVisiblePeerPatches and filterVisibleNodes now share one visiblePeerIDs
helper using the live MatchersForNode/ReduceNodes set, so paths cannot drift.
applyAuthNodeUpdate rotated the node's NodeKey to the client-supplied value without the 1:1 NodeKey/MachineKey check createAndSaveNewNode (f8f08cf7) and getAndValidateNode enforce. A re-authenticating node could thus rotate its key to a victim's and poison the NodeStore NodeKey index, denying the victim service. Apply the same uniqueness check on the re-auth path.
createAndSaveNewNode trusted the client NodeKey without checking it was already bound to a different machine, so an authenticated party could register a node carrying a victim's public NodeKey, poison the NodeStore NodeKey index, and make the victim's MapRequest resolve to the wrong node (rejected by getAndValidateNode = DoS). Enforce the 1:1 NodeKey/MachineKey binding at registration, as poll time already does.
Connect and Disconnect appended change.PolicyChange() on every reconnect. PolicyChange sets RequiresRuntimePeerComputation, so the batcher rebuilt a full netmap (packet filters, SSH policy, peer serialization) for every connected node — O(N) per reconnect, O(N^2) on a restart storm. On a small VM this saturated CPU after the v0.28 -> v0.29 upgrade.
Emit it only when the node's online state changes what peers compute: subnet routers, relay targets, and via targets. An ordinary reconnect now sends just the lightweight online/offline peer patch. Relay and via targets still recompute, so peers drop a stale PeerRelay allocation when a relay goes offline.
Fixes#3293
Splitting these functions does not buy clarity — each has been
extracted before and put back. Pin the //nolint:gocyclo on each
with the reason their shape resists clean decomposition.
policy/v2/policy.go ViaRoutesForPeer — three-pass
via-grant resolution
policy/v2/filter.go compileSSHPolicy — per-rule
branches with
intertwined
autogroup:self
handling
(annotated in the
earlier nil-error
commit)
state/state.go HandleNodeFromPreAuthKey — security-
sensitive
sequential
validation order
servertest/routes_test TestRoutes — table-driven
test driver with
many independent
subtests
Also: //nolint:recvcheck on policy/v2.SSHUser — UnmarshalJSON
requires a pointer receiver; the other methods on this string
newtype use value receivers by convention.
requirePrimaryStable in TestHASubnetRouterFailoverDockerDisconnect
Phase 5a (simultaneous cable-pull of both routers) intermittently
caught the primary flipping to the offline r1. Both probe goroutines
mark their target unhealthy back-to-back; SetNodeUnhealthy publishes a
fresh NodeStore snapshot each call, so the intermediate snapshot — r1
unhealthy, r2 still healthy — runs the election with one healthy
candidate left and picks it. The next snapshot then enters the
all-unhealthy preserve-prev path, which preserves the wrong choice.
Collect probe results from the cycle and apply them through a new
NodeStore.UpdateNodes batched op so the election only runs once, with
the cycle's final health state. PolicyChange dispatch moves outside
the wg.Go goroutines and fires once if the primary assignment
actually changed.
WithSelfNode and buildTailPeers merge each node's policy CapMap
into the tailcfg.Node.CapMap they emit. State.NodeCapMap and
State.NodeCapMaps wrap the policy manager: NodeCapMap returns a
defensive clone per call; NodeCapMaps snapshots the full per-node
map once for batched callers, amortising pm.mu acquisition across
a peer build.
generateDNSConfig grew a per-node CapMap argument so it can apply
nodeAttr-driven DNS overlays. The nextdns DoH rewrite hardens against
policy-controlled inputs:
- nextDNSDoHHost anchors the prefix match instead of substring,
so a hostile resolver URL cannot smuggle a nextdns hostname in
a path or query.
- nextDNSProfileFromCapMap accepts only profile names matching
[A-Za-z0-9._-]{1,64} and picks the lexicographically first when
multiple are granted -- deterministic, no shell metacharacters
or URL fragments through.
- addNextDNSMetadata composes the rewritten URL via url.Parse +
url.Values rather than fmt.Sprintf, so existing query strings
on the resolver URL survive and metadata cannot inject a new
component.
WithTaildropEnabled in servertest controls cfg.Taildrop.Enabled per
test so cap/file-sharing emission can be toggled in tests that need
to verify the off path.
Mirror the guard from HandleNodeFromPreAuthKey in HandleNodeFromAuthPath.
Both functions log the old user's name in the "different user" branch
when an existing NodeStore entry under the same machine key belongs to
another user. UserView.Name dereferences the backing User pointer
unconditionally, so when the cached node was loaded with a non-nil
UserID but a nil User (Preload join missed the row, or upstream code
left the snapshot in that shape), the log call panics with a nil-pointer
dereference at hscontrol/types/types_view.go:97.
The panic is caught by the http2 server's runHandler for the noise
control plane, so the process keeps running but every retry produces a
new panic — production has observed bursts of ~1.9k panics per hour
during a tailscaled reconnect loop. The gRPC/OIDC entry has no equivalent
recover and would surface the panic to the caller.
Guard both call sites with oldUser.Valid() and fall back to an empty
old-user name when the pointer is nil. The "Creating new node for
different user" log line still includes the existing node ID, hostname,
machine key, and new user, so operator visibility is preserved.
Add reproduction tests for both handlers seeding the orphan shape
directly into NodeStore via PutNodeInStoreForTest.
Co-Authored-By: Kristoffer Dalby <kristoffer@dalby.cc>
Restore the legacy auto-clear at write boundaries that drop HA
candidacy: Disconnect, SetApprovedRoutes(empty), and
UpdateNodeFromMapRequest shrinking advertised routes to empty.
Plus a defensive guard in SetNodeUnhealthy.
Updates #3203
Replace routes.PrimaryRoutes reads with NodeStore. Connect bumps
SessionEpoch; Disconnect re-checks it inside UpdateNode so the
check and mutation are atomic against a concurrent Connect on
the same node.
The connect_race regression test is carried in its final
SessionEpoch form.
Updates #3203
When a user owned node registers or re registers with a PreAuthKey and the
client sends zero client expiry while node.expiry is set to 0, the expiry
column ends up stored as 0001-01-01 00:00:00 instead of NULL. Two sites in
HandleNodeFromPreAuthKey build a non nil pointer to regReq.Expiry even when
the value is zero time, and the needsDefaultExpiry guard only replaces it
when s.cfg.Node.Expiry > 0, so the pointer to zero time survives to the
database.
Convert an unset regReq.Expiry to nil before handing it off so the
needsDefaultExpiry path is the only place that assigns a non nil pointer.
This is a narrower sibling of #3170 on the user owned PreAuthKey path. The
regression was introduced alongside the fix for #3111 in 6337a3db.
Ingest (registration and MapRequest updates) now calls
dnsname.SanitizeHostname directly and lets NodeStore auto-bump on
collision. Admin rename uses dnsname.ValidLabel + SetGivenName so
conflicts are surfaced to the caller instead of silently mutated.
Three duplicate invalidDNSRegex definitions, the old NormaliseHostname
and ValidateHostname helpers, EnsureHostname, InvalidString,
ApplyHostnameFromHostInfo, GivenNameHasBeenChanged, generateGivenName
and EnsureUniqueGivenName are removed along with their tests.
ValidateHostname's username half is retained as ValidateUsername for
users.go.
The SaaS-matching collision rule replaces the random "invalid-xxxxxx"
fallback and the 8-character hash suffix; the empty-input fallback is
the literal "node". TestUpdateHostnameFromClient now exercises the
rewrite end-to-end with awkward macOS/Windows names.
Fixes#3188Fixes#2926Fixes#2343Fixes#2762Fixes#2449
Updates #2177
Updates #2121
Updates #363
Resolve by GivenName (unique per tailnet) before Hostname (client-
reported, may collide); within each pass, pick the lowest NodeID so
results are deterministic across NodeStore snapshot iterations.
Updates #3157
Blocked callers waiting on a pingTracker response channel would
hang forever if the server Close()d mid-probe. Drain the pending map on
Close so those goroutines unblock and exit cleanly.
Updates #3157
End-to-end exercise of via-grant compilation against SaaS captures:
peer visibility, AllowedIPs, PrimaryRoutes, and per-rule src/dst
reachability from each viewer's perspective.
Updates #3157
Ping HA subnet routers each probe cycle and mark unresponsive nodes
unhealthy. Reconnecting a node clears its unhealthy state since the
fresh Noise session proves basic connectivity.
Updates #2129
Updates #2902
Implement tailcfg.PingRequest support so the control server can verify
whether a connected node is still reachable. This is the foundation for
faster offline detection (currently ~16min due to Go HTTP/2 TCP retransmit
behavior) and future C2N communication.
The server sends a PingRequest via MapResponse with a unique callback
URL. The Tailscale client responds with a HEAD request to that URL,
proving connectivity. Round-trip latency is measured.
Wire PingRequest through the Change → Batcher → MapResponse pipeline,
add a ping tracker on State for correlating requests with responses,
add ResolveNode for looking up nodes by ID/IP/hostname, and expose a
/debug/ping page (elem-go form UI) and /machine/ping-response endpoint.
Updates #2902
Updates #2129
Add a row-level check so concurrent registrations with the same
single-use key cannot both succeed. Skip the call on
re-registration where the key is already marked used (#2830).
SSHActionHandler now verifies that the Noise session's machine key
matches the dst node before proceeding. The (src, dst) pair is
captured at hold-and-delegate time via a new SSHCheckBinding on
AuthRequest so sshActionFollowUp can verify the follow-up URL
matches. The OIDC non-registration callback requires the
authenticated user to own the src node before approving.
Replace zcache with golang-lru/v2/expirable for both the state auth
cache and the OIDC state cache. Add tuning.register_cache_max_entries
(default 1024) to cap the number of pending registration entries.
Introduce types.RegistrationData to replace caching a full *Node;
only the fields the registration callback path reads are retained.
Remove the dead HSDatabase.regCache field. Drop zgo.at/zcache/v2
from go.mod.
routesChanged aliases newHI.RoutableIPs into a local variable then
sorts it in place, which mutates the caller's Hostinfo data. The
Hostinfo is subsequently stored on the node, so the mutation
propagates but the input contract is violated.
Clone the slice before sorting to avoid mutating the input.
When UpdateNodeFromMapRequest and SetNodeTags race on persistNodeToDB,
the first caller to run updatePolicyManagerNodes detects the tag change
and returns a PolicyChange. The second caller finds no change and falls
back to NodeAdded.
If UpdateNodeFromMapRequest wins the race, it checked
policyChange.IsFull() which is always false for PolicyChange (only sets
IncludePolicy and RequiresRuntimePeerComputation). This caused the
PolicyChange to be dropped, so affected clients never received
PeersRemoved and the stale peer remained in their NetMap indefinitely.
Fix: check !policyChange.IsEmpty() instead, which correctly detects
any non-trivial policy change including PolicyChange().
This fixes the root cause of TestACLTagPropagation/multiple-tags-partial-
removal flaking at ~20% on CI.
Updates #3125
GORM's struct-based Updates() silently skips nil pointer fields.
When SetNodeTags sets node.UserID = nil to transfer ownership to tags,
the in-memory NodeStore is correct but the database retains the old
user_id value. This causes tagged nodes to remain associated with the
original user in the database, preventing user deletion and risking
ON DELETE CASCADE destroying tagged nodes.
Add Select("*") before Omit() on all three node persistence paths
to force GORM to include all fields in the UPDATE statement, including
nil pointers. This is the same pattern already used in db/ip.go for
IPv4/IPv6 nil handling, and is documented GORM behavior:
db.Select("*").Omit("excluded").Updates(struct)
The three affected paths are:
- persistNodeToDB: used by SetNodeTags and MapRequest updates
- applyAuthNodeUpdate: used by re-authentication with --advertise-tags
- HandleNodeFromPreAuthKey: used by PAK re-registration
Fixes#3161
Connect and Disconnect previously only appended a PolicyChange when
the affected node was a subnet router (routeChange) or the database
persist returned a full change. For every other node the peers just
received a small PeerChangedPatch{Online: ...} and no filter rules
were recomputed. That was too narrow: a node going offline or coming
online can affect policy compilation in ways beyond subnet routes.
TestGrantCapRelay Phase 4 exposed this. When the cap/relay target node
went down with `tailscale down`, headscale only sent an Online=false
patch, peers never got a recomputed netmap, and their cached
PeerRelay allocation stayed populated until the 120s assertion
timeout. With a PolicyChange queued on Disconnect, peers immediately
receive a full netmap on relay loss and clear PeerRelay as expected;
the symmetric change on Connect lets Phase 5 re-publish the policy
when the relay comes back.
Drop the now-unused routeChange return from the Disconnect gate.
Updates #2180
Use the node.expiry config to apply a default expiry to non-tagged
nodes when the client does not request a specific expiry. This covers
all registration paths: new node creation, re-authentication, and
pre-auth key re-registration.
Tagged nodes remain exempt and never expire.
Fixes#1711
Kristoffer Dalby
Igor Serganov
Kristoffer Dalby
kloba
SAY-5
Akhilesh Arora
Florian Preinstorfer