AddNode registers the connection before sending the initial map, so a
batched delta could land first and become the stream's first frame;
Tailscale clients then kill the poll with "initial MapResponse lacked
Node" and a retry loop under steady change traffic leaves the netmap
empty. Skip connections awaiting their initial map and retry the
change next tick — the in-flight map may predate it, so it cannot be
dropped. Retries are prepended to keep patch order.
Sentinel ErrNodeKeyInUse (err113); key the visible-peer set by tailcfg.NodeID
to drop an int64->uint64 cast (gosec G115); NewRequestWithContext (noctx); wsl.
filterVisiblePeerPatches and filterVisibleNodes now share one visiblePeerIDs
helper using the live MatchersForNode/ReduceNodes set, so paths cannot drift.
addNextDNSMetadata dereferenced node.Hostinfo().OS() without the .Valid() guard its siblings (RequestTags, TailNode) apply, so building the DNS config for a node with nil Hostinfo (a legacy NULL host_info row) panicked whenever a NextDNS resolver was configured. Guard the OS() dereference.
buildFromChange's PeersChanged path passed an unfiltered changed-node slice to WithUserProfiles, while the full-map path uses the BuildPeerMap-filtered ListPeers. A node thus received the identities (login name, display name, avatar) of users owning peers its ACL forbids accessing. Filter the UserProfiles peer set via the same ReduceNodes visibility check buildTailPeers applies.
buildFromChange added PeersChangedPatch (online/offline, endpoint, key-expiry) directly, skipping the policy.ReduceNodes visibility filter that buildTailPeers applies to full peers. A node thus received the existence, presence, and addresses of peers its ACL forbids accessing. Restrict patches to the recipient's visible peer set.
Regenerate capver for tailscale v1.98 — `MinSupportedCapabilityVersion`
slides 109 → 113 (v1.78 → v1.80). v1.80+ clients understand
`Node.HomeDERP`, which superseded `LegacyDERPString` upstream at capver
111. Drops the emit, plumbing, the trip-wire that caught this cleanup,
the golden test fixtures, and the integration check on
`peer.LegacyDERPString()` — the sibling `HomeDERP` check covers intent.
`v1.98` was added by hand to `capver_generated.go` because tailscale
skipped publishing v1.97/v1.98 to ghcr until late in the cycle; a clean
regeneration produces the same content.
Every Go-identifier reference in // and /* */ comments now uses
godoc's [Name] linking syntax so pkg.go.dev and `go doc` render
them as clickable cross-references. No behaviour change.
Pattern applied across the tree:
In-package [Foo], [Foo.Bar]
Cross-package [pkg.Foo], [pkg.Foo.Bar]
Stdlib [netip.Prefix], [errors.Is], [context.Context]
Tailscale [tailcfg.MapResponse], [tailcfg.Node.CapMap],
[tailcfg.NodeAttrSuggestExitNode]
Skip rules:
- File:line refs left as plain text
- HuJSON wire keys inside backtick raw strings untouched
- ACL/policy syntax tokens (tag:foo, autogroup:self, ...) not Go
symbols, left as plain text
- JSON/OIDC wire keys, gorm tags, RFC IPv6 placeholders, markdown
link tags, decorative dividers — all left as-is
hscontrol/debug.go — pre-size nodes []nodeStatus to len(debugInfo)
so the loop does not grow under append.
hscontrol/mapper/batcher_test.go — testing.TB parameter on
setupBatcherWithTestData renamed t → tb so thelper sees the
expected name.
hscontrol/db/text_serialiser.go — reflect.Ptr → reflect.Pointer
(deprecated alias).
policyChangeResponse already includes everything else; carry DNSConfig
too so the client's netmap DNS is anchored on every policy change
rather than relying on the previous snapshot.
Send the MagicDNS root domains as empty non-nil Resolver slices instead
of nil values. tailcfg.DNSConfig.Clone and net/dns.Config.Clone in
tailscale drop map entries whose value is nil (tailcfg_clone.go and
dns_clone.go both contain `if sv == nil { continue }`). On a major
LinkChange the client's wgengine handler clones lastDNSConfig and
re-applies it; with nil values the cloned config has Routes:{}, dns.Set
wipes Nameservers in /etc/resolv.conf, and curl-by-FQDN fails until
the next route-changing netmap, typically about six minutes later.
Empty slice survives Clone and carries the same "resolve locally"
semantics for Routes entries.
When a node carries the disable-ipv4 nodeAttr documented at
https://tailscale.com/docs/reference/troubleshooting/network-configuration/cgnat-conflicts,
SaaS stops sending the node's CGNAT IPv4 prefix in MapResponse. The
allocator keeps assigning IPv4 server-side; only the wire-shape
delivery is filtered. Subnet routes the node advertises -- including
IPv4 prefixes -- survive in AllowedIPs and PrimaryRoutes.
TailNode now drops Is4 prefixes from Addresses and from the node's
own /32 slot in AllowedIPs when selfPolicyCaps carries
disable-ipv4. Mapper.buildTailPeers passes each peer's policy
CapMap so the filter applies in viewer netmaps too; the CapMap
merge that follows is overwritten by PeerCapMap so only the address
filter survives on the peer path.
Two captures land in testdata/nodeattrs_results to anchor the
behaviour:
- nodeattrs-attr-c15-disable-ipv4 (on tag:client)
- nodeattrs-attr-c16-disable-ipv4-router (on tag:router, which
advertises 10.33.0.0/16, confirming subnet routes survive)
Tailscale stamps tailcfg.NodeAttrDefaultAutoUpdate on every node's
CapMap with a JSON bool reflecting the tailnet-wide auto-update
default. Headscale grows an auto_update.enabled config option and
emits the cap accordingly from TailNode -- the cap leaves the
unmodelledTailnetStateCaps strip list and is compared in full by the
nodeAttrs compat suite.
testNodeAttrsSuccess drives cfg.AutoUpdate.Enabled from
tf.Input.Tailnet.Settings.DevicesAutoUpdatesOn so each capture's
expected emission matches the SaaS state it was taken under. Two
captures cover both branches:
- nodeattrs-tailnet-devices-auto-updates-on -> [true]
- nodeattrs-tailnet-devices-auto-updates-off -> [false]
The Tailscale v2 TailnetSettings API does not expose the Send Files
toggle, so the compat suite cannot vary cfg.Taildrop.Enabled per
capture. TestTaildropDisabledWithholdsFileSharingCap covers the off
path directly in servertest.
Taildrive (drive:share and drive:access) is policy-driven per
Tailscale's documented behaviour
(https://tailscale.com/docs/features/taildrive). The previous
always-on baseline emission diverged from SaaS for every node not
targeted by a drive nodeAttr -- a real semantic divergence that the
compat suite caught once the test moved to comparing TailNode output
against the captured netmaps.
types.Node.TailNode no longer stamps the drive pair. Operators
wanting taildrive add a nodeAttrs entry:
"nodeAttrs": [
{ "target": ["*"], "attr": ["drive:share", "drive:access"] }
]
unmodelledTailnetStateCaps shrinks accordingly. The baseline-divergence
group is gone; every entry left in the list is genuinely unmodelled
(user-role caps, unimplemented features, tailnet metadata, internal
tuning).
servertest's TestNodeAttrsBaselineCapsAlwaysOn expects the smaller
baseline (admin + ssh + file-sharing). Integration TestGrantCapDrive
grants the drive caps explicitly via NodeAttrs to exercise the
policy-driven emission path.
types.NodeView.TailNode takes a selfPolicyCaps tailcfg.NodeCapMap
parameter and merges it into the baseline. The mapper's WithSelfNode
hands it the policy result via state.NodeCapMap; peer-path callers
pass nil because peer-side CapMap is set downstream via
policyv2.PeerCapMap.
The nodeAttrs compat test now diffs the full TailNode self-view
output against captured SaaS netmaps. Before this change the test
compared compileNodeAttrs alone -- the policy-only output -- and
needed a strip list to compensate for the missing baseline. With
TailNode on the diff path, baseline emission is exercised end-to-end
by every capture; a regression in TailNode breaks the suite.
unmodelledTailnetStateCaps drops cap/ssh and cap/file-sharing now
that both sides emit them identically. The file header is rewritten
to read as 'caps SaaS emits where headscale has no equivalent yet'
rather than the more confusing 'shape divergence' framing.
The Tailscale client surfaces 'use this peer as your exit node' when
the peer's CapMap carries the tailcfg.NodeAttrSuggestExitNode cap.
SaaS emits it only on peers whose advertised exit routes are
approved -- not every peer that just has the cap in its own
nodeAttrs slot.
policyv2.PeerCapMap encodes that emission rule: it walks the
peer's own self-CapMap (built from compileNodeAttrs) and surfaces
the gated entries (today just suggest-exit-node when the peer
IsExitNode). Mapper.buildTailPeers calls it for each peer instead
of merging the peer's full nodeAttrs CapMap onto its peer view.
allCapMaps snapshots the full per-node CapMap once per peer-list
build so pm.mu is acquired once rather than per peer.
Tailscale models the randomize-client-port toggle as a top-level
field on the ACL policy. Headscale now matches that shape: the
server-config randomize_client_port key is removed, the toggle
lives in the policy file as randomizeClientPort, and per-node
opt-in via nodeAttrs is also supported.
Operators upgrading from a config-set randomize_client_port hit
depr.fatalWithHint at startup, which prints the deprecation message
and points at the new policy field rather than silently dropping
the toggle. The default carries over (false) so operators who never
set it are unaffected. config-example.yaml ships a REMOVED stanza
showing the migration.
types/node.go drops the cfg.RandomizeClientPort read from
TailNode -- the cap is now policy-driven through compileNodeAttrs
and the tail_test.go expectations follow.
WithSelfNode and buildTailPeers merge each node's policy CapMap
into the tailcfg.Node.CapMap they emit. State.NodeCapMap and
State.NodeCapMaps wrap the policy manager: NodeCapMap returns a
defensive clone per call; NodeCapMaps snapshots the full per-node
map once for batched callers, amortising pm.mu acquisition across
a peer build.
generateDNSConfig grew a per-node CapMap argument so it can apply
nodeAttr-driven DNS overlays. The nextdns DoH rewrite hardens against
policy-controlled inputs:
- nextDNSDoHHost anchors the prefix match instead of substring,
so a hostile resolver URL cannot smuggle a nextdns hostname in
a path or query.
- nextDNSProfileFromCapMap accepts only profile names matching
[A-Za-z0-9._-]{1,64} and picks the lexicographically first when
multiple are granted -- deterministic, no shell metacharacters
or URL fragments through.
- addNextDNSMetadata composes the rewritten URL via url.Parse +
url.Values rather than fmt.Sprintf, so existing query strings
on the resolver URL survive and metadata cannot inject a new
component.
WithTaildropEnabled in servertest controls cfg.Taildrop.Enabled per
test so cap/file-sharing emission can be toggled in tests that need
to verify the off path.
Replace routes.PrimaryRoutes reads with NodeStore. Connect bumps
SessionEpoch; Disconnect re-checks it inside UpdateNode so the
check and mutation are atomic against a concurrent Connect on
the same node.
The connect_race regression test is carried in its final
SessionEpoch form.
Updates #3203
Tests were dumping megabytes of zerolog output on failure; silence
at init and let individual tests opt in via SetGlobalLevel when they need
log-driven assertions.
Updates #3157
Implement tailcfg.PingRequest support so the control server can verify
whether a connected node is still reachable. This is the foundation for
faster offline detection (currently ~16min due to Go HTTP/2 TCP retransmit
behavior) and future C2N communication.
The server sends a PingRequest via MapResponse with a unique callback
URL. The Tailscale client responds with a HEAD request to that URL,
proving connectivity. Round-trip latency is measured.
Wire PingRequest through the Change → Batcher → MapResponse pipeline,
add a ping tracker on State for correlating requests with responses,
add ResolveNode for looking up nodes by ID/IP/hostname, and expose a
/debug/ping page (elem-go form UI) and /machine/ping-response endpoint.
Updates #2902
Updates #2129
Replace zcache with golang-lru/v2/expirable for both the state auth
cache and the OIDC state cache. Add tuning.register_cache_max_entries
(default 1024) to cap the number of pending registration entries.
Introduce types.RegistrationData to replace caching a full *Node;
only the fields the registration callback path reads are retained.
Remove the dead HSDatabase.regCache field. Drop zgo.at/zcache/v2
from go.mod.
When send() is called on a node with zero active connections
(disconnected but kept for rapid reconnection), it returns nil
(success). handleNodeChange then calls updateSentPeers, recording
peers as delivered when no client received the data.
This corrupts lastSentPeers: future computePeerDiff calculations
produce wrong results because they compare against phantom state.
After reconnection, the node's initial map resets lastSentPeers,
but any changes processed during the disconnect window leave
stale entries that cause asymmetric peer visibility.
Return errNoActiveConnections from send() when there are no
connections. handleNodeChange treats this as a no-op (the change
was generated but not deliverable) and skips updateSentPeers,
keeping lastSentPeers consistent with what clients actually
received.
When a FullUpdate produces zero visible peers (e.g., a restrictive
policy isolates a node), the MapResponse has Peers: [] (empty
non-nil). The Tailscale client only processes Peers as a full
replacement when len(Peers) > 0 (controlclient/map.go:462), so an
empty list is silently ignored and stale peers persist.
This triggers when a FullUpdate() replaces a pending PolicyChange()
in the batcher. The PolicyChange would have used computePeerDiff to
send explicit PeersRemoved, but the FullUpdate goes through
buildFromChange which sets Peers: [] that the client ignores.
When a full update produces zero peers, compute the peer diff
against lastSentPeers and add explicit PeersRemoved entries so the
client correctly clears its stale peer state.
Add NodeAttrsTaildriveShare and NodeAttrsTaildriveAccess to the node capability map, enabling Taildrive file sharing when granted via policy. Add integration test verifying the full cap/drive grant lifecycle.
Updates #2180
Via grants steer routes to specific nodes per viewer. Until now,
all clients saw the same routes for each peer because route
assembly was viewer-independent. This implements per-viewer route
visibility so that via-designated peers serve routes only to
matching viewers, while non-designated peers have those routes
withdrawn.
Add ViaRouteResult type (Include/Exclude prefix lists) and
ViaRoutesForPeer to the PolicyManager interface. The v2
implementation iterates via grants, resolves sources against the
viewer, matches destinations against the peer's advertised routes
(both subnet and exit), and categorizes prefixes by whether the
peer has the via tag.
Add RoutesForPeer to State which composes global primary election,
via Include/Exclude filtering, exit routes, and ACL reduction.
When no via grants exist, it falls back to existing behavior.
Update the mapper to call RoutesForPeer per-peer instead of using
a single route function for all peers. The route function now
returns all routes (subnet + exit), and TailNode filters exit
routes out of the PrimaryRoutes field for HA tracking.
Updates #2180
When a node disconnects, serveLongPoll defers a cleanup that starts a
grace period goroutine. This goroutine polls batcher.IsConnected() and,
if the node has not reconnected within ~10 seconds, calls
state.Disconnect() to mark it offline. A TOCTOU race exists: the node
can reconnect (calling Connect()) between the IsConnected check and
the Disconnect() call, causing the stale Disconnect() to overwrite
the new session's online status.
Fix with a monotonic per-node generation counter:
- State.Connect() increments the counter and returns the current
generation alongside the change list.
- State.Disconnect() accepts the generation from the caller and
rejects the call if a newer generation exists, making stale
disconnects from old sessions a no-op.
- serveLongPoll captures the generation at Connect() time and passes
it to Disconnect() in the deferred cleanup.
- RemoveNode's return value is now checked: if another session already
owns the batcher slot (reconnect happened), the old session skips
the grace period entirely.
Update batcher_test.go to track per-node connect generations and
pass them through to Disconnect(), matching production behavior.
Fixes the following test failures:
- server_state_online_after_reconnect_within_grace
- update_history_no_false_offline
- nodestore_correct_after_rapid_reconnect
- rapid_reconnect_peer_never_sees_offline
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:
1. MapResponses delivered out of order — a later change could finish
generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
updateSentPeers does Clear() + Store() which is not atomic relative
to a concurrent Range() in computePeerDiff.
Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.
Fixes#3140
The Batcher's connected field (*xsync.Map[types.NodeID, *time.Time])
encoded three states via pointer semantics:
- nil value: node is connected
- non-nil time: node disconnected at that timestamp
- key missing: node was never seen
This was error-prone (nil meaning 'connected' inverts Go idioms),
redundant with b.nodes + hasActiveConnections(), and required keeping
two parallel maps in sync. It also contained a bug in RemoveNode where
new(time.Now()) was used instead of &now, producing a zero time.
Replace the separate connected map with a disconnectedAt field on
multiChannelNodeConn (atomic.Pointer[time.Time]), tracked directly
on the object that already manages the node's connections.
Changes:
- Add disconnectedAt field and helpers (markConnected, markDisconnected,
isConnected, offlineDuration) to multiChannelNodeConn
- Remove the connected field from Batcher
- Simplify IsConnected from two map lookups to one
- Simplify ConnectedMap and Debug from two-map iteration to one
- Rewrite cleanupOfflineNodes to scan b.nodes directly
- Remove the markDisconnectedIfNoConns helper
- Update all tests and benchmarks
Fixes#3141
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:
1. MapResponses delivered out of order — a later change could finish
generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
updateSentPeers does Clear() + Store() which is not atomic relative
to a concurrent Range() in computePeerDiff.
Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.
Fixes#3140
Remove XTestBatcherChannelClosingRace (~95 lines) and
XTestBatcherScalability (~515 lines). These were disabled by
prefixing with X (making them invisible to go test) and served
as dead code. The functionality they covered is exercised by the
active test suite.
Updates #2545
L8: Rename SCREAMING_SNAKE_CASE test constants to idiomatic Go
camelCase. Remove highLoad* and extremeLoad* constants that were
only referenced by disabled (X-prefixed) tests.
L10: Fix misleading assert message that said "1337" while checking
for region ID 999.
L12: Remove emoji from test log output to avoid encoding issues
in CI environments.
Updates #2545
L1: Replace crypto/rand with an atomic counter for generating
connection IDs. These identifiers are process-local and do not need
cryptographic randomness; a monotonic counter is cheaper and
produces shorter, sortable IDs.
L5: Use getActiveConnectionCount() in Debug() instead of directly
locking the mutex and reading the connections slice. This avoids
bypassing the accessor that already exists for this purpose.
L6: Extract the hardcoded 15*time.Minute cleanup threshold into
the named constant offlineNodeCleanupThreshold.
L7: Inline the trivial addWork wrapper; AddWork now calls addToBatch
directly.
Updates #2545
Move connectionEntry, multiChannelNodeConn, generateConnectionID, and
all their methods from batcher.go into a dedicated file. This reduces
batcher.go from ~1170 lines to ~800 and separates per-node connection
management from batcher orchestration.
Pure move — no logic changes.
Updates #2545
- TestBatcher_CloseBeforeStart_DoesNotHang: verifies Close() before
Start() returns promptly now that done is initialized in NewBatcher.
- TestBatcher_QueueWorkAfterClose_DoesNotHang: verifies queueWork
returns via the done channel after Close(), even without Start().
- TestIsConnected_FalseAfterAddNodeFailure: verifies IsConnected
returns false after AddNode fails and removes the last connection.
- TestRemoveConnectionAtIndex_NilsTrailingSlot: verifies the backing
array slot is nil-ed after removal to avoid retaining pointers.
Updates #2545
M7: Nil out trailing *connectionEntry pointers in the backing array
after slice removal in removeConnectionAtIndexLocked and send().
Without this, the GC cannot collect removed entries until the slice
is reallocated.
M1: Initialize the done channel in NewBatcher instead of Start().
Previously, calling Close() or queueWork before Start() would select
on a nil channel, blocking forever. Moving the make() to the
constructor ensures the channel is always usable.
M2: Move b.connected.Delete and b.totalNodes decrement inside the
Compute callback in cleanupOfflineNodes. Previously these ran after
the Compute returned, allowing a concurrent AddNode to reconnect
between the delete and the bookkeeping update, which would wipe the
fresh connected state.
M3: Call markDisconnectedIfNoConns on AddNode error paths. Previously,
when initial map generation or send timed out, the connection was
removed but b.connected retained its old nil (= connected) value,
making IsConnected return true for a node with zero connections.
Updates #2545
Add four unit tests guarding fixes introduced in recent commits:
- TestConnectionEntry_SendFastPath_TimerStopped: verifies the
time.NewTimer fix (H1) does not leak goroutines after many
fast-path sends on a buffered channel.
- TestBatcher_CloseWaitsForWorkers: verifies Close() blocks until all
worker goroutines exit (H3), preventing sends on torn-down channels.
- TestBatcher_CloseThenStartIsNoop: verifies the one-shot lifecycle
contract; Start() after Close() must not spawn new goroutines.
- TestBatcher_CloseStopsTicker: verifies Close() stops the internal
ticker to prevent resource leaks.
Updates #2545
Remove Caller(), channel pointer formatting (fmt.Sprintf("%p",...)),
and mutex timing from send(), addConnection(), and
removeConnectionByChannel(). Move per-broadcast summary and
no-connection logs from Debug to Trace. Remove per-connection
"attempting"/"succeeded" logs entirely; keep Warn for failures.
These methods run on every MapResponse delivery, so the savings
compound quickly under load.
Updates #2545
Kristoffer Dalby
Kristoffer Dalby
Lealem Amedie