electPrimaryRoutes' all-unhealthy fallback picked candidates[0] when
the previous primary was no longer a candidate. The Phase-5
simultaneous dual-disconnect path in TestHASubnetRouterFailoverDocker
Disconnect hits this asymmetrically: a batched probe cycle marks both
routers unhealthy with prev=r2 preserved, then the grace-period
Disconnect for r2 drops it from candidates. With prev gone and the
remaining r1 still carrying its Unhealthy bit, the fallback pointed
peers at the cable-pulled r1 — flapping primary to an unreachable
node and tripping requirePrimaryStable.
Leave the prefix unmapped when prev is gone and every candidate is
unhealthy. Peers see no advertiser instead of an unreachable one,
which is honest: the next probe cycle re-evaluates and picks
whichever node responds. The property-test model that mirrored the
old behaviour is updated to match.
requirePrimaryStable in TestHASubnetRouterFailoverDockerDisconnect
Phase 5a (simultaneous cable-pull of both routers) intermittently
caught the primary flipping to the offline r1. Both probe goroutines
mark their target unhealthy back-to-back; SetNodeUnhealthy publishes a
fresh NodeStore snapshot each call, so the intermediate snapshot — r1
unhealthy, r2 still healthy — runs the election with one healthy
candidate left and picks it. The next snapshot then enters the
all-unhealthy preserve-prev path, which preserves the wrong choice.
Collect probe results from the cycle and apply them through a new
NodeStore.UpdateNodes batched op so the election only runs once, with
the cycle's final health state. PolicyChange dispatch moves outside
the wg.Go goroutines and fires once if the primary assignment
actually changed.
The HA prober dispatches a PingRequest, waits ProbeTimeout (5s), and
marks the node unhealthy if no callback arrives. A node that bounced
its poll session between probe cycles satisfies two conditions that
conspire to fail TestHASubnetRouterFailover: a probe queued against
the previous session is silently dropped when the worker writes to
the closed connection (timeout always fires), and a probe sent
immediately after reconnect lands while wgengine is still rebuilding
magicsock state from the new netmap. Either path installs a spurious
unhealthy bit, which sends the preserved-primary anti-flap the wrong
way.
Record the session observed at dispatch time and drop the timeout
path if the node reconnected since. Require the session to survive
a full probe cycle before a timeout can drive a failover.
policyChangeResponse already includes everything else; carry DNSConfig
too so the client's netmap DNS is anchored on every policy change
rather than relying on the previous snapshot.
Send the MagicDNS root domains as empty non-nil Resolver slices instead
of nil values. tailcfg.DNSConfig.Clone and net/dns.Config.Clone in
tailscale drop map entries whose value is nil (tailcfg_clone.go and
dns_clone.go both contain `if sv == nil { continue }`). On a major
LinkChange the client's wgengine handler clones lastDNSConfig and
re-applies it; with nil values the cloned config has Routes:{}, dns.Set
wipes Nameservers in /etc/resolv.conf, and curl-by-FQDN fails until
the next route-changing netmap, typically about six minutes later.
Empty slice survives Clone and carries the same "resolve locally"
semantics for Routes entries.
tsic.Curl returned ("", nil) when curl exited 0 with a zero-byte body —
the usual signature of a mid-stream reset — so EventuallyWithT could
not retry. Return an error on empty body instead.
Replace the 56 inline curl + assert.Len(13) blocks with
assertCurlDockerHostname so the empty-body fix benefits every callsite
without further touch-ups.
Gate ACL waits on actual filter visibility: snapshotClientFilters +
waitForClientFilterChange ensure the new PacketFilter has reached the
client before assertions fire; SyncOption + WithPreBarrier feeds a
server-side policy-loaded check into WaitForTailscaleSyncPerUser.
Move advertise-routes mutation out of EventuallyWithT in route_test
(cmd/hi/README forbids retrying state-mutating calls). Pace the
TestNodeOnlineStatus outer loop with a Ticker, not a Sleep.
Categorised timeouts in integrationutil/timeouts.go remove the drift
opportunity between same-purpose budgets repeated across the suite.
The auth, cli, dns, derp, ssh, and tags tests are swept; acl, route,
and general tests follow in later commits alongside their other
ergonomic fixes.
DisconnectContainerFromNetwork and ReconnectContainerToNetwork
returned as soon as the docker API call completed, but libnetwork
bridge reprogramming continued for several seconds after. The HA
disconnect tests then raced and bounced between healthy and broken
bridges. Poll until the container's endpoint is gone (on
disconnect) or reconciled (on reconnect), and on the
"conflicts with existing route" surface clear the stale subnet
route from the netns and retry. Settle is now baked into the
primitive so every caller benefits.
Anonymous Hub pulls trip the 100/6h IP cap on shared CI runners, turning
into singleton FAIL reports whenever the runner egress IP crosses the
quota. Route every pull through Docker Hub credentials when present, and
retry transient errors with backoff. tsic and hi use the same helper so
both surfaces honour ~/.docker/config.json and the GHA secrets.
Docker 29 itself works; the breakage on the GHA runner image was the
overlayfs default. Setting storage-driver=overlay2 restores the
long-standing default and lets the suite run on the current Docker
without the apt downgrade dance.
Fixes#3094
chi middleware.RealIP was mounted unconditionally on both the
public router and the noise router, so any client could send
X-Real-IP or X-Forwarded-For and have the spoofed value land in
r.RemoteAddr and the access-log remote= field.
Add a top-level trusted_proxies config option (list of CIDRs) and
replace middleware.RealIP with a gated middleware that:
- honours True-Client-IP / X-Real-IP / X-Forwarded-For only when
r.RemoteAddr is inside one of the configured prefixes;
- strips those three headers from every request whose peer is
not trusted, so downstream handlers cannot read them.
X-Forwarded-For is parsed via realclientip-go's
RightmostTrustedRangeStrategy so a prepended value cannot win in a
proxy chain. trustedProxies() rejects 0.0.0.0/0 and ::/0 at config
load.
Empty trusted_proxies (the default) skips the mount entirely;
r.RemoteAddr is the directly-connecting TCP peer.
TestGrantViaSubnetFilterRules pins exact-equality dst. Add a sibling
for the broader-dst case so the regression sits at the server level
alongside the policy-engine unit test.
Updates #3267
slices.Contains required exact equality between grant dst and the
advertised subnet route. Any non-identical pair was rejected, so a
via grant with broader (or narrower) dst emitted no filter rule and
added no route to the viewer's AllowedIPs. Tailscale SaaS uses
containment in either direction.
Switch to slices.ContainsFunc(routes, dst.Overlaps) for filter rule
emission (keep dst literal in DstPorts), and append overlapping
advertised routes to ViaRoutesForPeer.Include / Exclude. Rewrite the
multi-router HA election and regular-grant overlap detection to key
off the matched routes rather than the dst. Resolve *Host aliases to
*Prefix once in compileOneViaGrant and at the top of ViaRoutesForPeer
so the switch arms reach them.
Fixes#3267
Both helpers existed to write the literal "[]" when clearing a slice
column — a workaround for GORM's struct-Updates skipping nil slices.
The State path goes exclusively through persistNodeToDB, which is now
correct end-to-end thanks to the named IsZero slice types, so the
helpers are dead in production. The remaining callers were tests.
TestSetTags is dropped — TestSetTags_* in hscontrol/grpcv1_test.go
already covers the State path that production uses. TestAutoApproveRoutes
now writes routes via DB.Save on the loaded node, which is the path
gRPC SetApprovedRoutes drives in production.
Updates #3110
Drives the persist path for ApprovedRoutes, Tags and Endpoints —
seed a non-empty value, clear to nil, read the column back from disk,
then close the State and reopen one against the same sqlite file to
simulate a server restart. Pins the contract the named IsZero slice
types enforce so future changes to the persist path cannot silently
drop a cleared slice column.
Updates #3110
Endpoints, Tags and ApprovedRoutes serialize as JSON on Node. GORM's
struct Updates path skips fields it considers zero, and reflect treats
a nil slice as zero — clearing any of these columns via the State
persist path would leave the previous value in the database.
Introduce Strings, Prefixes and AddrPorts as named slice types whose
IsZero() always reports false, so GORM keeps the column in the UPDATE
regardless of the slice being nil or empty. JSON marshalling is
unchanged: nil serializes to null, empty to []. List() returns the
underlying unnamed slice for callers (mainly testify assertions over
reflect.DeepEqual) that distinguish the named type from its base.
Regenerated types_clone.go and types_view.go follow the field-type
swap. Test assertions across hscontrol/{db,state,servertest} updated
to call .List() where reflect.DeepEqual previously matched the raw
slice type.
Fixes#3110
Replays recorded policy responses for the sshTests block. 200 captures must evaluate; non-200 captures must reject with the recorded body as a substring of the headscale error. Divergences are listed in knownSSHTesterDivergences.
SetPolicy and policy check now compile per-dst SSH rules and replay each sshTests entry. The accept assertion treats check-action rules as reachable; the check assertion requires HoldAndDelegate on the matching rule. Boot reload warns and continues.
Adds SSHPolicyTest plus parse-time validation: empty src/dst, port/CIDR/autogroup-internet destinations, and tag references missing from tagOwners are rejected. Engine evaluation comes in a follow-up.
The guard added for #2862 in handleRegister checked
node.Expiry().Valid() before preserving node state on
Auth=nil + Expiry=zero registration requests. Valid() returns false
when node.Expiry is nil, the default for tagged nodes and for untagged
nodes registered against a preauth key with no default node.expiry
configured. Both fell through to handleLogout, which wrote
&time.Time{} (0001-01-01T00:00:00Z) over the original nil — the
user-visible 0001-01-01 expiry that `headscale nodes list` reports
after restart.
IsExpired() already returns false for both nil and zero-time, so the
Valid() check was redundant. Drop it so all nil-expiry nodes are
covered by the same preservation path.
Fixes#3170Fixes#3262
When a node carries the disable-ipv4 nodeAttr documented at
https://tailscale.com/docs/reference/troubleshooting/network-configuration/cgnat-conflicts,
SaaS stops sending the node's CGNAT IPv4 prefix in MapResponse. The
allocator keeps assigning IPv4 server-side; only the wire-shape
delivery is filtered. Subnet routes the node advertises -- including
IPv4 prefixes -- survive in AllowedIPs and PrimaryRoutes.
TailNode now drops Is4 prefixes from Addresses and from the node's
own /32 slot in AllowedIPs when selfPolicyCaps carries
disable-ipv4. Mapper.buildTailPeers passes each peer's policy
CapMap so the filter applies in viewer netmaps too; the CapMap
merge that follows is overwritten by PeerCapMap so only the address
filter survives on the peer path.
Two captures land in testdata/nodeattrs_results to anchor the
behaviour:
- nodeattrs-attr-c15-disable-ipv4 (on tag:client)
- nodeattrs-attr-c16-disable-ipv4-router (on tag:router, which
advertises 10.33.0.0/16, confirming subnet routes survive)
Tailscale stamps tailcfg.NodeAttrDefaultAutoUpdate on every node's
CapMap with a JSON bool reflecting the tailnet-wide auto-update
default. Headscale grows an auto_update.enabled config option and
emits the cap accordingly from TailNode -- the cap leaves the
unmodelledTailnetStateCaps strip list and is compared in full by the
nodeAttrs compat suite.
testNodeAttrsSuccess drives cfg.AutoUpdate.Enabled from
tf.Input.Tailnet.Settings.DevicesAutoUpdatesOn so each capture's
expected emission matches the SaaS state it was taken under. Two
captures cover both branches:
- nodeattrs-tailnet-devices-auto-updates-on -> [true]
- nodeattrs-tailnet-devices-auto-updates-off -> [false]
The Tailscale v2 TailnetSettings API does not expose the Send Files
toggle, so the compat suite cannot vary cfg.Taildrop.Enabled per
capture. TestTaildropDisabledWithholdsFileSharingCap covers the off
path directly in servertest.
Taildrive (drive:share and drive:access) is policy-driven per
Tailscale's documented behaviour
(https://tailscale.com/docs/features/taildrive). The previous
always-on baseline emission diverged from SaaS for every node not
targeted by a drive nodeAttr -- a real semantic divergence that the
compat suite caught once the test moved to comparing TailNode output
against the captured netmaps.
types.Node.TailNode no longer stamps the drive pair. Operators
wanting taildrive add a nodeAttrs entry:
"nodeAttrs": [
{ "target": ["*"], "attr": ["drive:share", "drive:access"] }
]
unmodelledTailnetStateCaps shrinks accordingly. The baseline-divergence
group is gone; every entry left in the list is genuinely unmodelled
(user-role caps, unimplemented features, tailnet metadata, internal
tuning).
servertest's TestNodeAttrsBaselineCapsAlwaysOn expects the smaller
baseline (admin + ssh + file-sharing). Integration TestGrantCapDrive
grants the drive caps explicitly via NodeAttrs to exercise the
policy-driven emission path.
types.NodeView.TailNode takes a selfPolicyCaps tailcfg.NodeCapMap
parameter and merges it into the baseline. The mapper's WithSelfNode
hands it the policy result via state.NodeCapMap; peer-path callers
pass nil because peer-side CapMap is set downstream via
policyv2.PeerCapMap.
The nodeAttrs compat test now diffs the full TailNode self-view
output against captured SaaS netmaps. Before this change the test
compared compileNodeAttrs alone -- the policy-only output -- and
needed a strip list to compensate for the missing baseline. With
TailNode on the diff path, baseline emission is exercised end-to-end
by every capture; a regression in TailNode breaks the suite.
unmodelledTailnetStateCaps drops cap/ssh and cap/file-sharing now
that both sides emit them identically. The file header is rewritten
to read as 'caps SaaS emits where headscale has no equivalent yet'
rather than the more confusing 'shape divergence' framing.
The Tailscale client surfaces 'use this peer as your exit node' when
the peer's CapMap carries the tailcfg.NodeAttrSuggestExitNode cap.
SaaS emits it only on peers whose advertised exit routes are
approved -- not every peer that just has the cap in its own
nodeAttrs slot.
policyv2.PeerCapMap encodes that emission rule: it walks the
peer's own self-CapMap (built from compileNodeAttrs) and surfaces
the gated entries (today just suggest-exit-node when the peer
IsExitNode). Mapper.buildTailPeers calls it for each peer instead
of merging the peer's full nodeAttrs CapMap onto its peer view.
allCapMaps snapshots the full per-node CapMap once per peer-list
build so pm.mu is acquired once rather than per peer.
Adds a data-driven test that loads testdata/nodeattrs_results/*.hujson
and diffs the captured SaaS-rendered netmaps against headscale's
compileNodeAttrs output. Each capture is one scenario the SaaS
control plane has rendered against the same policy headscale is asked
to compile -- the test enforces shape parity per node.
tailnet_state_caps.go enumerates the caps SaaS emits where headscale
has no equivalent concept yet (user-role admin/owner, tailnet lock,
services host, app connectors, internal magicsock and SSH tuning,
tailnet-state metadata) plus the always-on baseline (admin, ssh,
file-sharing) and the taildrive pair. stripUnmodelledTailnetStateCaps
filters both sides of cmp.Diff so the comparison focuses on the
policy-driven caps. PeerCapMap encodes which caps the Tailscale
client reads from the peer view (suggest-exit-node when exit routes
are approved, etc.) for use by the mapper.
testcapture switches to typed tailcfg/netmap/filtertype/apitype
values so schema drift between the capture tool and headscale
becomes a compile error rather than a silent test failure. Existing
compat suites (acl, grants, routes, ssh, issue_3212) move to the
typed shape.
The 53 SelfNode netmap captures and the 7 anonymizer-corrupted
suggest-charmander -> suggest-exit-node restorations in
routes_results / issue_3212 ride along.
Tailscale models the randomize-client-port toggle as a top-level
field on the ACL policy. Headscale now matches that shape: the
server-config randomize_client_port key is removed, the toggle
lives in the policy file as randomizeClientPort, and per-node
opt-in via nodeAttrs is also supported.
Operators upgrading from a config-set randomize_client_port hit
depr.fatalWithHint at startup, which prints the deprecation message
and points at the new policy field rather than silently dropping
the toggle. The default carries over (false) so operators who never
set it are unaffected. config-example.yaml ships a REMOVED stanza
showing the migration.
types/node.go drops the cfg.RandomizeClientPort read from
TailNode -- the cap is now policy-driven through compileNodeAttrs
and the tail_test.go expectations follow.
WithSelfNode and buildTailPeers merge each node's policy CapMap
into the tailcfg.Node.CapMap they emit. State.NodeCapMap and
State.NodeCapMaps wrap the policy manager: NodeCapMap returns a
defensive clone per call; NodeCapMaps snapshots the full per-node
map once for batched callers, amortising pm.mu acquisition across
a peer build.
generateDNSConfig grew a per-node CapMap argument so it can apply
nodeAttr-driven DNS overlays. The nextdns DoH rewrite hardens against
policy-controlled inputs:
- nextDNSDoHHost anchors the prefix match instead of substring,
so a hostile resolver URL cannot smuggle a nextdns hostname in
a path or query.
- nextDNSProfileFromCapMap accepts only profile names matching
[A-Za-z0-9._-]{1,64} and picks the lexicographically first when
multiple are granted -- deterministic, no shell metacharacters
or URL fragments through.
- addNextDNSMetadata composes the rewritten URL via url.Parse +
url.Values rather than fmt.Sprintf, so existing query strings
on the resolver URL survive and metadata cannot inject a new
component.
WithTaildropEnabled in servertest controls cfg.Taildrop.Enabled per
test so cap/file-sharing emission can be toggled in tests that need
to verify the off path.
ACL policies now accept a top-level nodeAttrs block. Each entry hands
a list of tailcfg node capabilities to every node matching target.
Accepted target forms are the same as acls.src and grants.src: users,
groups, tags, hosts, prefixes, autogroup:member, autogroup:tagged,
and *. autogroup:self, autogroup:internet, and autogroup:danger-all
are rejected at validate time because none describes a stable
identity set a node-level attribute can attach to.
NodeAttrGrant carries Targets, Attrs, and IPPool. IPPool is parsed
but rejected at validate time -- the allocator that consumes it is
not yet implemented. nodeAttrUnsupportedCaps lists caps SaaS accepts
that headscale cannot act on (funnel today) and rejects them with a
tracking-issue link in the error.
compileNodeAttrs resolves each entry's targets, then maps every
targeted node to a tailcfg.NodeCapMap of the entry's attrs. Per-node
IPs are cached once per call so the inner attr loop is O(grants)
instead of O(grants * nodes) IP allocations.
PolicyManager grows NodeCapMap (per-node), NodeCapMaps (snapshot for
batched callers), and NodesWithChangedCapMap (drain buffer for the
self-broadcast diff). refreshNodeAttrsLocked appends to the drain
rather than overwriting so a SetUsers/SetNodes between SetPolicy and
the drain cannot lose the policy-reload diff.
- Mention policy as generic term that covers ACLs or Grants
- Refresh routes policy examples
- Remove Headscale specific exit node separation. Use via instead.
Fixes: #3087
- Rename from acl.md to policy.md and setup redirect links
- Mention both ACLs and Grants
- Remove most old ACL docs and replace with links to Tailscale docs
- Add "Getting started" section
- Add section about notable differences
