docker network disconnect of one router triggers bridge
reconfiguration that transiently fails probes on OTHER routers
on the same network — a test-infrastructure quirk that does not
occur with a real cable pull. The lifecycle test's
"stays primary across X seconds" assertions on the prev-primary
return legs (phases 2b, 3b, 4d) tripped on this side effect.
Drop the identity assertions on those legs; keep traffic-flow
checks. Phases 4b and 5a (the no-flap regression barriers for
the algorithm fix) keep their stability windows.
Updates #3203
electPrimaryRoutes' all-unhealthy fallback used to pick
candidates[0] (lowest NodeID) regardless of who was prev primary.
Under cable-pull semantics IsOnline lags reality (long-poll TCP
half-open), so both routers stay in candidates and both go
Unhealthy via the prober — the fallback then churned primary to
a node that was itself unreachable.
Prefer prev when still in candidates; fall through to
candidates[0] only when prev is gone. Anti-blackhole property is
preserved (peers still see *some* primary), but no flap to a
known-bad node.
Update the property test's reference model to mirror the new
algorithm and split the unit test into existence
(KeepsAPrimary) and identity (PreservesPrevious) cases — the
old "lowest-ID wins" assertion encoded the bug.
Fixes#3203
Cover 5 phases of cable-pull lifecycle: initial primary, single
failure+recovery either side, sequential dual failure (the bug),
simultaneous dual failure. Phases 4b and 5a are the no-flap
regression barriers; the rest guard against future algorithm
changes regressing one lifecycle while fixing another.
Currently fails at phase 4b on this branch — primary flaps to
offline r1 within ~10s of r2 cable pull.
Updates #3203
Add a docker-network-disconnect primitive — `DisconnectFromNetwork`
/ `ReconnectToNetwork` on `TailscaleClient`, backed by
`pool.Client.DisconnectNetwork` — so an integration test can
faithfully simulate the cable-pull scenario the issue reporter
observed in Proxmox. Iptables-based simulations leave the socket
open at the container kernel and miss the bug.
Use it in `TestHASubnetRouterFailoverDockerDisconnect` to assert
the no-flap invariant: once primary has failed over from r1 to r2
and r2 also loses connectivity, primary must not switch back to
the still-offline r1. The assertion fails on the current branch —
the all-unhealthy fallback in `electPrimaryRoutes` flips primary
to the lowest-NodeID candidate regardless of online state. Locking
the failure in via TDD before designing the fix.
Updates #3203
The legacy routes.PrimaryRoutes.SetRoutes(node, empty) implicitly
deleted the node from the unhealthy set; the snapshot refactor lost
that side-effect, so a probe timeout that landed during a
SetApprovedRoutes(empty) (or just before a Disconnect) left
Unhealthy=true forever. The bit then surfaced in DebugRoutes and
broke TestHASubnetRouterFailover.
Restore the invariant at the write boundaries that drop HA
candidacy (Disconnect, SetApprovedRoutes, UpdateNodeFromMapRequest)
and add a defensive guard in SetNodeUnhealthy so the prober's
post-timeout write cannot install a stale bit when the node has
since left the candidate set.
Updates #3203
The primary route algorithm now lives in hscontrol/state/node_store.go
(snapshotFromNodes / computePrimaries). Move the test coverage that
used to live in hscontrol/routes alongside it: primaries_test.go ports
the named scenarios (anti-flap, HA failover, recovery-no-flap, exit
routes excluded, the issue #3203 both-offline regression) and HANodes,
primaries_property_test.go ports the rapid-driven invariant check.
Both tests drive the algorithm via NodeStore.UpdateNode rather than
the deleted PrimaryRoutes API. With every consumer migrated and the
algorithm relocated, the routes package has no remaining users and
gets removed.
Updates #3203
primaryRoutes lived as a separate write-then-read store guarded by an
external mutex. Each Connect, Disconnect, SetApprovedRoutes, and HA
prober write touched both NodeStore and primaryRoutes in sequence,
opening a TOCTOU window in which a stale Disconnect could overwrite a
fresh Connect's route assignment (issue #3203).
The election algorithm is a pure function of the snapshot: every node
contributes its IsOnline + AllApprovedRoutes + a runtime Unhealthy
bit. Move the computation into snapshotFromNodes so it runs in the
NodeStore writer goroutine and recomputes from immutable input on
every batch — no separate primaryRoutes mutation is required.
Caller-side changes are local. GetNodePrimaryRoutes, RoutesForPeer,
the HA prober and the debug endpoint all read PrimaryRoutesForNode /
HANodes / DebugRoutes off NodeStore. Health writes flow through
State.SetNodeUnhealthy, which captures the snapshot's primaries
before/after and signals back whether failover happened. The
SetApprovedRoutes and UpdateNodeFromMapRequest paths use the same
prevPrimaries / maps.Equal pattern to detect when announced/approved
changes shifted a primary.
The per-node lock and connectGen counter that closed the original
TOCTOU window are gone too: Connect bumps a SessionEpoch field on
the node inside its UpdateNode closure, Disconnect re-checks it in
its closure and no-ops on mismatch. The NodeStore writer goroutine
serialises both batches so the check + mutation are atomic by
construction.
Updates #3203
Routes package gets folded into NodeStore in a follow-up; the debug
endpoint payload moves to a neutral home so integration consumers
do not need to import the soon-removed routes package.
Updates #3203
Add primaries + isPrimary fields to Snapshot. snapshotFromNodes
now elects per-prefix primary advertisers from the current node
set, mirroring routes.PrimaryRoutes.updatePrimaryLocked: skip
exit routes, prefer the previous primary if still a valid
healthy advertiser, fall back to the lowest healthy NodeID, then
to the lowest NodeID overall when all are unhealthy.
Anti-flap memory rides the previous snapshot through applyBatch.
Caller-side coordination is unnecessary; the writer goroutine
serialises every mutation that could change the primary set.
Add NodeStore reader methods (PrimaryFor,
PrimaryRoutesForNode, HANodes, IsNodeHealthy) that mirror the
legacy routes.PrimaryRoutes API on the snapshot. Consumers will
switch in the next commit.
No behaviour change yet — primaries are computed but not read.
Updates #3203
Runtime-only fields (gorm:"-") that the upcoming refactor folds
into NodeStore's snapshot:
- Unhealthy: replaces routes.PrimaryRoutes.unhealthy set; written
by HA prober.
- SessionEpoch: replaces the external connectGen sync.Map; bumped
inside Connect's NodeStore update closure so stale Disconnect
rejection becomes atomic with the mutation.
No behaviour change yet — fields are unread.
Updates #3203
The generation check added in b09af384 closed the most obvious
window for stale Disconnect calls but left a residual TOCTOU:
the read of the gen counter was not atomic with the subsequent
NodeStore.UpdateNode and primaryRoutes.SetRoutes mutations. A
concurrent Connect that bumped the gen and rewrote primary
routes could be silently overtaken by a stale Disconnect's
SetRoutes(empty), leaving the node online in NodeStore but with
no primary route — the persistent broken state described in
the issue.
Replace the bare sync.Map[NodeID]*atomic.Uint64 with a
nodeLock pairing the counter with a sync.Mutex. Connect bumps
gen and runs its mutations under the lock; Disconnect acquires
the lock first, re-checks gen, and only proceeds on a match.
Per-node, so unrelated nodes do not contend; same-node steady
state has near-zero contention (one Connect at session start,
one Disconnect ~10s after grace period). Holding the lock
across persistNodeToDB is safe — that path goes through
nodeStore.RebuildPeerMaps which queues onto the writer
goroutine but is invoked from the caller, so no re-entrance.
TestConnectDisconnectRace drives the race directly via State,
deterministically reproducing the empty-primary state on main
and passing under the lock.
Fixes#3203
Cover the user-reported sequence in #3203: two subnet routers
advertise the same prefix, both go offline, one returns. Three
variants:
- servertest TestRoutes/ha_secondary_recovers_after_all_offline:
in-process Disconnect/Reconnect.
- integration TestHASubnetRouterFailoverBothOffline: tailscale
down/up between two routers, asserts both server and client
state restore.
- integration TestHASubnetRouterFailoverBothOfflineCablePull:
iptables -j DROP between router and headscale, mimicking a
real cable pull that breaks the ESTABLISHED long-poll.
All three pass on main — the production failure narrows to a
direct State.Connect/State.Disconnect race that the cable-pull
harness does not reliably hit, addressed in a follow-up commit.
The tests stay as regression coverage for the user's scenario.
Updates #3203
Generate randomised sequences of ConnectAdvertise, Disconnect,
ProbeUnhealthy, ProbeHealthy and ApprovedRoutesChange operations
with rapid and check the per-prefix primary chosen by the
implementation matches a reference model after every step.
Used to broaden the search for #3203 beyond hand-enumerated
disconnect/reconnect sequences. Algorithm passes 5000 checks;
kept as a safety net for future changes to primary route
selection.
Updates #3203
compileFilterRules, compileGrants, and updateLocked guarded the
"no rules so allow all" fallback with len(pol.Grants) == 0, which
matches both an absent grants field and an explicit empty array.
JSON {"grants": []} unmarshals to a non-nil empty slice; it should
compile to zero filter rules (deny all) to match Tailscale SaaS,
but the length check sent it down the FilterAllowAll path.
Distinguish absent (nil) from explicit-empty by switching the guard
to pol.Grants == nil, the same asymmetry already used for ACLs.
{} keeps allowing all; {"acls": []} and {"grants": []} now both
deny all.
Fixes#3211
The scheduled job has failed every night since 2026-04-13. Two
prior fixes (race-condition guards and splitting the combined
where predicate in #3200) did not address the actual cause:
`flatten` collapses nested records in a list-of-records pipeline,
so after `gh api ... | from json | flatten` the `label` and `user`
columns no longer exist - their fields are lifted to the top
level (with prefix only on naming collisions). `where label.name
== ...` and `where user.type != "Bot"` then both reference a
column that is not there.
`gh api --paginate` already returns a single concatenated JSON
array, so `from json` produces a list of records directly and no
flattening is needed. Drop both `| flatten` calls.
Verified locally with nu 0.108 against the events stream of issue
#3178: without flatten, `where event == "labeled" | where
label.name == "needs-more-info" | last` returns the labeled event
with its `label` record intact.
GitHub's /issues/:n/events endpoint returns a mixed-schema table.
Only labeled/unlabeled rows carry a `label` column. Nu's `where`
does not short-circuit on missing columns, so the combined
predicate `event == "labeled" and label.name == ...` dereferenced
`label.name` on every row and crashed on the first non-labeled
event with `nu::shell::column_not_found`.
The scheduled job has failed every night since 2026-04-13 (the
first run with a labeled issue), so no `needs-more-info` issue
has been auto-closed.
Split into two sequential `where` filters so `label.name` is only
accessed on rows that have the column.
Summarise the ingest rewrite, the SaaS-matching collision rule, and the
BREAKING change from random-suffix to numeric-suffix collision labels
and from "invalid-<rand>" to the literal "node" fallback.
Updates #3188
Ingest (registration and MapRequest updates) now calls
dnsname.SanitizeHostname directly and lets NodeStore auto-bump on
collision. Admin rename uses dnsname.ValidLabel + SetGivenName so
conflicts are surfaced to the caller instead of silently mutated.
Three duplicate invalidDNSRegex definitions, the old NormaliseHostname
and ValidateHostname helpers, EnsureHostname, InvalidString,
ApplyHostnameFromHostInfo, GivenNameHasBeenChanged, generateGivenName
and EnsureUniqueGivenName are removed along with their tests.
ValidateHostname's username half is retained as ValidateUsername for
users.go.
The SaaS-matching collision rule replaces the random "invalid-xxxxxx"
fallback and the 8-character hash suffix; the empty-input fallback is
the literal "node". TestUpdateHostnameFromClient now exercises the
rewrite end-to-end with awkward macOS/Windows names.
Fixes#3188Fixes#2926Fixes#2343Fixes#2762Fixes#2449
Updates #2177
Updates #2121
Updates #363
NodeStore's writer goroutine now resolves GivenName collisions inside
applyBatch: on PutNode/UpdateNode the landing label gets -N appended
until unique, matching Tailscale SaaS. Empty labels fall back to the
literal "node".
SetGivenName exposes the admin-rename path: validates via
dnsname.ValidLabel and rejects on collision with ErrGivenNameTaken,
so renames do not silently rewrite behind the caller.
Updates #3188
Updates #2926
Updates #2343
Updates #2762
Existing HA tests verify server-side primary election; these add
end-to-end assertions from a viewer client's perspective, that marking
the primary unhealthy or revoking its approved route propagates through
the netmap so the viewer sees the flip.
Updates #3157
Resolve by GivenName (unique per tailnet) before Hostname (client-
reported, may collide); within each pass, pick the lowest NodeID so
results are deterministic across NodeStore snapshot iterations.
Updates #3157
change.Merge keeps the first PingRequest seen when merging, which
means a later probe's callback URL is silently dropped if a stale merge
is in-flight. Document the contract at Merge and at doPing so callers
know to serialise probes.
Updates #3157
dev derives additional ports from --port + offsets (metrics, gRPC,
debug). A --port near uint16 max would overflow silently; add up-front
validation that rejects values that would push derived ports over 65535.
Updates #3157
DestsIsTheInternet now reports the internet when either family's /0
is contained (0.0.0.0/0 or ::/0), matching what operators expect when
they write the /0 directly. Also documents MatchFromStrings fail-open.
Updates #3157
The /debug/ping callback hits /machine/ping-response on the main
TLS router, not the noise chain, so URLIsNoise stays false. Document
this at the emit site to prevent accidental changes.
Updates #3157
Blocked callers waiting on a pingTracker response channel would
hang forever if the server Close()d mid-probe. Drain the pending map on
Close so those goroutines unblock and exit cleanly.
Updates #3157
X-Frame-Options: DENY and frame-ancestors 'none' stop clickjacking
of OIDC, register-confirm, and debug HTML pages. nosniff and no-referrer
are cheap defence-in-depth for the same surfaces.
Updates #3157
chi routes only HEAD to the handler, but assert explicitly so a
future router config change cannot silently accept GET/POST and leak
latency bytes or side-effects.
Updates #3157
elem-go does not escape attribute values, so the raw query reaches
the rendered HTML verbatim. Pre-escape with html.EscapeString to prevent
reflected XSS.
Updates #3157
Subnet routers that advertise routes must not accept peer routes.
With co-router visibility the HA primary's subnet appears in co-routers'
AllowedIPs, and --accept-routes installs a system route that conflicts
with local subnet forwarding.
Updates #3157
Data-driven tests for via grants combined with HA primary routes:
crossed via tags on same prefix, mixed via+regular across HA pairs,
four-way HA, and the kitchen-sink scenario. Each case uses an inline
topology captured from SaaS.
Updates #3157
End-to-end exercise of via-grant compilation against SaaS captures:
peer visibility, AllowedIPs, PrimaryRoutes, and per-rule src/dst
reachability from each viewer's perspective.
Updates #3157
Typed Capture/Input/Node/Topology structs for golden SaaS captures.
Schema drift between the tscap capture tool and headscale now becomes a
compile error instead of a silent test pass.
Updates #3157
Tests were dumping megabytes of zerolog output on failure; silence
at init and let individual tests opt in via SetGlobalLevel when they need
log-driven assertions.
Updates #3157
reduceCapGrantRule was dropping rules whose CapGrant IPs overlap a
subnet route; treat subnet routes as part of node identity so those rules
survive reduction. ReduceFilterRules now also reduces route-reachable
destinations.
Updates #3157
Threads PolicyManager into compiledGrant so via grants resolve
viewer identity at compile time instead of re-resolving per MapRequest.
Adds a matchersForNodeMap cache invalidated on policy reload and on node
add/remove.
Updates #3157
CanAccess now treats a node's advertised subnet routes as part of
its source identity, so an ACL granting the subnet-owner as source lets
traffic from the subnet through. Matches SaaS semantics.
Updates #3157
Golden captures of SaaS filter-rules and netmaps across the ACL,
grant, routes, and SSH corpora. These back the data-driven compat tests
that verify headscale's policy output against Tailscale SaaS verbatim.
Updates #3157
Block ping callbacks via iptables while keeping the Noise session alive
to simulate a zombie-connected router. Verify the prober detects it,
fails over, and does not flap on recovery.
Updates #2129
Updates #2902
Kristoffer Dalby
QEDeD
Florian Preinstorfer