Compare commits

...

113 Commits

Author SHA1 Message Date
Kristoffer Dalby d6082cb9f6 testdata: re-run tscap with populated approved_routes and tag fixes
Replaces the previous capture corpus with the output of a fresh tscap
batch built from:

  - kradalby/tscap@d2cd4f9 (anonymize: substitute tags too)
  - kradalby/tscap@3080147 (runner: capture autoApprover decisions in
    ApprovedRoutes)

Two regressions in the previous corpus are fixed:

  1. approved_routes was committed as [] on every node in all 617
     files because the runner never asked the SaaS API which routes had
     been auto-approved. The new corpus has 533/1296 routes_results and
     462/3765 grant_results entries with non-empty approved sets,
     matching what SaaS actually returned.

  2. Anonymization rewrote tag references in policy JSON
     (tag:exit-a → tag:pidgey, tag:router-a → tag:spearow, …) but left
     node.Tags untouched, so via-grant-* and other scenarios that use
     these collision-prone tag names had policies pointing at tag
     identifiers no node carried. The new corpus has node.Tags routed
     through the same substitution table.

Compat-test impact (subtest level):

                  before        after        delta
    Routes        1697 / 825    3397 / 13    -812 fail
    ACL           1799 /   8    1805 /  0      -8 fail
    Grants        3426 /  48    3463 / 11     -37 fail
    SSH            521 /  71     536 / 56     -15 fail
    ViaGrant         0 /   4      31 / 21      n/a (test now actually
                                                  loads files)
    Total         7443 / 956    9232 / 101   -855 fail (-89.4%)

Verified zero true accuracy regressions: every test that passed
against the old corpus and now fails against the new corpus is a
false-pass-becoming-true-fail. The single nominal regression
(grant-j3/beedrill) used to pass because both sides were empty under
the broken approved_routes; the new file exposes a real divergence in
filter.go srcIPsWithRoutes where headscale appends approved subnet
routes to wildcard SrcIPs and SaaS does not.

Updates #3157
2026-04-09 06:20:03 +00:00
Kristoffer Dalby 0c577301c6 policy/v2: ignore implementation-specific SSH check action fields
The recapture of the SSH golden corpus by tscap revealed five fields
on tailcfg.SSHAction that the headscale compiler emits with different
defaults from Tailscale SaaS for "check" actions:

  - SessionDuration: headscale uses SSHCheckPeriodDefault (12h) when
    the rule omits checkPeriod; SaaS emits 0s.

  - AllowAgentForwarding, AllowLocalPortForwarding,
    AllowRemotePortForwarding: filter.go sshCheck/sshAccept hardcode
    these to true; SaaS emits false on check.

  - HoldAndDelegate URL template: headscale embeds /from/…?ssh_user=
    so its own check handler can identify the requested SSH user; SaaS
    omits both the /from/ path segment and the ssh_user query
    parameter.

These are deliberate headscale design choices, not capture artifacts:
tscap pulls SSHRules straight out of the netmap as a json.RawMessage
and never rewrites them, so the new captures are the authoritative
SaaS output.

Pass an IgnoreFields option to cmp.Diff for those five fields so the
SSH compat test stops flagging the divergence on every check rule, and
add a follow-up presence assertion that asserts headscale and SaaS
agree on whether each rule has HoldAndDelegate set so a regression
that drops the URL entirely is still caught.

71 → 56 SSH compat failures; the residual 56 are unrelated bugs in
compileSSHPolicy (missing rules, duplicate principals) that surface
on a different set of files.

Updates #3157
2026-04-09 06:19:17 +00:00
Kristoffer Dalby 1e20972db0 servertest: align via grant compat test with new tscap golden format
The previous tscap regeneration replaced the legacy uppercase
GRANT-V*.hujson files with lowercase via-grant-v*.hujson and switched
the topology nodes to anonymized pokémon identifiers, but
TestViaGrantMapCompat was still loading files by their old IDs and
matching topology entries by their old hostnames, so it could not even
open a single golden file.

  - viaCompatTests: GRANT-V29/V30/V31/V36 → via-grant-v29/v30/v31/v36
    so the path Join finds the renamed files on disk.

  - taggedNodes: rewrite the hardcoded servertest hostname list to the
    pokémon names tscap/anonymize writes (big-router→caterpie,
    exit-a→pidgey, …) so the topology lookup picks up the matching
    entries instead of skipping every node.

  - goldenNode: track the tscap schema. Replace advertised_routes →
    routable_ips and is_exit_node → approved_routes; the AdvertisedRoutes
    field name is kept on the Go side because the rest of the test
    code already references it.

  - convertViaPolicy: also rewrite the anonymized odin/thor/freya
    @example.com emails so the served policy resolves users when the
    file came from tscap/anonymize.

Updates #3157
2026-04-09 06:18:43 +00:00
Kristoffer Dalby c0fd9b60ed mapper, db: drop redundant zerolog SetGlobalLevel calls in tests
The previous commit centralizes test log silencing in
hscontrol/types/testlog.go, so the per-test SetGlobalLevel calls scattered
across the batcher and db tests are now redundant. Most of them were also
buggy: they restored to DebugLevel instead of capturing and restoring the
prior level, polluting global state for subsequent tests in the same binary.

Remove all such pairs from batcher_bench_test.go, batcher_concurrency_test.go,
batcher_scale_bench_test.go, and suite_test.go, along with the now-unused
zerolog imports. The one correct save/restore pattern in batcher_test.go is
left untouched: it captures originalLevel via zerolog.GlobalLevel() and works
transparently with the new ErrorLevel default.
2026-04-08 20:50:15 +00:00
Kristoffer Dalby d3a6afdbf0 hscontrol/types: silence zerolog by default in tests
Running go test ./... produced an extreme amount of zerolog output because
test binaries inherited the package default of DebugLevel. Per-test mitigations
were scattered across roughly fifty test functions and most of them were
subtly broken: they restored to DebugLevel instead of the prior level, leaking
state into subsequent tests in the same binary.

Add a single init() in hscontrol/types/testlog.go that detects test mode via
testing.Testing() and lowers the global zerolog level to ErrorLevel. The types
package is transitively imported by every test binary that emits zerolog
output, so this one insertion point silences go test ./... without any
per-package boilerplate. Production binaries are unaffected because
testing.Testing() returns false outside of test execution; the same pattern
already exists in hscontrol/db/{users,node}.go.

Set HEADSCALE_TEST_LOG_LEVEL=trace|debug|info|warn|error|disabled to override
when debugging a specific test.

hscontrol/util/zlog/zlog_test.go asserts on Info-level output from a
zerolog.New(&buf) logger, which is also gated by the global level. Add a
defensive init_test.go in that package pinning the global level to TraceLevel
so the assertions stay reliable if a future import chain pulls
hscontrol/types into the zlog test binary.

Document the mechanism, the env var override, the save/restore idiom for
per-test overrides, and the relevant pitfalls under Testing Guidelines in
AGENTS.md.
2026-04-08 20:49:47 +00:00
Kristoffer Dalby 800fed8d50 testdata: replace with multi-user tscap output
Re-run of the full corpus using tscap's new per-user API keys so each
untagged node is owned by its intended user (odin/thor/freya) instead
of all being collapsed onto the admin account. Jumps compat pass rate:

  before multi-user: 6564 / 7289 (90.0%)
  after multi-user:  6979 / 7289 (95.7%)

Source: github.com/kradalby/tscap @ c2c5a9e

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 15:28:23 +00:00
Kristoffer Dalby 818a8eb8e3 compat tests: build node topology per-scenario, not globally
tscap uses clean-slate mode: each scenario wipes every device from the
tailnet before logging new ones in, so node IPs change from scenario to
scenario. The previous compat tests built a single nodes slice from the
first file's topology (or from a hardcoded setup) and used it for every
scenario, which produced IP mismatches in the filter rule comparisons.

Rebuild types.Nodes per scenario from the current file's Topology. ACL
already had a build-from-topology helper; extract the equivalent for
grants (buildGrantsNodesFromCapture) and use it for SSH too. Drop the
"first 8 nodes only" shim in grants.

Pass count jumped ~2500 tests across the four compat suites.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:35:11 +00:00
Kristoffer Dalby 9e96adfa49 testdata: replace legacy captures with anonymized tscap output
Drop the old testdata/{acl,grant,routes,ssh}_results/ files (mixed
uppercase ACL-*/ROUTES-*/GRANT-*/SSH-* filenames with the pre-tscap
schema) and replace with the 617 captures tscap produced for the
kradalby/3157-subnet-to-subnet branch. Files are lowercase and
anonymized (odin/thor/freya users, pokemon hostnames) per
kradalby/tscap package anonymize.

Relax pre-commit check-added-large-files: tscap captures include full
netmaps and can exceed 500 KB per file; testdata/ is now excluded
from the size check and the global ceiling is bumped to 2 MB.

Source: github.com/kradalby/tscap @ cb375ec

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 11:29:41 +00:00
Kristoffer Dalby 9f5e49d3ee compat tests: switch to anonymized norse-god + pokemon names
tscap now anonymizes captures at write time (see kradalby/tscap
package anonymize): SaaS user identifiers become odin/thor/freya and
node hostnames become original-151 pokemon names. Update the four
policy v2 compat test setups to construct types.User and types.Node
objects with the new names so the in-test topology matches what the
captured testdata files contain.

The convertPolicyUserEmails / convertSaaSEmail helpers become
passthroughs because tscap already produces the final email form.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 08:21:31 +00:00
Kristoffer Dalby d1bd6e5b88 testcapture: add capture format package and rewire compat tests
Add hscontrol/types/testcapture, mirroring the type that the tscap tool
emits. The package has no tailscale.com/tailcfg import so it stays
dependency-free; wire-format Tailscale data is stored as
json.RawMessage and consumers unmarshal into the typed shape they need.

Drop the four private *TestFile structs from the policy v2 compat tests
and read directly into testcapture.Capture. Switch globs to lowercase
(acl-/grant-/routes-/ssh-) to match the new tscap output filenames.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 08:05:45 +00:00
Kristoffer Dalby f213a8c22a policy/v2: build ACL compat nodes from golden file topology
Replace the hardcoded 8-node setupACLCompatNodes with
buildACLUsersAndNodes which reads from the golden file topology.

Updates #3157
2026-04-06 08:15:49 +00:00
Kristoffer Dalby 6a8a0b4b14 testdata: recapture all golden files with 19-node tailnet and netmap
Recapture all golden test data from Tailscale SaaS with:
- All 19 nodes online (12 routes + 7 grant v2 topology)
- All routes approved via API
- Trimmed netmap data per capture (Peers with AllowedIPs,
  PrimaryRoutes, Tags; SelfNode with RoutableIPs)
- 20-30s propagation wait for correct data

Topology expanded from netmap peers to include all 19 nodes that
contribute to filter rule SrcIPs. Each node has routable_ips and
approved_routes derived from netmap SelfNode data.

New files: b11-b16 (autogroup:internet), i8-i9 (IPv6 fd00:1::/64),
pa1-pa2 (partial route approval), V31-SLOW.

Updates #3157
2026-04-06 08:07:07 +00:00
Kristoffer Dalby 2469c3eb80 policy/v2: remove obsolete global vs per-node equivalence tests
With the unified compiledGrant infrastructure, there is only one
filter compilation path. The equivalence tests that compared the
old global path against the old per-node path no longer serve a
purpose since both callers now go through the same compileGrants
resolution.

Remove TestACLCompatGlobalEquivalence, TestGrantsCompatGlobalEquivalence,
and TestRoutesCompatGlobalEquivalence.

Updates #3157
2026-04-04 16:15:44 +00:00
Kristoffer Dalby 3065b45310 policy/v2: make ACL error validation strict
Replace the lenient testACLError handler (t.Logf on mismatch) with
strict validation matching the grant error handler pattern:

- Extract assertACLErrorContains helper with progressive fallbacks:
  direct match, mapped equivalent, key-part extraction, then t.Errorf
- Convert HeadscaleDiffers from t.Logf to t.Skipf for visibility
- Add aclErrorMessageMap for known wording differences (empty for now)

All 16 ACL error golden files pass with the strict handler.

Updates #3157
2026-04-04 13:21:05 +00:00
Kristoffer Dalby d77ce3582d policy: document CanAccessRoute and filterForNodeLocked design
Add source comments explaining design decisions:

CanAccessRoute (types/node.go): Explain why DestsIsTheInternet is
intentionally absent. Exit routes are handled by RoutesForPeer via
ExitRoutes(), not ACL-based filtering. autogroup:internet is skipped
during filter compilation, so no matchers contain 'the internet'.

Updates #3157
2026-04-04 13:20:45 +00:00
Kristoffer Dalby bbea92033c policy/v2: wire PolicyManager through compiledGrant, delete old paths
Store compiledGrants and userNodeIndex on PolicyManager. Both
filterForNodeLocked and BuildPeerMap now derive from the same
compiled grants, eliminating the dual-path architecture.

Delete:
- compileViaGrant (filter.go): replaced by compileViaForNode
- compileGrantWithAutogroupSelf (filter.go): replaced by
  compileGrants + compileAutogroupSelf
- compileFilterRulesForNodeLocked (policy.go): replaced by
  filterRulesForNodeLocked
- compiledFilterRulesMap cache (policy.go): no longer needed
- usesAutogroupSelf field and method: detection now happens
  during grant compilation (grantCategorySelf/grantCategoryVia)
- hasViaGrants method: same

The needsPerNodeFilter flag is now derived from the compiled grants
via hasPerNodeGrants. filterForNodeLocked no longer returns error
since the new path cannot fail.

Updates #3157
2026-04-04 13:18:15 +00:00
Kristoffer Dalby ac8f9c7ecc policyutil: fix reduceCapGrantRule and add reduction tests
Fix fragile IP indexing in reduceCapGrantRule: replace hard-coded
nodeIPs[0]/nodeIPs[1] access with a range loop, matching the pattern
used by the adjacent broad-prefix branch. The old code would panic
if a node had zero IPs.

Remove redundant SubnetRoutes check in ReduceFilterRules: the check
was strictly redundant with the RoutableIPs check since SubnetRoutes
is always a subset of RoutableIPs. Document that golden data always
has routable_ips == approved_routes so we cannot determine which set
Tailscale actually checks.

Add tests:
- TestReduceFilterRulesCapGrant: 7 cases covering IP narrowing,
  non-matching filtered out, subnet route overlap, exit route
  skipping, mixed DstPorts+CapGrant, IPv4-only, and zero-IP nodes
- TestReduceFilterRulesPartialApproval: 3 cases documenting behavior
  for approved, unapproved-but-advertised, and non-advertised routes

Updates #3157
2026-04-04 12:51:32 +00:00
Kristoffer Dalby 6e5508444f policy/v2: add false-positive peer test for all 98 golden files
TestRoutesCompatNoPeersBeyondCaptures extends the negative peer
assertion from 6 hardcoded files (f10-f15) to all 98 ROUTES golden
files using a generic, data-driven approach.

For each golden file, the test derives expected peer pairs from
capture SrcIPs (subnet routes, direct node IPs) and DstPorts
(destination node IPs, route CIDRs), handling IP dash-range format
(100.64.0.0-100.115.91.255) and wildcards. It then asserts that no
unexpected CanAccess=true pairs exist.

Also adds addSrcIPToBuilder helper and deriveAllPeerPairsFromCaptures
for comprehensive peer derivation from golden data.

Updates #3157
2026-04-04 12:51:32 +00:00
Kristoffer Dalby 8754ed10dc policy/v2: add global vs per-node filter compilation equivalence tests
Add equivalence tests that compare the global filter path
(PolicyManager.FilterForNode) against the per-node path
(compileFilterRulesForNode + ReduceFilterRules) for all golden files
where the two paths should produce identical output.

TestRoutesCompatGlobalEquivalence: 98 ROUTES files, 12 nodes each.
All pass (1,176 comparisons).

TestACLCompatGlobalEquivalence: ~152 ACL files (excluding error and
autogroup:self cases), 8 nodes each. Surfaces 2 divergences (ACL-M06,
ACL-K01) where the global path loses individual non-wildcard source
IPs absorbed into the CGNAT range.

TestGrantsCompatGlobalEquivalence: ~186 GRANT files (excluding error,
autogroup:self, and via cases), 8-15 nodes each. Surfaces 3
divergences (GRANT-P09_7A, GRANT-K29, GRANT-H6) with the same root
cause.

The per-node path matches Tailscale SaaS golden data in all 5
divergent cases, confirming it preserves nonWildcardSrcs correctly.

Updates #3157
2026-04-04 12:51:32 +00:00
Kristoffer Dalby db8ac49680 servertest: improve via compat test infrastructure
Add big-router to the taggedNodes list for via compat tests. This
node has tags [tag:router, tag:exit] and is needed for future
V-series golden file tests. Existing tests are unaffected.

Enhance compareNetmap to compare PacketFilter rule content (destination
prefixes, source non-emptiness), not just rule count. Six golden files
have full SrcIPs/DstPorts data that was previously unvalidated.

Add Tier 3 route inference to inferNodeRoutes: scan each node's own
packet_filter_rules DstPorts for non-Tailscale-IP route prefixes.
This catches secondary HA routers whose routes don't appear in
AllowedIPs but do receive filter rules.

Updates #3157
2026-04-04 12:51:32 +00:00
Kristoffer Dalby 8a3a4af03d policy/matcher: add unit tests for matcher package
The matcher_test.go file was empty (just the package declaration).
Add comprehensive unit tests covering all public methods:

- TestMatchFromStrings: CIDR, wildcard, single IP, IPv6, empty inputs
- TestMatchFromFilterRule: standard rules, wildcard dst, empty DstPorts
- TestMatchesFromFilterRules: multi-rule conversion
- TestSrcsOverlapsPrefixes: exact, parent/child, no overlap, IPv6
- TestDestsOverlapsPrefixes: exact, parent/child, wildcard
- TestDestsIsTheInternet: 0.0.0.0/0, ::/0, wildcard, private ranges
- TestDebugString: output format validation

DestsIsTheInternet had zero test coverage (neither direct nor indirect)
prior to this change.

Updates #3157
2026-04-03 09:16:48 +00:00
Kristoffer Dalby bbcbcd3ea5 policy/v2: add auto-approval and route access compat tests
Add two new data-driven compatibility tests validated against all 98
Tailscale SaaS golden files:

- TestRoutesCompatAutoApproval: verifies NodeCanApproveRoute produces
  the same approval decisions as captured in golden files. Exit routes
  (0.0.0.0/0, ::/0) are skipped because Tailscale SaaS stores them
  under autoApprovers.routes while headscale uses a separate
  autoApprovers.exitNode field.

- TestRoutesCompatReduceRoutes: verifies CanAccessRoute produces route
  visibility decisions consistent with golden captures. Extends the
  coverage from 6 files (f10-f15) to all 98 golden files by deriving
  expected access from the SrcIPs in each node's captured filter rules.

Updates #3157
2026-04-03 07:26:23 +00:00
Kristoffer Dalby 85a03db016 policy/v2: add peer visibility compat tests for subnet-to-subnet ACLs
Add TestRoutesCompatPeerVisibility and TestRoutesCompatNoFalsePositivePeers
which use the Tailscale SaaS golden file data to validate the CanAccess /
ReduceNodes / ReduceRoutes code paths for subnet-to-subnet ACL scenarios.

These tests derive expected peer relationships from the golden file
captures: if Tailscale SaaS delivers filter rules to a node with SrcIPs
overlapping another node's subnet routes, those nodes must be peers. This
exercises the CanAccess fix where subnet routes are treated as source
identity, ensuring the fix is validated against real SaaS behavior.

Also fix buildRoutesUsersAndNodes to auto-assign unique node IDs when the
golden files don't provide them, which is required for ReduceNodes to
correctly skip self-comparisons.

Updates #3157
2026-04-02 20:47:06 +00:00
Kristoffer Dalby 34fd7d00ea testdata: add subnet-to-subnet route compatibility golden files
Add 6 golden files captured from Tailscale SaaS that cover subnet-to-subnet
ACL scenarios where both source and destination are subnet CIDRs. These fill
a complete gap in the existing 92 ROUTES golden files, none of which tested
this pattern.

Scenarios covered:
  f10: exact issue #3157 reproduction (disjoint subnets, host aliases)
  f11: bidirectional with overlapping big-router
  f12: host aliases where dst is a superset route
  f13: disjoint subnets, bidirectional
  f14: both src and dst overlap on one router
  f15: cross-router bidirectional

All 6 pass TestRoutesCompat, confirming headscale matches Tailscale SaaS
behavior for subnet-to-subnet filter rule distribution.

Updates #3157
2026-04-02 16:01:40 +00:00
Ubuntu 232a9c8f9a integration: add subnet-to-subnet ACL routing test
Add TestSubnetToSubnetACL which creates two subnet routers on separate
Docker networks and verifies that ACL rules referencing only subnet
CIDRs (not node IPs) as source and destination produce correct peer
visibility, route distribution, and end-to-end connectivity.

Updates #3157
2026-04-02 13:25:34 +00:00
Ubuntu ea85dcbc14 types: consider subnet routes as source identity in ACL matching
CanAccess and CanAccessRoute only used a node's Tailscale IPs as
source identity when matching ACL rules. A subnet router acting on
behalf of its advertised subnets was invisible to ACLs that reference
subnet CIDRs as source (e.g. src=10.88.8.0/24 dst=10.99.9.0/24).

Extend both functions to also check whether the node's approved
subnet routes overlap the matcher's source set. This makes subnet
routers visible to each other when ACLs reference their subnet CIDRs,
enabling subnet-to-subnet traffic routing.

The new checks are guarded by len(subnetRoutes) > 0, so non-router
nodes (the vast majority) pay zero additional cost.

Fixes #3157
2026-04-02 13:23:23 +00:00
Ubuntu 45d5544736 types,policy: add tests for subnet-to-subnet route visibility
Add unit tests covering the subnet-to-subnet ACL scenario where both
src and dst are subnet CIDRs served by different subnet routers.

Tests cover CanAccess (peer visibility), ReduceRoutes (route filtering),
and ReduceNodes (peer list reduction). Positive cases for routers that
advertise the referenced subnets, and negative cases ensuring regular
nodes and unrelated routers gain no extra visibility.

These tests currently fail and will be fixed in the next commit.

Updates #3157
2026-04-02 13:23:14 +00:00
Kristoffer Dalby 835db974b5 testdata: strip unused fields from all test data files (23MB -> 4MB)
Strip fields not consumed by any test from all 594 HuJSON test data files:

grant_results/ (248 files, 21MB -> 1.8MB):
  - Remove: timestamp, propagation_wait_seconds, input.policy_file,
    input.grants_section, input.api_endpoint, input.api_method,
    topology.nodes.mts_name, topology.nodes.socket, topology.nodes.user_id,
    captures.commands, captures.packet_filter_matches, captures.whois
  - V14-V16, V26-V36: keep stripped netmap (Peers.Name/AllowedIPs/PrimaryRoutes
    + PacketFilterRules) for via_compat_test.go compatibility
  - V17-V25: strip netmap (old topology, incompatible with via_compat harness)

acl_results/ (215 files, 1.4MB -> 1.2MB):
  - Remove: timestamp, propagation_wait_seconds, input.policy_file,
    input.api_endpoint, input.api_response_code, entire topology section
    (parsed by Go struct but completely ignored — nodes are hardcoded)

routes_results/ (92 files, unchanged — topology is actively used):
  - Remove: timestamp, propagation_wait_seconds, input.policy_file,
    input.api_endpoint, input.api_response_code

ssh_results/ (39 files, unchanged — minimal to begin with):
  - Remove: policy_file
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 30dce30a9d testdata: convert .json to .hujson with header comments
Rename all 594 test data files from .json to .hujson and add
descriptive header comments to each file documenting what policy
rules are under test and what outcome is expected.

Update test loaders in all 5 _test.go files to parse HuJSON via
hujson.Parse/Standardize/Pack before json.Unmarshal.

Add cross-dependency warning to via_compat_test.go documenting
that GRANT-V29/V30/V31/V36 are shared with TestGrantsCompat.

Add .gitignore exemption for testdata HuJSON files.
2026-04-01 14:10:42 +01:00
Kristoffer Dalby f693cc0851 CHANGELOG: document grants support for 0.29.0
Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby abd2b15db5 policy/v2: clean up dead error variables, stale TODO, and test skip reasons
Remove unused error variables (ErrGrantViaNotSupported, ErrGrantEmptySources, ErrGrantEmptyDestinations, ErrGrantViaOnlyTag) and the stale TODO for via implementation. Update compat test skip reasons to reflect that user:*@passkey wildcard is a known unsupported feature, not a pending implementation.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby b762e4c350 integration: remove exit node via grant tests
Remove TestGrantViaExitNodeSteering and TestGrantViaMixedSteering.
Exit node traffic forwarding through via grants cannot be validated
with curl/traceroute in Docker containers because Tailscale exit nodes
strip locally-connected subnets from their forwarding filter.

The correctness of via exit steering is validated by:
- Golden MapResponse comparison (TestViaGrantMapCompat with GRANT-V31
  and GRANT-V36) comparing full netmap output against Tailscale SaaS
- Filter rule compatibility (TestGrantsCompat with GRANT-V14 through
  GRANT-V36) comparing per-node PacketFilter rules against Tailscale SaaS
- TestGrantViaSubnetSteering (kept) validates via subnet steering with
  actual curl/traceroute through Docker, which works for subnet routes

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby c36cedc32f policy/v2: fix via grants in BuildPeerMap, MatchersForNode, and ViaRoutesForPeer
Use per-node compilation path for via grants in BuildPeerMap and MatchersForNode to ensure via-granted nodes appear in peer maps. Fix ViaRoutesForPeer golden test route inference to correctly resolve via grant effects.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 6a55f7d731 policy/v2: add via exit steering golden captures and tests
Add golden test data for via exit route steering and fix via exit grant compilation to match Tailscale SaaS behavior. Includes MapResponse golden tests for via grant route steering verification.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby bca6e6334d integration: add custom subnet support and fix exit node tests
Add NetworkSpec struct with optional Subnet field to ScenarioSpec.Networks.
When Subnet is set, the Docker network is created with that specific CIDR
instead of Docker's auto-assigned RFC1918 range.

Fix all exit node integration tests to use curl + traceroute. Tailscale
exit nodes strip locally-connected subnets from their forwarding filter
(shrinkDefaultRoute + localInterfaceRoutes), so exit nodes cannot
forward to IPs on their Docker network via the default route alone.
This is by design: exit nodes provide internet access, not LAN access.
To also get LAN access, the subnet must be explicitly advertised as a
route — matching real-world Tailscale deployment requirements.

- TestSubnetRouterMultiNetworkExitNode: advertise usernet1 subnet
  alongside exit route, upgraded from ping to curl + traceroute
- TestGrantViaExitNodeSteering: usernet1 subnet in via grants and
  auto-approvers alongside autogroup:internet
- TestGrantViaMixedSteering: externet subnet in auto-approvers and
  route advertisement for exit traffic

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 0431039f2a servertest: add regression tests for via grant filter rules
Add three tests that verify control plane behavior for grant policies:

- TestGrantViaSubnetFilterRules: verifies the router's PacketFilter
  contains destination rules for via-steered subnets. Without per-node
  filter compilation for via grants, these rules were missing and the
  router would drop forwarded traffic.

- TestGrantViaExitNodeFilterRules: same verification for exit nodes
  with via grants steering autogroup:internet traffic.

- TestGrantIPv6OnlyPrefixACL: verifies that address-based aliases
  (Prefix, Host) resolve to exactly the literal prefix and do not
  expand to include the matching node's other IP addresses. An
  IPv6-only host definition produces only IPv6 filter rules.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby ccd284c0a5 policy/v2: use per-node filter compilation for via grants
Via grants compile filter rules that depend on the node's route state
(SubnetRoutes, ExitRoutes). Without per-node compilation, these rules
were only included in the global filter path which explicitly skips via
grants (compileFilterRules skips grants with non-empty Via fields).

Add a needsPerNodeFilter flag that is true when the policy uses either
autogroup:self or via grants. filterForNodeLocked now uses this flag
instead of usesAutogroupSelf alone, ensuring via grant rules are
compiled per-node through compileFilterRulesForNode/compileViaGrant.

The filter cache also needs to account for route-dependent compilation:

- nodesHavePolicyAffectingChanges now treats route changes as
  policy-affecting when needsPerNodeFilter is true, so SetNodes
  triggers updateLocked and clears caches through the normal flow.

- invalidateGlobalPolicyCache now clears compiledFilterRulesMap
  (the unreduced per-node cache) alongside filterRulesMap when
  needsPerNodeFilter is true and routes changed.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 9db5fb6393 integration: fix error message assertion for invalid ACL action
Action.UnmarshalJSON produces the format
'action="unknown-action" is not supported: invalid ACL action',
not the reversed format the test expected.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 3ca4ff8f3f state,servertest: add grant control plane tests and fix via route ReduceRoutes filtering
Add servertest grant policy control plane tests covering basic grants, via grants, and cap grants. Fix ReduceRoutes in State to apply route reduction to non-via routes first, then append via-included routes, preventing via grant routes from being incorrectly filtered.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 5cd5e5de69 policy/v2: add unit tests for ViaRoutesForPeer
Test via route computation for viewer-peer pairs: self-steering returns
empty, viewer not in source returns empty, peer without advertised
destination returns empty, peer with/without via tag populates
Include/Exclude respectively, mixed prefix and autogroup:internet
destinations, and exit route steering.

7 subtests covering all code paths in ViaRoutesForPeer.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 08d26e541c policy/v2: add unit tests for grant filter compilation helpers
Test companionCapGrantRules, sourcesHaveWildcard, sourcesHaveDangerAll,
srcIPsWithRoutes, the FilterAllowAll fix for grant-only policies,
compileViaGrant, compileGrantWithAutogroupSelf grant paths, and
destinationsToNetPortRange autogroup:internet skipping.

51 subtests across 8 test functions covering all grant-specific code
paths in filter.go that previously had no test coverage.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby d243adaedd types,mapper,integration: enable Taildrive and add cap/drive grant lifecycle test
Add NodeAttrsTaildriveShare and NodeAttrsTaildriveAccess to the node capability map, enabling Taildrive file sharing when granted via policy. Add integration test verifying the full cap/drive grant lifecycle.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 9b1a6b6c05 integration: add cap/relay grant peer relay lifecycle test
Add ConnectToNetwork to the TailscaleClient interface for multi-network test scenarios and implement peer relay ping support. Use these to test that cap/relay grants correctly enable peer-to-peer relay connections between tagged nodes.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 8573ff9158 policy/v2: fix grant-only policies returning FilterAllowAll
compileFilterRules checked only pol.ACLs == nil to decide whether
to return FilterAllowAll (permit-any). Policies that use only Grants
(no ACLs) had nil ACLs, so the function short-circuited before
compiling any CapGrant rules. This meant cap/relay, cap/drive, and
any other App-based grant capabilities were silently ignored.

Check both ACLs and Grants are empty before returning FilterAllowAll.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby a739862c65 integration: add via grant route steering tests
Add integration tests validating that via grants correctly steer
routes to designated nodes per client group:

- TestGrantViaSubnetSteering: two routers advertise the same
  subnet, via grants steer each client group to a specific router.
  Verifies per-client route visibility, curl reachability, and
  traceroute path.

- TestGrantViaExitNodeSteering: two exit nodes, via grants steer
  each client group to a designated exit node. Verifies exit
  routes are withdrawn from non-designated nodes and the client
  rejects setting a non-designated exit node.

- TestGrantViaMixedSteering: cross-steering where subnet routes
  and exit routes go to different servers per client group.
  Verifies subnet traffic uses the subnet-designated server while
  exit traffic uses the exit-designated server.

Also add autogroupp helper for constructing AutoGroup aliases in
grant policy configurations.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 8358017dcf policy/v2,state,mapper: implement per-viewer via route steering
Via grants steer routes to specific nodes per viewer. Until now,
all clients saw the same routes for each peer because route
assembly was viewer-independent. This implements per-viewer route
visibility so that via-designated peers serve routes only to
matching viewers, while non-designated peers have those routes
withdrawn.

Add ViaRouteResult type (Include/Exclude prefix lists) and
ViaRoutesForPeer to the PolicyManager interface. The v2
implementation iterates via grants, resolves sources against the
viewer, matches destinations against the peer's advertised routes
(both subnet and exit), and categorizes prefixes by whether the
peer has the via tag.

Add RoutesForPeer to State which composes global primary election,
via Include/Exclude filtering, exit routes, and ACL reduction.
When no via grants exist, it falls back to existing behavior.

Update the mapper to call RoutesForPeer per-peer instead of using
a single route function for all peers. The route function now
returns all routes (subnet + exit), and TailNode filters exit
routes out of the PrimaryRoutes field for HA tracking.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 28be15f8ea policy/v2: handle autogroup:internet in via grant compilation
compileViaGrant only handled *Prefix destinations, skipping
*AutoGroup entirely. This meant via grants with
dst=[autogroup:internet] produced no filter rules even when the
node was an exit node with approved exit routes.

Switch the destination loop from a type assertion to a type switch
that handles both *Prefix (subnet routes) and *AutoGroup (exit
routes via autogroup:internet). Also check ExitRoutes() in
addition to SubnetRoutes() so the function doesn't bail early
when a node only has exit routes.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 687cf0882f policy/v2: implement autogroup:danger-all support
Add autogroup:danger-all as a valid source alias that matches ALL IP
addresses including non-Tailscale addresses. When used as a source,
it resolves to 0.0.0.0/0 + ::/0 internally but produces SrcIPs: ["*"]
in filter rules. When used as a destination, it is rejected with an
error matching Tailscale SaaS behavior.

Key changes:
- Add AutoGroupDangerAll constant and validation
- Add sourcesHaveDangerAll() helper and hasDangerAll parameter to
  srcIPsWithRoutes() across all compilation paths
- Add ErrAutogroupDangerAllDst for destination rejection
- Remove 3 AUTOGROUP_DANGER_ALL skip entries (K6, K7, K8)

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 4f040dead2 policy/v2: implement grant validation rules matching Tailscale SaaS
Implement comprehensive grant validation including: accept empty sources/destinations (they produce no rules), validate grant ip/app field requirements, capability name format, autogroup constraints, via tag existence, and default route CIDR restrictions.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 54db47badc policy/v2: implement via route compilation for grants
Compile grants with "via" field into FilterRules that are placed only
on nodes matching the via tag and actually advertising the destination
subnets. Key behavior:

- Filter rules go exclusively to via-nodes with matching approved routes
- Destination subnets not advertised by the via node are silently dropped
- App-only via grants (no ip field) produce no packet filter rules
- Via grants are skipped in the global compileFilterRules since they
  are node-specific

Reduces grant compat test skips from 41 to 30 (11 newly passing).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 0e3acdd8ec policy/v2: implement CapGrant compilation with companion capabilities
Compile grant app fields into CapGrant FilterRules matching Tailscale
SaaS behavior. Key changes:

- Generate CapGrant rules in compileFilterRules and
  compileGrantWithAutogroupSelf, with node-specific /32 and /128
  Dsts for autogroup:self grants
- Add reversed companion rules for drive→drive-sharer and
  relay→relay-target capabilities, ordered by original cap name
- Narrow broad CapGrant Dsts to node-specific prefixes in
  ReduceFilterRules via new reduceCapGrantRule helper
- Skip merging CapGrant rules in mergeFilterRules to preserve
  per-capability structure
- Remove ip+app mutual exclusivity validation (Tailscale accepts both)
- Add semantic JSON comparison for RawMessage types and netip.Prefix
  comparators in test infrastructure

Reduces grant compat test skips from 99 to 41 (58 newly passing).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby ebe0f4078d policy/v2: preserve non-wildcard source IPs alongside wildcard ranges
When an ACL source list contains a wildcard (*) alongside explicit
sources (tags, groups, hosts, etc.), Tailscale preserves the individual
IPs from non-wildcard sources in SrcIPs alongside the merged wildcard
CGNAT ranges. Previously, headscale's IPSetBuilder would merge all
sources into a single set, absorbing the explicit IPs into the wildcard
range.

Track non-wildcard resolved addresses separately during source
resolution, then append their individual IP strings to the output
when a wildcard is also present. This fixes the remaining 5 ACL
compat test failures (K01 and M06 subtests).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby dda35847b0 policy/v2: reorder ACL self grants to match Tailscale rule ordering
When an ACL has non-autogroup destinations (groups, users, tags, hosts)
alongside autogroup:self, emit non-self grants before self grants to
match Tailscale's filter rule ordering. ACLs with only autogroup
destinations (self + member) preserve the policy-defined order.

This fixes ACL-A17, ACL-SF07, and ACL-SF11 compat test failures.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby f95b254ea9 policy/v2: exclude exit routes from ReduceFilterRules
Add exit route check in ReduceFilterRules to prevent exit nodes from receiving packet filter rules for destinations that only overlap via exit routes. Remove resolved SUBNET_ROUTE_FILTER_RULES grant skip entries and update error message formatting for grant validation.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby e05f45cfb1 policy/v2: use approved node routes in wildcard SrcIPs
Per Tailscale documentation, the wildcard (*) source includes "any
approved subnets" — the actually-advertised-and-approved routes from
nodes, not the autoApprover policy prefixes.

Change Asterix.resolve() to return just the base CGNAT+ULA set, and
add approved subnet routes as separate SrcIPs entries in the filter
compilation path. This preserves individual route prefixes that would
otherwise be merged by IPSet (e.g., 10.0.0.0/8 absorbing 10.33.0.0/16).

Also swap rule ordering in compileGrantWithAutogroupSelf() to emit
non-self destination rules before autogroup:self rules, matching the
Tailscale FilterRule wire format ordering.

Remove the unused AutoApproverPolicy.prefixes() method.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 995ed0187c policy/v2: add advertised routes to compat test topologies
Add routable_ips and approved_routes fields to the node topology
definitions in all golden test files. These represent the subnet
routes actually advertised by nodes on the Tailscale SaaS network
during data capture:

  Routes topology (92 files, 6 router nodes):
    big-router:     10.0.0.0/8
    subnet-router:  10.33.0.0/16
    ha-router1:     192.168.1.0/24
    ha-router2:     192.168.1.0/24
    multi-router:   172.16.0.0/24
    exit-node:      0.0.0.0/0, ::/0

  ACL topology (199 files, 1 router node):
    subnet-router:  10.33.0.0/16

  Grants topology (203 files, 1 router node):
    subnet-router:  10.33.0.0/16

The route assignments were deduced from the golden data by analyzing
which router nodes receive FilterRules for which destination CIDRs
across all test files, and cross-referenced with the MTS setup
script (setup_grant_nodes.sh).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 927ce418d2 policy/v2: use bare IPs in autogroup:self DstPorts
Use ip.String() instead of netip.PrefixFrom(ip, ip.BitLen()).String()
when building DstPorts for autogroup:self destinations. This produces
bare IPs like "100.90.199.68" instead of CIDR notation like
"100.90.199.68/32", matching the Tailscale FilterRule wire format.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 93d79d8da9 policy: include IPv6 in identity-based alias resolution
AppendToIPSet now adds both IPv4 and IPv6 addresses for nodes, matching Tailscale's FilterRule wire format where identity-based aliases (tags, users, groups, autogroups) resolve to both address families. Update ReduceFilterRules test expectations to include IPv6 entries.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 500442c8f1 policy/v2: convert routes compat tests to data-driven format with Tailscale SaaS captures
Replace 8,286 lines of inline Go test expectations with 92 JSON golden files captured from Tailscale SaaS. The data-driven test driver validates route filtering, auto-approval, HA routing, and exit node behavior against real Tailscale output.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 2fb71690e8 policy/v2: convert ACL compat tests to data-driven format with Tailscale SaaS captures
Replace 9,937 lines of inline Go test expectations with 215 JSON golden files captured from Tailscale SaaS. The new data-driven test driver compares headscale's filter compilation output against real Tailscale behavior for each node in an 8-node topology.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 9f7aa55689 policy/v2: refactor alias resolution to use ResolvedAddresses
Introduce ResolvedAddresses type for structured IP set results. Refactor all Alias.Resolve() methods to return ResolvedAddresses instead of raw IPSets. Restrict identity-based aliases to matching address families, fix nil dereferences in partial resolution paths, and update test expectations for the new IP format (bare IPs, IP ranges instead of CIDR prefixes).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 0fa9dcaff8 policy/v2: add data-driven grants compatibility test with Tailscale SaaS captures
Rename tailscale_compat_test.go to tailscale_acl_compat_test.go to make room for the grants compat test. Add 237 GRANT-*.json golden test files captured from Tailscale SaaS and a data-driven test driver that compares headscale's grant filter compilation against real Tailscale behavior.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby f74ea5b8ed hscontrol/policy/v2: add Grant policy format support
Add support for the Grant policy format as an alternative to ACL format,
following Tailscale's policy v2 specification. Grants provide a more
structured way to define network access rules with explicit separation
of IP-based and capability-based permissions.

Key changes:

- Add Grant struct with Sources, Destinations, InternetProtocols (ip),
  and App (capabilities) fields
- Add ProtocolPort type for unmarshaling protocol:port strings
- Add Grant validation in Policy.validate() to enforce:
  - Mutual exclusivity of ip and app fields
  - Required ip or app field presence
  - Non-empty sources and destinations
- Refactor compileFilterRules to support both ACLs and Grants
- Convert ACLs to Grants internally via aclToGrants() for unified
  processing
- Extract destinationsToNetPortRange() helper for cleaner code
- Rename parseProtocol() to toIANAProtocolNumbers() for clarity
- Add ProtocolNumberToName mapping for reverse lookups

The Grant format allows policies to be written using either the legacy
ACL format or the new Grant format. ACLs are converted to Grants
internally, ensuring backward compatibility while enabling the new
format's benefits.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 53b8a81d48 servertest: support tagged pre-auth keys in test clients
WithTags was defined but never passed through to CreatePreAuthKey.
Fix NewClient to use CreateTaggedPreAuthKey when tags are specified,
enabling tests that need tagged nodes (e.g. via grant steering).

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby 15c1cfd778 types: include ExitRoutes in HasNetworkChanges
When exit routes are approved, SubnetRoutes remains empty because exit
routes (0.0.0.0/0, ::/0) are classified separately. Without checking
ExitRoutes, the PolicyManager cache is not invalidated on exit route
approval, causing stale filter rules that lack via grant entries for
autogroup:internet destinations.

Updates #2180
2026-04-01 14:10:42 +01:00
Kristoffer Dalby a76b4bd46c ci: switch integration tests to ARM runners
Switch all integration test jobs (build, build-postgres, test
template) from ubuntu-latest (x86_64) to ubuntu-24.04-arm (aarch64).

ARM runners on GitHub Actions are free for public repos and tend
to have more consistent performance characteristics than the
shared x86_64 pool. This should reduce flakiness caused by
resource contention on congested runners.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby a9a2001ae7 integration: scale remaining hardcoded timeouts and replace pingAllHelper
Apply CI-aware scaling to all remaining hardcoded timeouts:

- requireAllClientsOfflineStaged: scale the three internal stage
  timeouts (15s/20s/60s) with ScaledTimeout.
- validateReloginComplete: scale requireAllClientsOnline (120s)
  and requireAllClientsNetInfoAndDERP (3min) calls.
- WaitForTailscaleSyncPerUser callers in acl_test.go (3 sites, 60s).
- WaitForRunning callers in tags_test.go (10 sites): switch to
  PeerSyncTimeout() to match convention.
- WaitForRunning/WaitForPeers direct callers in route_test.go.
- requireAllClientsOnline callers in general_test.go and
  auth_key_test.go.

Replace pingAllHelper with assertPingAll/assertPingAllWithCollect:

- Wraps pings in EventuallyWithT so transient docker exec timeouts
  are retried instead of immediately failing the test.
- Timeout scales with the ping matrix size (2s per ping budget for
  2 full sweeps) so large tests get proportionally more time.
- Uses CollectT correctly, fixing the broken EventuallyWithT usage
  in TestEphemeral where the old t.Errorf bypassed CollectT.
- Follows the established assert*/assertWithCollect naming.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby acb8cfc7ee integration: make docker execute and ping timeouts CI-aware
The default docker execute timeout (10s) is the root cause of
"dockertest command timed out" errors across many integration tests
on CI. On congested GitHub Actions runners, docker exec latency
alone can consume 2-5 seconds of this budget before the command
even starts inside the container.

Replace the hardcoded 10s constant with a function that returns
20s on CI, doubling the budget for all container commands
(tailscale status, headscale CLI, curl, etc.).

Similarly, scale the default tailscale ping timeout from 200ms to
400ms on CI. This doubles the per-attempt budget and the docker
exec timeout for pings (from 200ms*5=1s to 400ms*5=2s), giving
more headroom for docker exec overhead.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby f1e5f1346d integration/acl: add tag verification step to TestACLTagPropagationPortSpecific
TestACLTagPropagationPortSpecific failed twice on CI because it jumped
from SetNodeTags directly to checking curl, without first verifying the
tag change was applied on the server. This races against server-side
processing.

Add a tag verification step (matching TestACLTagPropagation's pattern)
and bump the Step 4 timeout from 60s to 90s since port-specific filter
changes require both endpoints to process the new PacketFilter from
the MapResponse while the WireGuard tunnel stays up.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby 210f58f62e integration: use CI-scaled timeouts for all EventuallyWithT assertions
Wrap all 329 hardcoded EventuallyWithT timeouts across 12 test files
with integrationutil.ScaledTimeout(), which applies a 2x multiplier
on CI runners. This addresses the systemic issue where hardcoded
timeouts that work locally are insufficient under CI resource
contention.

Variable-based timeouts (propagationTime, assertTimeout in
route_test.go and totalWaitTime in auth_oidc_test.go) are wrapped
at their definition site so all downstream usages benefit.

The retry intervals (second duration parameter) are intentionally
NOT scaled, as they control polling frequency, not total wait time.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby a147b0cd87 integration/acl: use CurlFailFast for all negative curl assertions
Replace Curl() with CurlFailFast() in all negative curl assertions
(where the test expects the connection to fail). CurlFailFast uses
1 retry and 2s max time instead of 3 retries and 5s max, which
avoids wasting time on unnecessary retries when we expect the
connection to be blocked.

This affects 21 call sites across 7 test functions:

- TestACLAllowUser80Dst
- TestACLDenyAllPort80
- TestACLAllowUserDst
- TestACLAllowStarDst
- TestACLNamedHostsCanReach
- TestACLDevice1CanAccessDevice2
- TestPolicyUpdateWhileRunningWithCLIInDatabase
- TestACLAutogroupSelf
- TestACLPolicyPropagationOverTime

Where possible, the inline Curl+Error pattern is replaced with the
assertCurlFailWithCollect helper introduced in the previous commit.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby a7edcf3b0f integration: add CI-scaled timeouts and curl helpers for flaky ACL tests
Add ScaledTimeout to scale EventuallyWithT timeouts by 2x on CI,
consistent with the existing PeerSyncTimeout (60s/120s) and
dockertestMaxWait (300s/600s) conventions.

Add assertCurlSuccessWithCollect and assertCurlFailWithCollect helpers
following the existing *WithCollect naming convention.
assertCurlFailWithCollect uses CurlFailFast internally for aggressive
timeouts, avoiding wasted retries when expecting blocked connections.

Apply these to the three flakiest ACL tests:

- TestACLTagPropagation: swap NetMap and curl verification order so
  the fast NetMap check (confirms MapResponse arrived) runs before
  the slower curl check. Use curl helpers and scaled timeouts.

- TestACLTagPropagationPortSpecific: use curl helpers and scaled
  timeouts.

- TestACLHostsInNetMapTable: scale the 10s EventuallyWithT timeout.

Updates #3125
2026-03-31 22:06:25 +02:00
Kristoffer Dalby fda72ad1a3 Update main.md
Co-authored-by: nblock <nblock@users.noreply.github.com>
2026-03-31 13:36:31 +02:00
Kristoffer Dalby dfaf120f2a docs: add development builds install page
Move the container image and binary download details from the README
into a dedicated documentation page at setup/install/main. This gives
development builds a proper home in the docs site alongside the other
install methods. The README now links to the docs page instead.
2026-03-31 13:36:31 +02:00
Kristoffer Dalby e171d30179 ci: add build workflow for main branch
Build and push multi-arch container images (linux/amd64, linux/arm64)
to GHCR and Docker Hub on every push to main that changes Go or Nix
files. Images are tagged as main-<short-sha> using ko with the same
distroless base image as release builds.

Cross-compiled binaries for linux and darwin (amd64, arm64) are
uploaded as workflow artifacts. The README links to these via
nightly.link for stable download URLs.
2026-03-31 13:36:31 +02:00
Kristoffer Dalby 0c6b9f5348 goreleaser: remove unused ts2019 build tag
The ts2019 build tag is no longer used. Remove it from the
goreleaser build configuration.
2026-03-31 13:36:31 +02:00
Florian Preinstorfer f3512d50df Switch to mkdocs-materialx
The project mkdocs-material is in maintenance-only mode and their
successor is not ready yet.

Use the modern, refreshed theme and drop the pymdownx.magiclink
extension.
2026-03-25 22:30:03 +01:00
Florian Preinstorfer efd83da14e Explicitly mention that a headscale username should *not* end with @
See: #3149
2026-03-20 19:44:33 +01:00
Tanayk07 568baf3d02 fix: align banner right-side border to consistent 64-char width 2026-03-19 07:08:35 +01:00
Tanayk07 5105033224 feat: add prominent warning banner for non-standard IP prefixes
Add a highly visible ASCII-art warning banner that is printed at
startup when the configured IP prefixes fall outside the standard
Tailscale CGNAT (100.64.0.0/10) or ULA (fd7a:115c:a1e0::/48) ranges.

The warning fires once even if both v4 and v6 are non-standard, and
the warnBanner() function is reusable for other critical configuration
warnings in the future.

Also updates config-example.yaml to clarify that subsets of the
default ranges are fine, but ranges outside CGNAT/ULA are not.

Closes #3055
2026-03-19 07:08:35 +01:00
Kristoffer Dalby 3d53f97c82 hscontrol/servertest: fix test expectations for eventual consistency
Three corrections to issue tests that had wrong assumptions about
when data becomes available:

1. initial_map_should_include_peer_online_status: use WaitForCondition
   instead of checking the initial netmap. Online status is set by
   Connect() which sends a PeerChange patch after the initial
   RegisterResponse, so it may not be present immediately.

2. disco_key_should_propagate_to_peers: use WaitForCondition. The
   DiscoKey is sent in the first MapRequest (not RegisterRequest),
   so peers may not see it until a subsequent map update.

3. approved_route_without_announcement: invert the test expectation.
   Tailscale uses a strict advertise-then-approve model -- routes are
   only distributed when the node advertises them (Hostinfo.RoutableIPs)
   AND they are approved. An approval without advertisement is a dormant
   pre-approval. The test now asserts the route does NOT appear in
   AllowedIPs, matching upstream Tailscale semantics.

Also fix TestClient.Reconnect to clear the cached netmap and drain
pending updates before re-registering. Without this, WaitForPeers
returned immediately based on the old session's stale data.
2026-03-19 07:05:58 +01:00
Kristoffer Dalby 1053fbb16b hscontrol/state: fix online status reset during re-registration
Two fixes to how online status is handled during registration:

1. Re-registration (applyAuthNodeUpdate, HandleNodeFromPreAuthKey) no
   longer resets IsOnline to false. Online status is managed exclusively
   by Connect()/Disconnect() in the poll session lifecycle. The reset
   caused a false offline blip: the auth handler's change notification
   triggered a map regeneration showing the node as offline to peers,
   even though Connect() would set it back to true moments later.

2. New node creation (createAndSaveNewNode) now explicitly sets
   IsOnline=false instead of leaving it nil. This ensures peers always
   receive a known online status rather than an ambiguous nil/unknown.
2026-03-19 07:05:58 +01:00
Kristoffer Dalby b09af3846b hscontrol/poll,state: fix grace period disconnect TOCTOU race
When a node disconnects, serveLongPoll defers a cleanup that starts a
grace period goroutine. This goroutine polls batcher.IsConnected() and,
if the node has not reconnected within ~10 seconds, calls
state.Disconnect() to mark it offline. A TOCTOU race exists: the node
can reconnect (calling Connect()) between the IsConnected check and
the Disconnect() call, causing the stale Disconnect() to overwrite
the new session's online status.

Fix with a monotonic per-node generation counter:

- State.Connect() increments the counter and returns the current
  generation alongside the change list.
- State.Disconnect() accepts the generation from the caller and
  rejects the call if a newer generation exists, making stale
  disconnects from old sessions a no-op.
- serveLongPoll captures the generation at Connect() time and passes
  it to Disconnect() in the deferred cleanup.
- RemoveNode's return value is now checked: if another session already
  owns the batcher slot (reconnect happened), the old session skips
  the grace period entirely.

Update batcher_test.go to track per-node connect generations and
pass them through to Disconnect(), matching production behavior.

Fixes the following test failures:
- server_state_online_after_reconnect_within_grace
- update_history_no_false_offline
- nodestore_correct_after_rapid_reconnect
- rapid_reconnect_peer_never_sees_offline
2026-03-19 07:05:58 +01:00
Kristoffer Dalby 00c41b6422 hscontrol/servertest: add race, stress, and poll race tests
Add three test files designed to stress the control plane under
concurrent and adversarial conditions:

- race_test.go: 14 tests exercising concurrent mutations, session
  replacement, batcher contention, NodeStore access, and map response
  delivery during disconnect. All pass the Go race detector.

- poll_race_test.go: 8 tests targeting the poll.go grace period
  interleaving. These confirm a logical TOCTOU race: when a node
  disconnects and reconnects within the grace period, the old
  session's deferred Disconnect() can overwrite the new session's
  Connect(), leaving IsOnline=false despite an active poll session.

- stress_test.go: sustained churn, rapid mutations, rolling
  replacement, data integrity checks under load, and verification
  that rapid reconnects do not leak false-offline notifications.

Known failing tests (grace period TOCTOU race):
- server_state_online_after_reconnect_within_grace
- update_history_no_false_offline
- rapid_reconnect_peer_never_sees_offline
2026-03-19 07:05:58 +01:00
Kristoffer Dalby ab4e205ce7 hscontrol/servertest: expand issue tests to 24 scenarios, surface 4 issues
Split TestIssues into 7 focused test functions to stay under cyclomatic
complexity limits while testing more aggressively.

Issues surfaced (4 failing tests):

1. initial_map_should_include_peer_online_status: Initial MapResponse
   has Online=nil for peers. Online status only arrives later via
   PeersChangedPatch.

2. disco_key_should_propagate_to_peers: DiscoPublicKey set by client
   is not visible to peers. Peers see zero disco key.

3. approved_route_without_announcement_is_visible: Server-side route
   approval without client-side announcement silently produces empty
   SubnetRoutes (intersection of empty announced + approved = empty).

4. nodestore_correct_after_rapid_reconnect: After 5 rapid reconnect
   cycles, NodeStore reports node as offline despite having an active
   poll session. The connect/disconnect grace period interleaving
   leaves IsOnline in an incorrect state.

Passing tests (20) verify:
- IP uniqueness across 10 nodes
- IP stability across reconnect
- New peers have addresses immediately
- Node rename propagates to peers
- Node delete removes from all peer lists
- Hostinfo changes (OS field) propagate
- NodeStore/DB consistency after route mutations
- Grace period timing (8-20s window)
- Ephemeral node deletion (not just offline)
- 10-node simultaneous connect convergence
- Rapid sequential node additions
- Reconnect produces complete map
- Cross-user visibility with default policy
- Same-user multiple nodes get distinct IDs
- Same-hostname nodes get unique GivenNames
- Policy change during connect still converges
- DERP region references are valid
- User profiles present for self and peers
- Self-update arrives after route approval
- Route advertisement stored as AnnouncedRoutes
2026-03-19 07:05:58 +01:00
Kristoffer Dalby f87b08676d hscontrol/servertest: add policy, route, ephemeral, and content tests
Extend the servertest harness with:
- TestClient.Direct() accessor for advanced operations
- TestClient.WaitForPeerCount and WaitForCondition helpers
- TestHarness.ChangePolicy for ACL policy testing
- AssertDERPMapPresent and AssertSelfHasAddresses

New test suites:
- content_test.go: self node, DERP map, peer properties, user profiles,
  update history monotonicity, and endpoint update propagation
- policy_test.go: default allow-all, explicit policy, policy triggers
  updates on all nodes, multiple policy changes, multi-user mesh
- ephemeral_test.go: ephemeral connect, cleanup after disconnect,
  mixed ephemeral/regular, reconnect prevents cleanup
- routes_test.go: addresses in AllowedIPs, route advertise and approve,
  advertised routes via hostinfo, CGNAT range validation

Also fix node_departs test to use WaitForCondition instead of
assert.Eventually, and convert concurrent_join_and_leave to
interleaved_join_and_leave with grace-period-tolerant assertions.
2026-03-19 07:05:58 +01:00
Kristoffer Dalby ca7362e9aa hscontrol/servertest: add control plane lifecycle and consistency tests
Add three test files exercising the servertest harness:

- lifecycle_test.go: connection, disconnection, reconnection, session
  replacement, and mesh formation at various sizes.
- consistency_test.go: symmetric visibility, consistent peer state,
  address presence, concurrent join/leave convergence.
- weather_test.go: rapid reconnects, flapping stability, reconnect
  with various delays, concurrent reconnects, and scale tests.

All tests use table-driven patterns with subtests.
2026-03-19 07:05:58 +01:00
Kristoffer Dalby 0288614bdf hscontrol: add servertest harness for in-process control plane testing
Add a new hscontrol/servertest package that provides a test harness
for exercising the full Headscale control protocol in-process, using
Tailscale's controlclient.Direct as the client.

The harness consists of:
- TestServer: wraps a Headscale instance with an httptest.Server
- TestClient: wraps controlclient.Direct with NetworkMap tracking
- TestHarness: orchestrates N clients against a single server
- Assertion helpers for mesh completeness, visibility, and consistency

Export minimal accessor methods on Headscale (HTTPHandler, NoisePublicKey,
GetState, SetServerURL, StartBatcher, StartEphemeralGC) so the servertest
package can construct a working server from outside the hscontrol package.

This enables fast, deterministic tests of connection lifecycle, update
propagation, and network weather scenarios without Docker.
2026-03-19 07:05:58 +01:00
Kristoffer Dalby 82c7efccf8 mapper/batcher: serialize per-node work to prevent out-of-order delivery
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:

1. MapResponses delivered out of order — a later change could finish
   generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
   updateSentPeers does Clear() + Store() which is not atomic relative
   to a concurrent Range() in computePeerDiff.

Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.

Fixes #3140
2026-03-19 07:05:58 +01:00
Kristoffer Dalby 81b871c9b5 integration/acl: replace custom entrypoints with WithPackages
Replace inline WithDockerEntrypoint shell scripts in
TestACLTagPropagation and TestACLTagPropagationPortSpecific with
the standard WithPackages and WithWebserver options.

The custom entrypoints used fragile fixed sleeps and lacked the
robust network/cert readiness waits that buildEntrypoint provides.

Updates #3139
2026-03-16 03:57:05 -07:00
Kristoffer Dalby e5ebe3205a integration: standardize test infrastructure options
Make embedded DERP server and TLS the default configuration for all
integration tests, replacing the per-test opt-in model that led to
inconsistent and flaky test behavior.

Infrastructure changes:
- DefaultConfigEnv() includes embedded DERP server settings
- New() auto-generates a proper CA + server TLS certificate pair
- CA cert is installed into container trust stores and returned by
  GetCert() so clients and internal tools (curl) trust the server
- CreateCertificate() now returns (caCert, cert, key) instead of
  discarding the CA certificate
- Add WithPublicDERP() and WithoutTLS() opt-out options
- Remove WithTLS(), WithEmbeddedDERPServerOnly(), and WithDERPAsIP()
  since all their behavior is now the default or unnecessary

Test cleanup:
- Remove all redundant WithTLS/WithEmbeddedDERPServerOnly/WithDERPAsIP
  calls from test files
- Give every test a unique WithTestName by parameterizing aclScenario,
  sshScenario, and derpServerScenario helpers
- Add WithTestName to tests that were missing it
- Document all non-standard options with inline comments explaining
  why each is needed

Updates #3139
2026-03-16 03:57:05 -07:00
Kristoffer Dalby 87b8507ac9 mapper/batcher: replace connected map with per-node disconnectedAt
The Batcher's connected field (*xsync.Map[types.NodeID, *time.Time])
encoded three states via pointer semantics:

  - nil value:    node is connected
  - non-nil time: node disconnected at that timestamp
  - key missing:  node was never seen

This was error-prone (nil meaning 'connected' inverts Go idioms),
redundant with b.nodes + hasActiveConnections(), and required keeping
two parallel maps in sync. It also contained a bug in RemoveNode where
new(time.Now()) was used instead of &now, producing a zero time.

Replace the separate connected map with a disconnectedAt field on
multiChannelNodeConn (atomic.Pointer[time.Time]), tracked directly
on the object that already manages the node's connections.

Changes:
  - Add disconnectedAt field and helpers (markConnected, markDisconnected,
    isConnected, offlineDuration) to multiChannelNodeConn
  - Remove the connected field from Batcher
  - Simplify IsConnected from two map lookups to one
  - Simplify ConnectedMap and Debug from two-map iteration to one
  - Rewrite cleanupOfflineNodes to scan b.nodes directly
  - Remove the markDisconnectedIfNoConns helper
  - Update all tests and benchmarks

Fixes #3141
2026-03-16 02:22:56 -07:00
Kristoffer Dalby 60317064fd mapper/batcher: serialize per-node work to prevent out-of-order delivery
processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:

1. MapResponses delivered out of order — a later change could finish
   generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
   updateSentPeers does Clear() + Store() which is not atomic relative
   to a concurrent Range() in computePeerDiff.

Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.

Fixes #3140
2026-03-16 02:22:46 -07:00
Juan Font 4d427cfe2a noise: limit request body size to prevent unauthenticated OOM
The Noise handshake accepts any machine key without checking
registration, so all endpoints behind the Noise router are reachable
without credentials. Three handlers used io.ReadAll without size
limits, allowing an attacker to OOM-kill the server.

Fix:
- Add http.MaxBytesReader middleware (1 MiB) on the Noise router.
- Replace io.ReadAll + json.Unmarshal with json.NewDecoder in
  PollNetMapHandler and RegistrationHandler.
- Stop reading the body in NotImplementedHandler entirely.
2026-03-16 09:28:31 +01:00
Kristoffer Dalby afd3a6acbc mapper/batcher: remove disabled X-prefixed test functions
Remove XTestBatcherChannelClosingRace (~95 lines) and
XTestBatcherScalability (~515 lines). These were disabled by
prefixing with X (making them invisible to go test) and served
as dead code. The functionality they covered is exercised by the
active test suite.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby feaf85bfbc mapper/batcher: clean up test constants and output
L8: Rename SCREAMING_SNAKE_CASE test constants to idiomatic Go
camelCase. Remove highLoad* and extremeLoad* constants that were
only referenced by disabled (X-prefixed) tests.

L10: Fix misleading assert message that said "1337" while checking
for region ID 999.

L12: Remove emoji from test log output to avoid encoding issues
in CI environments.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 86e279869e mapper/batcher: minor production code cleanup
L1: Replace crypto/rand with an atomic counter for generating
connection IDs. These identifiers are process-local and do not need
cryptographic randomness; a monotonic counter is cheaper and
produces shorter, sortable IDs.

L5: Use getActiveConnectionCount() in Debug() instead of directly
locking the mutex and reading the connections slice. This avoids
bypassing the accessor that already exists for this purpose.

L6: Extract the hardcoded 15*time.Minute cleanup threshold into
the named constant offlineNodeCleanupThreshold.

L7: Inline the trivial addWork wrapper; AddWork now calls addToBatch
directly.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 7881f65358 mapper: extract node connection types to node_conn.go
Move connectionEntry, multiChannelNodeConn, generateConnectionID, and
all their methods from batcher.go into a dedicated file. This reduces
batcher.go from ~1170 lines to ~800 and separates per-node connection
management from batcher orchestration.

Pure move — no logic changes.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 2d549e579f mapper/batcher: add regression tests for M1, M3, M7 fixes
- TestBatcher_CloseBeforeStart_DoesNotHang: verifies Close() before
  Start() returns promptly now that done is initialized in NewBatcher.

- TestBatcher_QueueWorkAfterClose_DoesNotHang: verifies queueWork
  returns via the done channel after Close(), even without Start().

- TestIsConnected_FalseAfterAddNodeFailure: verifies IsConnected
  returns false after AddNode fails and removes the last connection.

- TestRemoveConnectionAtIndex_NilsTrailingSlot: verifies the backing
  array slot is nil-ed after removal to avoid retaining pointers.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 50e8b21471 mapper/batcher: fix pointer retention, done-channel init, and connected-map races
M7: Nil out trailing *connectionEntry pointers in the backing array
after slice removal in removeConnectionAtIndexLocked and send().
Without this, the GC cannot collect removed entries until the slice
is reallocated.

M1: Initialize the done channel in NewBatcher instead of Start().
Previously, calling Close() or queueWork before Start() would select
on a nil channel, blocking forever. Moving the make() to the
constructor ensures the channel is always usable.

M2: Move b.connected.Delete and b.totalNodes decrement inside the
Compute callback in cleanupOfflineNodes. Previously these ran after
the Compute returned, allowing a concurrent AddNode to reconnect
between the delete and the bookkeeping update, which would wipe the
fresh connected state.

M3: Call markDisconnectedIfNoConns on AddNode error paths. Previously,
when initial map generation or send timed out, the connection was
removed but b.connected retained its old nil (= connected) value,
making IsConnected return true for a node with zero connections.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 8e26651f2c mapper/batcher: add regression tests for timer leak and Close lifecycle
Add four unit tests guarding fixes introduced in recent commits:

- TestConnectionEntry_SendFastPath_TimerStopped: verifies the
  time.NewTimer fix (H1) does not leak goroutines after many
  fast-path sends on a buffered channel.

- TestBatcher_CloseWaitsForWorkers: verifies Close() blocks until all
  worker goroutines exit (H3), preventing sends on torn-down channels.

- TestBatcher_CloseThenStartIsNoop: verifies the one-shot lifecycle
  contract; Start() after Close() must not spawn new goroutines.

- TestBatcher_CloseStopsTicker: verifies Close() stops the internal
  ticker to prevent resource leaks.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 57a38b5678 mapper/batcher: reduce hot-path log verbosity
Remove Caller(), channel pointer formatting (fmt.Sprintf("%p",...)),
and mutex timing from send(), addConnection(), and
removeConnectionByChannel(). Move per-broadcast summary and
no-connection logs from Debug to Trace. Remove per-connection
"attempting"/"succeeded" logs entirely; keep Warn for failures.

These methods run on every MapResponse delivery, so the savings
compound quickly under load.

Updates #2545
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 051a38a4c4 mapper/batcher: track worker goroutines and stop ticker on Close
Close() previously closed the done channel and returned immediately,
without waiting for worker goroutines to exit. This caused goroutine
leaks in tests and allowed workers to race with connection teardown.
The ticker was also never stopped, leaking its internal goroutine.

Add a sync.WaitGroup to track the doWork goroutine and every worker
it spawns. Close() now calls wg.Wait() after signalling shutdown,
ensuring all goroutines have exited before tearing down connections.
Also stop the ticker to prevent resource leaks.

Document that a Batcher must not be reused after Close().
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 3276bda0c0 mapper/batcher: replace time.After with NewTimer to avoid timer leak
connectionEntry.send() is on the hot path: called once per connection
per broadcast tick. time.After allocates a timer that sits in the
runtime timer heap until it fires (50 ms), even when the channel send
succeeds immediately. At 1000 connected nodes, every tick leaks 1000
timers into the heap, creating continuous GC pressure.

Replace with time.NewTimer + defer timer.Stop() so the timer is
removed from the heap as soon as the fast-path send completes.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby ebc57d9a38 integration/acl: fix TestACLPolicyPropagationOverTime infrastructure
Add embedded DERP server, TLS, and netfilter=off to match the
infrastructure configuration used by all other ACL integration tests.

Without these options, the test fails intermittently because traffic
routes through external DERP relays and iptables initialization fails
in Docker containers.

Updates #3139
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 2058343ad6 mapper: remove Batcher interface, rename to Batcher struct
Remove the Batcher interface since there is only one implementation.
Rename LockFreeBatcher to Batcher and merge batcher_lockfree.go into
batcher.go.

Drop type assertions in debug.go now that mapBatcher is a concrete
*mapper.Batcher pointer.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 9b24a39943 mapper/batcher: add scale benchmarks
Add benchmarks that systematically test node counts from 100 to
50,000 to identify scaling limits and validate performance under
load.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 3ebe4d99c1 mapper/batcher: reduce lock contention with two-phase send
Rewrite multiChannelNodeConn.send() to use a two-phase approach:
1. RLock: snapshot connections slice (cheap pointer copy)
2. Unlock: send to all connections (50ms timeouts happen here)
3. Lock: remove failed connections by pointer identity

Previously, send() held the write lock for the entire duration of
sending to all connections. With N stale connections each timing out
at 50ms, this blocked addConnection/removeConnection for N*50ms.
The two-phase approach holds the lock only for O(N) pointer
operations, not for N*50ms I/O waits.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby da33795e79 mapper/batcher: fix race conditions in cleanup and lookups
Replace the two-phase Load-check-Delete in cleanupOfflineNodes with
xsync.Map.Compute() for atomic check-and-delete. This prevents the
TOCTOU race where a node reconnects between the hasActiveConnections
check and the Delete call.

Add nil guards on all b.nodes.Load() and b.nodes.Range() call sites
to prevent nil pointer panics from concurrent cleanup races.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 57070680a5 mapper/batcher: restructure internals for correctness
Move per-node pending changes from a shared xsync.Map on the batcher
into multiChannelNodeConn, protected by a dedicated mutex. The new
appendPending/drainPending methods provide atomic append and drain
operations, eliminating data races in addToBatch and
processBatchedChanges.

Add sync.Once to multiChannelNodeConn.close() to make it idempotent,
preventing panics from concurrent close calls on the same channel.

Add started atomic.Bool to guard Start() against being called
multiple times, preventing orphaned goroutines.

Add comprehensive concurrency tests validating these changes.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 21e02e5d1f mapper/batcher: add unit tests and benchmarks
Add comprehensive unit tests for the LockFreeBatcher covering
AddNode/RemoveNode lifecycle, addToBatch routing (broadcast, targeted,
full update), processBatchedChanges deduplication, cleanup of offline
nodes, close/shutdown behavior, IsConnected state tracking, and
connected map consistency.

Add benchmarks for connection entry send, multi-channel send and
broadcast, peer diff computation, sentPeers updates, addToBatch at
various scales (10/100/1000 nodes), processBatchedChanges, broadcast
delivery, IsConnected lookups, connected map enumeration, connection
churn, and concurrent send+churn scenarios.

Widen setupBatcherWithTestData to accept testing.TB so benchmarks can
reuse the same database-backed test setup as unit tests.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 2f94b80e70 go.mod: add stress tool dependency
Add golang.org/x/tools/cmd/stress as a tool dependency for running
tests under repeated stress to surface flaky failures.

Update flake vendorHash for the new go.mod dependencies.
2026-03-14 02:52:28 -07:00
Kristoffer Dalby 3e0a96ec3a all: fix test flakiness and improve test infrastructure
Buffer the AuthRequest verdict channel to prevent a race where the
sender blocks indefinitely if the receiver has already timed out, and
increase the auth followup test timeout from 100ms to 5s to prevent
spurious failures under load.

Skip postgres-backed tests when the postgres server is unavailable
instead of calling t.Fatal, which was preventing the rest of the test
suite from running.

Add TestMain to db, types, and policy/v2 packages to chdir to the
source directory before running tests. This ensures relative testdata/
paths resolve correctly when the test binary is executed from an
arbitrary working directory (e.g., via "go tool stress").
2026-03-14 02:52:28 -07:00
773 changed files with 8586429 additions and 24972 deletions
+112
View File
@@ -0,0 +1,112 @@
---
name: Build (main)
on:
push:
branches:
- main
paths:
- "*.nix"
- "go.*"
- "**/*.go"
- ".github/workflows/container-main.yml"
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.sha }}
cancel-in-progress: true
jobs:
container:
if: github.repository == 'juanfont/headscale'
runs-on: ubuntu-latest
permissions:
packages: write
contents: read
steps:
- name: Checkout
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- name: Login to DockerHub
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GHCR
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef # v3.6.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
- uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
with:
primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
'**/flake.lock') }}
restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
- name: Set commit timestamp
run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> $GITHUB_ENV
- name: Build and push to GHCR
env:
KO_DOCKER_REPO: ghcr.io/juanfont/headscale
KO_DEFAULTBASEIMAGE: gcr.io/distroless/base-debian13
CGO_ENABLED: "0"
run: |
nix develop --command -- ko build \
--bare \
--platform=linux/amd64,linux/arm64 \
--tags=main-${GITHUB_SHA::7} \
./cmd/headscale
- name: Push to Docker Hub
env:
KO_DOCKER_REPO: headscale/headscale
KO_DEFAULTBASEIMAGE: gcr.io/distroless/base-debian13
CGO_ENABLED: "0"
run: |
nix develop --command -- ko build \
--bare \
--platform=linux/amd64,linux/arm64 \
--tags=main-${GITHUB_SHA::7} \
./cmd/headscale
binaries:
if: github.repository == 'juanfont/headscale'
runs-on: ubuntu-latest
strategy:
matrix:
include:
- goos: linux
goarch: amd64
- goos: linux
goarch: arm64
- goos: darwin
goarch: amd64
- goos: darwin
goarch: arm64
steps:
- name: Checkout
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
- uses: nixbuild/nix-quick-install-action@2c9db80fb984ceb1bcaa77cdda3fdf8cfba92035 # v34
- uses: nix-community/cache-nix-action@135667ec418502fa5a3598af6fb9eb733888ce6a # v6.1.3
with:
primary-key: nix-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('**/*.nix',
'**/flake.lock') }}
restore-prefixes-first-match: nix-${{ runner.os }}-${{ runner.arch }}
- name: Build binary
env:
CGO_ENABLED: "0"
GOOS: ${{ matrix.goos }}
GOARCH: ${{ matrix.goarch }}
run: nix develop --command -- go build -o headscale ./cmd/headscale
- uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
with:
name: headscale-${{ matrix.goos }}-${{ matrix.goarch }}
path: headscale
@@ -16,7 +16,7 @@ on:
jobs:
test:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04-arm
env:
# Github does not allow us to access secrets in pull requests,
# so this env var is used to check if we have the secret or not.
+5 -2
View File
@@ -12,7 +12,7 @@ jobs:
# sqlite: Runs all integration tests with SQLite backend.
# postgres: Runs a subset of tests with PostgreSQL to verify database compatibility.
build:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04-arm
outputs:
files-changed: ${{ steps.changed-files.outputs.files }}
steps:
@@ -119,7 +119,7 @@ jobs:
path: tailscale-head-image.tar.gz
retention-days: 10
build-postgres:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04-arm
needs: build
if: needs.build.outputs.files-changed == 'true'
steps:
@@ -233,6 +233,8 @@ jobs:
- TestNodeOnlineStatus
- TestPingAllByIPManyUpDown
- Test2118DeletingOnlineNodePanics
- TestGrantCapRelay
- TestGrantCapDrive
- TestEnablingRoutes
- TestHASubnetRouterFailover
- TestSubnetRouteACL
@@ -246,6 +248,7 @@ jobs:
- TestAutoApproveMultiNetwork/webauth-user.*
- TestAutoApproveMultiNetwork/webauth-group.*
- TestSubnetRouteACLFiltering
- TestGrantViaSubnetSteering
- TestHeadscale
- TestTailscaleNodesJoiningHeadcale
- TestSSHOneUserToAll
+1
View File
@@ -29,6 +29,7 @@ config*.yaml
!config-example.yaml
derp.yaml
*.hujson
!hscontrol/policy/v2/testdata/*/*.hujson
*.key
/db.sqlite
*.sqlite3
-2
View File
@@ -27,8 +27,6 @@ builds:
- linux_arm64
flags:
- -mod=readonly
tags:
- ts2019
archives:
- id: golang-cross
+2
View File
@@ -13,6 +13,8 @@ repos:
rev: v6.0.0
hooks:
- id: check-added-large-files
args: [--maxkb=2000]
exclude: ^hscontrol/policy/v2/testdata/
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: check-json
+31
View File
@@ -447,6 +447,37 @@ All test runs save comprehensive debugging artifacts to `control_logs/TIMESTAMP-
## Testing Guidelines
### Test Log Output
By default, `go test ./...` runs with zerolog set to `ErrorLevel`. This is configured centrally by an `init()` in `hscontrol/types/testlog.go`, which uses `testing.Testing()` to detect test mode and only applies to test binaries — production binaries are unaffected. Because `hscontrol/types` is transitively imported by every package that emits zerolog output, this single insertion point silences all `go test` runs without any per-package boilerplate.
To re-enable verbose logs while debugging a specific test, set the `HEADSCALE_TEST_LOG_LEVEL` environment variable to any zerolog level (`trace`, `debug`, `info`, `warn`, `error`, `fatal`, `panic`, `disabled`):
```bash
# Show all zerolog output for one package
HEADSCALE_TEST_LOG_LEVEL=trace go test -v ./hscontrol/mapper/...
# See debug and above for a single test
HEADSCALE_TEST_LOG_LEVEL=debug go test -v ./hscontrol/db/... -run TestNodeRegistration
# Make CI fully silent (suppress even errors)
HEADSCALE_TEST_LOG_LEVEL=disabled go test ./...
```
**Do not add per-test `zerolog.SetGlobalLevel` calls.** The central mechanism handles the default. If a specific test genuinely needs a different level (e.g., a benchmark that wants to assert on log output, or a test that needs to capture errors), use the save/restore idiom so subsequent tests in the same binary are not affected:
```go
originalLevel := zerolog.GlobalLevel()
defer zerolog.SetGlobalLevel(originalLevel)
zerolog.SetGlobalLevel(zerolog.DebugLevel)
```
**Pitfalls**:
- `log.Fatal()` still calls `os.Exit(1)` and `log.Panic()` still panics regardless of level — only the rendered message is suppressed.
- Local buffer loggers created via `zerolog.New(&buf)` are also gated by the global level. Tests that assert on log output (currently only `hscontrol/util/zlog`) re-enable trace level via their own `init_test.go`.
- Avoid the broken `defer zerolog.SetGlobalLevel(zerolog.DebugLevel)` pattern. It restores to `DebugLevel` instead of the prior level, polluting global state for subsequent tests in the same binary.
### Integration Test Patterns
#### **CRITICAL: EventuallyWithT Pattern for External Calls**
+28 -1
View File
@@ -25,11 +25,26 @@ A new `headscale auth` CLI command group supports the approval flow:
[#1850](https://github.com/juanfont/headscale/pull/1850)
### Grants
We now support [Tailscale grants](https://tailscale.com/kb/1324/grants) alongside ACLs. Grants
extend what you can express in a policy beyond packet filtering: the `app` field controls
application-level features like Taildrive file sharing and peer relay, and the `via` field steers
traffic through specific tagged subnet routers or exit nodes. The `ip` field works like an ACL rule.
Grants can be mixed with ACLs in the same policy file.
[#2180](https://github.com/juanfont/headscale/pull/2180)
As part of this, we added `autogroup:danger-all`. It resolves to `0.0.0.0/0` and `::/0` — all IP
addresses, including those outside the tailnet. This replaces the old behaviour where `*` matched
all IPs (see BREAKING below). The name is intentionally scary: accepting traffic from the entire
internet is a security-sensitive choice. `autogroup:danger-all` can only be used as a source.
### BREAKING
- **ACL Policy**: Wildcard (`*`) in ACL sources and destinations now resolves to Tailscale's CGNAT range (`100.64.0.0/10`) and ULA range (`fd7a:115c:a1e0::/48`) instead of all IPs (`0.0.0.0/0` and `::/0`) [#3036](https://github.com/juanfont/headscale/pull/3036)
- This better matches Tailscale's security model where `*` means "any node in the tailnet" rather than "any IP address"
- Policies relying on wildcard to match non-Tailscale IPs will need to use explicit CIDR ranges instead
- Policies that need to match all IP addresses including non-Tailscale IPs should use `autogroup:danger-all` as a source, or explicit CIDR ranges as destinations [#2180](https://github.com/juanfont/headscale/pull/2180)
- `autogroup:danger-all` can only be used as a source; it cannot be used as a destination
- **Note**: Users with non-standard IP ranges configured in `prefixes.ipv4` or `prefixes.ipv6` (which is unsupported and produces a warning) will need to explicitly specify their CIDR ranges in ACL rules instead of using `*`
- **ACL Policy**: Validate autogroup:self source restrictions matching Tailscale behavior - tags, hosts, and IPs are rejected as sources for autogroup:self destinations [#3036](https://github.com/juanfont/headscale/pull/3036)
- Policies using tags, hosts, or IP addresses as sources for autogroup:self destinations will now fail validation
@@ -58,6 +73,18 @@ A new `headscale auth` CLI command group supports the approval flow:
- Deprecate `headscale nodes register --key` in favour of `headscale auth register --auth-id` [#1850](https://github.com/juanfont/headscale/pull/1850)
- Generalise auth templates into reusable `AuthSuccess` and `AuthWeb` components [#1850](https://github.com/juanfont/headscale/pull/1850)
- Unify auth pipeline with `AuthVerdict` type, supporting registration, reauthentication, and SSH checks [#1850](https://github.com/juanfont/headscale/pull/1850)
- Add support for policy grants with `ip`, `app`, and `via` fields [#2180](https://github.com/juanfont/headscale/pull/2180)
- Add `autogroup:danger-all` as a source-only autogroup resolving to all IP addresses [#2180](https://github.com/juanfont/headscale/pull/2180)
- Add capability grants for Taildrive (`cap/drive`) and peer relay (`cap/relay`) with automatic companion capabilities [#2180](https://github.com/juanfont/headscale/pull/2180)
- Add per-viewer via route steering — grants with `via` tags control which subnet router or exit node handles traffic for each group of viewers [#2180](https://github.com/juanfont/headscale/pull/2180)
- Enable Taildrive node attributes on all nodes; actual access is controlled by `cap/drive` grants [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix exit nodes incorrectly receiving filter rules for destinations that only overlap via exit routes [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix address-based aliases (hosts, raw IPs) incorrectly expanding to include the matching node's other address family [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix identity-based aliases (tags, users, groups) resolving to IPv4 only; they now include both IPv4 and IPv6 matching Tailscale behavior [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix wildcard (`*`) source in ACLs now using actually-approved subnet routes instead of autoApprover policy prefixes [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix non-wildcard source IPs being dropped when combined with wildcard `*` in the same ACL rule [#2180](https://github.com/juanfont/headscale/pull/2180)
- Fix exit node approval not triggering filter rule recalculation for peers [#2180](https://github.com/juanfont/headscale/pull/2180)
- Policy validation error messages now include field context (e.g., `src=`, `dst=`) and are more descriptive [#2180](https://github.com/juanfont/headscale/pull/2180)
## 0.28.0 (2026-02-04)
+6
View File
@@ -65,6 +65,12 @@ Please have a look at the [`documentation`](https://headscale.net/stable/).
For NixOS users, a module is available in [`nix/`](./nix/).
## Builds from `main`
Development builds from the `main` branch are available as container images and
binaries. See the [development builds](https://headscale.net/stable/setup/install/main/)
documentation for details.
## Talks
- Fosdem 2026 (video): [Headscale & Tailscale: The complementary open source clone](https://fosdem.org/2026/schedule/event/KYQ3LL-headscale-the-complementary-open-source-clone/)
+15 -5
View File
@@ -50,12 +50,21 @@ noise:
# List of IP prefixes to allocate tailaddresses from.
# Each prefix consists of either an IPv4 or IPv6 address,
# and the associated prefix length, delimited by a slash.
# It must be within IP ranges supported by the Tailscale
# client - i.e., subnets of 100.64.0.0/10 and fd7a:115c:a1e0::/48.
# See below:
# IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
#
# WARNING: These prefixes MUST be subsets of the standard Tailscale ranges:
# - IPv4: 100.64.0.0/10 (CGNAT range)
# - IPv6: fd7a:115c:a1e0::/48 (Tailscale ULA range)
#
# Using a SUBSET of these ranges is supported and useful if you want to
# limit IP allocation to a smaller block (e.g., 100.64.0.0/24).
#
# Using ranges OUTSIDE of CGNAT/ULA is NOT supported and will cause
# undefined behaviour. The Tailscale client has hard-coded assumptions
# about these ranges and will break in subtle, hard-to-debug ways.
#
# See:
# IPv4: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#L33
# Any other range is NOT supported, and it will cause unexpected issues.
# IPv6: https://github.com/tailscale/tailscale/blob/22ebb25e833264f58d7c3f534a8b166894a89536/net/tsaddr/tsaddr.go#LL81C52-L81C71
prefixes:
v4: 100.64.0.0/10
v6: fd7a:115c:a1e0::/48
@@ -423,6 +432,7 @@ taildrop:
# When enabled, nodes can send files to other nodes owned by the same user.
# Tagged devices and cross-user transfers are not permitted by Tailscale clients.
enabled: true
# Advanced performance tuning parameters.
# The defaults are carefully chosen and should rarely need adjustment.
# Only modify these if you have identified a specific performance issue.
+4 -2
View File
@@ -191,8 +191,10 @@ You may refer to users in the Headscale policy via:
!!! note "A user identifier in the policy must contain a single `@`"
The Headscale policy requires a single `@` to reference a user. If the username or provider identifier doesn't
already contain a single `@`, it needs to be appended at the end. For example: the username `ssmith` has to be
written as `ssmith@` to be correctly identified as user within the policy.
already contain a single `@`, it needs to be appended at the end. For example: the Headscale username `ssmith` has
to be written as `ssmith@` to be correctly identified as user within the policy.
Ensure that the Headscale username itself does not end with `@`.
!!! warning "Email address or username might be updated by users"
+5 -2
View File
@@ -33,7 +33,8 @@ node can be approved with:
- [Headscale API](api.md)
- Or delegated to an identity provider via [OpenID Connect](oidc.md)
Web authentication relies on the presence of a Headscale user. Use the `headscale users` command to create a new user:
Web authentication relies on the presence of a Headscale user. Use the `headscale users` command to create a new
user[^1]:
```console
headscale users create <USER>
@@ -98,7 +99,7 @@ Its best suited for automation.
=== "Personal devices"
A personal node is always assigned to a Headscale user. Use the `headscale users` command to create a new user:
A personal node is always assigned to a Headscale user. Use the `headscale users` command to create a new user[^1]:
```console
headscale users create <USER>
@@ -139,3 +140,5 @@ Its best suited for automation.
The registration of a tagged node is complete and it should be listed as "online" in the output of
`headscale nodes list`. The "User" column displays `tagged-devices` as the owner of the node. See the "Tags" column for the list of
assigned tags.
[^1]: [Ensure that the Headscale username does not end with `@`.](oidc.md#reference-a-user-in-the-policy)
+4 -4
View File
@@ -1,6 +1,6 @@
mike~=2.1
mkdocs-include-markdown-plugin~=7.1
mkdocs-macros-plugin~=1.3
mkdocs-material[imaging]~=9.5
mkdocs-minify-plugin~=0.7
mkdocs-include-markdown-plugin~=7.2
mkdocs-macros-plugin~=1.5
mkdocs-materialx[imaging]~=10.1
mkdocs-minify-plugin~=0.8
mkdocs-redirects~=1.2
+58
View File
@@ -0,0 +1,58 @@
# Development builds
!!! warning
Development builds are created automatically from the latest `main` branch
and are **not versioned releases**. They may contain incomplete features,
breaking changes, or bugs. Use them for testing only.
Each push to `main` produces container images and cross-compiled binaries.
Container images are multi-arch (amd64, arm64) and use the same distroless
base image as official releases.
## Container images
Images are available from both Docker Hub and GitHub Container Registry, tagged
with the short commit hash of the build (e.g. `main-abc1234`):
- Docker Hub: `docker.io/headscale/headscale:main-<sha>`
- GitHub Container Registry: `ghcr.io/juanfont/headscale:main-<sha>`
To find the latest available tag, check the
[GitHub Actions workflow](https://github.com/juanfont/headscale/actions/workflows/container-main.yml)
or the [GitHub Container Registry package page](https://github.com/juanfont/headscale/pkgs/container/headscale).
For example, to run a specific development build:
```shell
docker run \
--name headscale \
--detach \
--read-only \
--tmpfs /var/run/headscale \
--volume "$(pwd)/config:/etc/headscale:ro" \
--volume "$(pwd)/lib:/var/lib/headscale" \
--publish 127.0.0.1:8080:8080 \
--publish 127.0.0.1:9090:9090 \
--health-cmd "CMD headscale health" \
docker.io/headscale/headscale:main-<sha> \
serve
```
See [Running headscale in a container](./container.md) for full container setup instructions.
## Binaries
Pre-built binaries from the latest successful build on `main` are available
via [nightly.link](https://nightly.link/juanfont/headscale/workflows/container-main/main):
| OS | Arch | Download |
| ----- | ----- | -------------------------------------------------------------------------------------------------------------------------- |
| Linux | amd64 | [headscale-linux-amd64](https://nightly.link/juanfont/headscale/workflows/container-main/main/headscale-linux-amd64.zip) |
| Linux | arm64 | [headscale-linux-arm64](https://nightly.link/juanfont/headscale/workflows/container-main/main/headscale-linux-arm64.zip) |
| macOS | amd64 | [headscale-darwin-amd64](https://nightly.link/juanfont/headscale/workflows/container-main/main/headscale-darwin-amd64.zip) |
| macOS | arm64 | [headscale-darwin-arm64](https://nightly.link/juanfont/headscale/workflows/container-main/main/headscale-darwin-arm64.zip) |
After downloading and extracting the archive, make the binary executable and follow the
[standalone binary installation](./official.md#using-standalone-binaries-advanced)
instructions for setting up the service.
+3 -1
View File
@@ -61,7 +61,7 @@ options, run:
## Manage headscale users
In headscale, a node (also known as machine or device) is [typically assigned to a headscale
user](../ref/registration.md#identity-model). Such a headscale user may have many nodes assigned to them and can be
user](../ref/registration.md#identity-model). Such a headscale user[^1] may have many nodes assigned to them and can be
managed with the `headscale users` command. Invoke the built-in help for more information: `headscale users --help`.
### Create a headscale user
@@ -149,3 +149,5 @@ The command returns the preauthkey on success which is used to connect a node to
```shell
tailscale up --login-server <YOUR_HEADSCALE_URL> --authkey <YOUR_AUTH_KEY>
```
[^1]: [Ensure that the Headscale username does not end with `@`.](../ref/oidc.md#reference-a-user-in-the-policy)
+1 -1
View File
@@ -27,7 +27,7 @@
let
pkgs = nixpkgs.legacyPackages.${prev.stdenv.hostPlatform.system};
buildGo = pkgs.buildGo126Module;
vendorHash = "sha256-oUN53ELb3+xn4yA7lEfXyT2c7NxbQC6RtbkGVq6+RLU=";
vendorHash = "sha256-jom1279Lx2Knff93rfoEgGeBBk+EjJO7GAkaQYlchgY=";
in
{
headscale = buildGo {
+8 -7
View File
@@ -42,9 +42,9 @@ require (
github.com/tailscale/tailsql v0.0.0-20260105194658-001575c3ca09
github.com/tcnksm/go-latest v0.0.0-20170313132115-e3007ae9052e
go4.org/netipx v0.0.0-20231129151722-fdeea329fbba
golang.org/x/crypto v0.47.0
golang.org/x/crypto v0.48.0
golang.org/x/exp v0.0.0-20260112195511-716be5621a96
golang.org/x/net v0.49.0
golang.org/x/net v0.50.0
golang.org/x/oauth2 v0.34.0
golang.org/x/sync v0.19.0
google.golang.org/genproto/googleapis/api v0.0.0-20260203192932-546029d2fa20
@@ -225,18 +225,19 @@ require (
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
go4.org/mem v0.0.0-20240501181205-ae6ca9944745 // indirect
golang.org/x/mod v0.32.0 // indirect
golang.org/x/sys v0.40.0 // indirect
golang.org/x/term v0.39.0 // indirect
golang.org/x/text v0.33.0 // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.41.0 // indirect
golang.org/x/tools v0.42.0 // indirect
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 // indirect
golang.zx2c4.com/wireguard/windows v0.5.3 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260203192932-546029d2fa20 // indirect
)
tool (
golang.org/x/tools/cmd/stress
golang.org/x/tools/cmd/stringer
tailscale.com/cmd/viewer
)
+14 -14
View File
@@ -548,8 +548,8 @@ go4.org/netipx v0.0.0-20231129151722-fdeea329fbba/go.mod h1:PLyyIXexvUFg3Owu6p/W
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.47.0 h1:V6e3FRj+n4dbpw86FJ8Fv7XVOql7TEwpHapKoMJ/GO8=
golang.org/x/crypto v0.47.0/go.mod h1:ff3Y9VzzKbwSSEzWqJsJVBnWmRwRSHt/6Op5n9bQc4A=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/exp v0.0.0-20260112195511-716be5621a96 h1:Z/6YuSHTLOHfNFdb8zVZomZr7cqNgTJvA8+Qz75D8gU=
golang.org/x/exp v0.0.0-20260112195511-716be5621a96/go.mod h1:nzimsREAkjBCIEFtHiYkrJyT+2uy9YZJB7H1k68CXZU=
golang.org/x/exp/typeparams v0.0.0-20240314144324-c7f7c6466f7f h1:phY1HzDcf18Aq9A8KkmRtY9WvOFIxN8wgfvy6Zm1DV8=
@@ -558,15 +558,15 @@ golang.org/x/image v0.27.0 h1:C8gA4oWU/tKkdCfYT6T2u4faJu3MeNS5O8UPWlPF61w=
golang.org/x/image v0.27.0/go.mod h1:xbdrClrAUway1MUTEZDq9mz/UpRwYAkFFNUslZtcB+g=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c=
golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8=
golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o=
golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8=
golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -594,8 +594,8 @@ golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.40.0 h1:DBZZqJ2Rkml6QMQsZywtnjnnGvHza6BTfYFWY9kjEWQ=
golang.org/x/sys v0.40.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -603,24 +603,24 @@ golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuX
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.39.0 h1:RclSuaJf32jOqZz74CkPA9qFuVTX7vhLlpfj/IGWlqY=
golang.org/x/term v0.39.0/go.mod h1:yxzUCTP/U+FzoxfdKmLaA0RV1WgE0VY7hXBwKtY/4ww=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.33.0 h1:B3njUFyqtHDUI5jMn1YIr5B0IE2U0qck04r6d4KPAxE=
golang.org/x/text v0.33.0/go.mod h1:LuMebE6+rBincTi9+xWTY8TztLzKHc/9C1uBCG27+q8=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2 h1:B82qJJgjvYKsXS9jeunTOisW56dUokqW/FOteYJJ/yg=
golang.zx2c4.com/wintun v0.0.0-20230126152724-0fa3db229ce2/go.mod h1:deeaetjYA+DHMHg+sMSMI58GrEteJUUzzw7en6TJQcI=
+52 -1
View File
@@ -16,6 +16,7 @@ import (
"strings"
"sync"
"syscall"
"testing"
"time"
"github.com/cenkalti/backoff/v5"
@@ -101,7 +102,7 @@ type Headscale struct {
// Things that generate changes
extraRecordMan *dns.ExtraRecordsMan
authProvider AuthProvider
mapBatcher mapper.Batcher
mapBatcher *mapper.Batcher
clientStreamsOpen sync.WaitGroup
}
@@ -1069,6 +1070,56 @@ func (h *Headscale) Change(cs ...change.Change) {
h.mapBatcher.AddWork(cs...)
}
// HTTPHandler returns an http.Handler for the Headscale control server.
// The handler serves the Tailscale control protocol including the /key
// endpoint and /ts2021 Noise upgrade path.
func (h *Headscale) HTTPHandler() http.Handler {
return h.createRouter(grpcRuntime.NewServeMux())
}
// NoisePublicKey returns the server's Noise protocol public key.
func (h *Headscale) NoisePublicKey() key.MachinePublic {
return h.noisePrivateKey.Public()
}
// GetState returns the server's state manager for programmatic access
// to users, nodes, policies, and other server state.
func (h *Headscale) GetState() *state.State {
return h.state
}
// SetServerURLForTest updates the server URL in the configuration.
// This is needed for test servers where the URL is not known until
// the HTTP test server starts.
// It panics when called outside of tests.
func (h *Headscale) SetServerURLForTest(tb testing.TB, url string) {
tb.Helper()
h.cfg.ServerURL = url
}
// StartBatcherForTest initialises and starts the map response batcher.
// It registers a cleanup function on tb to stop the batcher.
// It panics when called outside of tests.
func (h *Headscale) StartBatcherForTest(tb testing.TB) {
tb.Helper()
h.mapBatcher = mapper.NewBatcherAndMapper(h.cfg, h.state)
h.mapBatcher.Start()
tb.Cleanup(func() { h.mapBatcher.Close() })
}
// StartEphemeralGCForTest starts the ephemeral node garbage collector.
// It registers a cleanup function on tb to stop the collector.
// It panics when called outside of tests.
func (h *Headscale) StartEphemeralGCForTest(tb testing.TB) {
tb.Helper()
go h.ephemeralGC.Start()
tb.Cleanup(func() { h.ephemeralGC.Close() })
}
// Provide some middleware that can inspect the ACME/autocert https calls
// and log when things are failing.
type acmeLogger struct {
+1 -1
View File
@@ -2507,7 +2507,7 @@ func TestAuthenticationFlows(t *testing.T) {
if req.Followup != "" {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(context.Background(), 100*time.Millisecond)
ctx, cancel = context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
}
+25
View File
@@ -0,0 +1,25 @@
package db
import (
"os"
"path/filepath"
"runtime"
"testing"
)
// TestMain ensures the working directory is set to the package source directory
// so that relative testdata/ paths resolve correctly when the test binary is
// executed from an arbitrary location (e.g., via "go tool stress").
func TestMain(m *testing.M) {
_, filename, _, ok := runtime.Caller(0)
if !ok {
panic("could not determine test source directory")
}
err := os.Chdir(filepath.Dir(filename))
if err != nil {
panic("could not chdir to test source directory: " + err.Error())
}
os.Exit(m.Run())
}
+1 -3
View File
@@ -9,7 +9,6 @@ import (
"testing"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/rs/zerolog"
"zombiezen.com/go/postgrestest"
)
@@ -20,7 +19,6 @@ func newSQLiteTestDB() (*HSDatabase, error) {
}
log.Printf("database path: %s", tmpDir+"/headscale_test.db")
zerolog.SetGlobalLevel(zerolog.Disabled)
db, err := NewHeadscaleDatabase(
&types.Config{
@@ -56,7 +54,7 @@ func newPostgresDBForTest(t *testing.T) *url.URL {
srv, err := postgrestest.Start(ctx)
if err != nil {
t.Fatal(err)
t.Skipf("start postgres: %s", err)
}
t.Cleanup(srv.Cleanup)
+17 -53
View File
@@ -7,7 +7,6 @@ import (
"strings"
"github.com/arl/statsviz"
"github.com/juanfont/headscale/hscontrol/mapper"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/prometheus/client_golang/prometheus/promhttp"
"tailscale.com/tsweb"
@@ -329,38 +328,18 @@ func (h *Headscale) debugBatcher() string {
var nodes []nodeStatus
// Try to get detailed debug info if we have a LockFreeBatcher
if batcher, ok := h.mapBatcher.(*mapper.LockFreeBatcher); ok {
debugInfo := batcher.Debug()
for nodeID, info := range debugInfo {
nodes = append(nodes, nodeStatus{
id: nodeID,
connected: info.Connected,
activeConnections: info.ActiveConnections,
})
totalNodes++
if info.Connected {
connectedCount++
}
}
} else {
// Fallback to basic connection info
connectedMap := h.mapBatcher.ConnectedMap()
connectedMap.Range(func(nodeID types.NodeID, connected bool) bool {
nodes = append(nodes, nodeStatus{
id: nodeID,
connected: connected,
activeConnections: 0,
})
totalNodes++
if connected {
connectedCount++
}
return true
debugInfo := h.mapBatcher.Debug()
for nodeID, info := range debugInfo {
nodes = append(nodes, nodeStatus{
id: nodeID,
connected: info.Connected,
activeConnections: info.ActiveConnections,
})
totalNodes++
if info.Connected {
connectedCount++
}
}
// Sort by node ID
@@ -410,28 +389,13 @@ func (h *Headscale) debugBatcherJSON() DebugBatcherInfo {
TotalNodes: 0,
}
// Try to get detailed debug info if we have a LockFreeBatcher
if batcher, ok := h.mapBatcher.(*mapper.LockFreeBatcher); ok {
debugInfo := batcher.Debug()
for nodeID, debugData := range debugInfo {
info.ConnectedNodes[fmt.Sprintf("%d", nodeID)] = DebugBatcherNodeInfo{
Connected: debugData.Connected,
ActiveConnections: debugData.ActiveConnections,
}
info.TotalNodes++
debugInfo := h.mapBatcher.Debug()
for nodeID, debugData := range debugInfo {
info.ConnectedNodes[fmt.Sprintf("%d", nodeID)] = DebugBatcherNodeInfo{
Connected: debugData.Connected,
ActiveConnections: debugData.ActiveConnections,
}
} else {
// Fallback to basic connection info
connectedMap := h.mapBatcher.ConnectedMap()
connectedMap.Range(func(nodeID types.NodeID, connected bool) bool {
info.ConnectedNodes[fmt.Sprintf("%d", nodeID)] = DebugBatcherNodeInfo{
Connected: connected,
ActiveConnections: 0,
}
info.TotalNodes++
return true
})
info.TotalNodes++
}
return info
+574 -24
View File
@@ -3,6 +3,8 @@ package mapper
import (
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/juanfont/headscale/hscontrol/state"
@@ -24,43 +26,31 @@ var (
ErrNodeNotFoundMapper = errors.New("node not found")
)
// offlineNodeCleanupThreshold is how long a node must be disconnected
// before cleanupOfflineNodes removes its in-memory state.
const offlineNodeCleanupThreshold = 15 * time.Minute
var mapResponseGenerated = promauto.NewCounterVec(prometheus.CounterOpts{
Namespace: "headscale",
Name: "mapresponse_generated_total",
Help: "total count of mapresponses generated by response type",
}, []string{"response_type"})
type batcherFunc func(cfg *types.Config, state *state.State) Batcher
// Batcher defines the common interface for all batcher implementations.
type Batcher interface {
Start()
Close()
AddNode(id types.NodeID, c chan<- *tailcfg.MapResponse, version tailcfg.CapabilityVersion, stop func()) error
RemoveNode(id types.NodeID, c chan<- *tailcfg.MapResponse) bool
IsConnected(id types.NodeID) bool
ConnectedMap() *xsync.Map[types.NodeID, bool]
AddWork(r ...change.Change)
MapResponseFromChange(id types.NodeID, r change.Change) (*tailcfg.MapResponse, error)
DebugMapResponses() (map[types.NodeID][]tailcfg.MapResponse, error)
}
func NewBatcher(batchTime time.Duration, workers int, mapper *mapper) *LockFreeBatcher {
return &LockFreeBatcher{
func NewBatcher(batchTime time.Duration, workers int, mapper *mapper) *Batcher {
return &Batcher{
mapper: mapper,
workers: workers,
tick: time.NewTicker(batchTime),
// The size of this channel is arbitrary chosen, the sizing should be revisited.
workCh: make(chan work, workers*200),
nodes: xsync.NewMap[types.NodeID, *multiChannelNodeConn](),
connected: xsync.NewMap[types.NodeID, *time.Time](),
pendingChanges: xsync.NewMap[types.NodeID, []change.Change](),
workCh: make(chan work, workers*200),
done: make(chan struct{}),
nodes: xsync.NewMap[types.NodeID, *multiChannelNodeConn](),
}
}
// NewBatcherAndMapper creates a Batcher implementation.
func NewBatcherAndMapper(cfg *types.Config, state *state.State) Batcher {
// NewBatcherAndMapper creates a new Batcher with its mapper.
func NewBatcherAndMapper(cfg *types.Config, state *state.State) *Batcher {
m := newMapper(cfg, state)
b := NewBatcher(cfg.Tuning.BatchChangeDelay, cfg.Tuning.BatcherWorkers, m)
m.batcher = b
@@ -180,8 +170,568 @@ type workResult struct {
}
// work represents a unit of work to be processed by workers.
// All pending changes for a node are bundled into a single work item
// so that one worker processes them sequentially. This prevents
// out-of-order MapResponse delivery and races on lastSentPeers
// that occur when multiple workers process changes for the same node.
type work struct {
c change.Change
changes []change.Change
nodeID types.NodeID
resultCh chan<- workResult // optional channel for synchronous operations
}
// Batcher errors.
var (
errConnectionClosed = errors.New("connection channel already closed")
ErrInitialMapSendTimeout = errors.New("sending initial map: timeout")
ErrBatcherShuttingDown = errors.New("batcher shutting down")
ErrConnectionSendTimeout = errors.New("timeout sending to channel (likely stale connection)")
)
// Batcher batches and distributes map responses to connected nodes.
// It uses concurrent maps, per-node mutexes, and a worker pool.
//
// Lifecycle: Call Start() to spawn workers, then Close() to shut down.
// Close() blocks until all workers have exited. A Batcher must not
// be reused after Close().
type Batcher struct {
tick *time.Ticker
mapper *mapper
workers int
nodes *xsync.Map[types.NodeID, *multiChannelNodeConn]
// Work queue channel
workCh chan work
done chan struct{}
doneOnce sync.Once // Ensures done is only closed once
// wg tracks the doWork and all worker goroutines so that Close()
// can block until they have fully exited.
wg sync.WaitGroup
started atomic.Bool // Ensures Start() is only called once
// Metrics
totalNodes atomic.Int64
workQueuedCount atomic.Int64
workProcessed atomic.Int64
workErrors atomic.Int64
}
// AddNode registers a new node connection with the batcher and sends an initial map response.
// It creates or updates the node's connection data, validates the initial map generation,
// and notifies other nodes that this node has come online.
// The stop function tears down the owning session if this connection is later declared stale.
func (b *Batcher) AddNode(
id types.NodeID,
c chan<- *tailcfg.MapResponse,
version tailcfg.CapabilityVersion,
stop func(),
) error {
addNodeStart := time.Now()
nlog := log.With().Uint64(zf.NodeID, id.Uint64()).Logger()
// Generate connection ID
connID := generateConnectionID()
// Create new connection entry
now := time.Now()
newEntry := &connectionEntry{
id: connID,
c: c,
version: version,
created: now,
stop: stop,
}
// Initialize last used timestamp
newEntry.lastUsed.Store(now.Unix())
// Get or create multiChannelNodeConn - this reuses existing offline nodes for rapid reconnection
nodeConn, loaded := b.nodes.LoadOrStore(id, newMultiChannelNodeConn(id, b.mapper))
if !loaded {
b.totalNodes.Add(1)
}
// Add connection to the list (lock-free)
nodeConn.addConnection(newEntry)
// Use the worker pool for controlled concurrency instead of direct generation
initialMap, err := b.MapResponseFromChange(id, change.FullSelf(id))
if err != nil {
nlog.Error().Err(err).Msg("initial map generation failed")
nodeConn.removeConnectionByChannel(c)
if !nodeConn.hasActiveConnections() {
nodeConn.markDisconnected()
}
return fmt.Errorf("generating initial map for node %d: %w", id, err)
}
// Use a blocking send with timeout for initial map since the channel should be ready
// and we want to avoid the race condition where the receiver isn't ready yet
select {
case c <- initialMap:
// Success
case <-time.After(5 * time.Second): //nolint:mnd
nlog.Error().Err(ErrInitialMapSendTimeout).Msg("initial map send timeout")
nlog.Debug().Caller().Dur("timeout.duration", 5*time.Second). //nolint:mnd
Msg("initial map send timed out because channel was blocked or receiver not ready")
nodeConn.removeConnectionByChannel(c)
if !nodeConn.hasActiveConnections() {
nodeConn.markDisconnected()
}
return fmt.Errorf("%w for node %d", ErrInitialMapSendTimeout, id)
}
// Mark the node as connected now that the initial map was sent.
nodeConn.markConnected()
// Node will automatically receive updates through the normal flow
// The initial full map already contains all current state
nlog.Debug().Caller().Dur(zf.TotalDuration, time.Since(addNodeStart)).
Int("active.connections", nodeConn.getActiveConnectionCount()).
Msg("node connection established in batcher")
return nil
}
// RemoveNode disconnects a node from the batcher, marking it as offline and cleaning up its state.
// It validates the connection channel matches one of the current connections, closes that specific connection,
// and keeps the node entry alive for rapid reconnections instead of aggressive deletion.
// Reports if the node still has active connections after removal.
func (b *Batcher) RemoveNode(id types.NodeID, c chan<- *tailcfg.MapResponse) bool {
nlog := log.With().Uint64(zf.NodeID, id.Uint64()).Logger()
nodeConn, exists := b.nodes.Load(id)
if !exists || nodeConn == nil {
nlog.Debug().Caller().Msg("removeNode called for non-existent node")
return false
}
// Remove specific connection
removed := nodeConn.removeConnectionByChannel(c)
if !removed {
nlog.Debug().Caller().Msg("removeNode: channel not found, connection already removed or invalid")
}
// Check if node has any remaining active connections
if nodeConn.hasActiveConnections() {
nlog.Debug().Caller().
Int("active.connections", nodeConn.getActiveConnectionCount()).
Msg("node connection removed but keeping online, other connections remain")
return true // Node still has active connections
}
// No active connections - keep the node entry alive for rapid reconnections
// The node will get a fresh full map when it reconnects
nlog.Debug().Caller().Msg("node disconnected from batcher, keeping entry for rapid reconnection")
nodeConn.markDisconnected()
return false
}
// AddWork queues a change to be processed by the batcher.
func (b *Batcher) AddWork(r ...change.Change) {
b.addToBatch(r...)
}
func (b *Batcher) Start() {
if !b.started.CompareAndSwap(false, true) {
return
}
b.wg.Add(1)
go b.doWork()
}
func (b *Batcher) Close() {
// Signal shutdown to all goroutines, only once.
// Workers and queueWork both select on done, so closing it
// is sufficient for graceful shutdown. We intentionally do NOT
// close workCh here because processBatchedChanges or
// MapResponseFromChange may still be sending on it concurrently.
b.doneOnce.Do(func() {
close(b.done)
})
// Wait for all worker goroutines (and doWork) to exit before
// tearing down node connections. This prevents workers from
// sending on connections that are being closed concurrently.
b.wg.Wait()
// Stop the ticker to prevent resource leaks.
b.tick.Stop()
// Close the underlying channels supplying the data to the clients.
b.nodes.Range(func(nodeID types.NodeID, conn *multiChannelNodeConn) bool {
if conn == nil {
return true
}
conn.close()
return true
})
}
func (b *Batcher) doWork() {
defer b.wg.Done()
for i := range b.workers {
b.wg.Add(1)
go b.worker(i + 1)
}
// Create a cleanup ticker for removing truly disconnected nodes
cleanupTicker := time.NewTicker(5 * time.Minute)
defer cleanupTicker.Stop()
for {
select {
case <-b.tick.C:
// Process batched changes
b.processBatchedChanges()
case <-cleanupTicker.C:
// Clean up nodes that have been offline for too long
b.cleanupOfflineNodes()
case <-b.done:
log.Info().Msg("batcher done channel closed, stopping to feed workers")
return
}
}
}
func (b *Batcher) worker(workerID int) {
defer b.wg.Done()
wlog := log.With().Int(zf.WorkerID, workerID).Logger()
for {
select {
case w, ok := <-b.workCh:
if !ok {
wlog.Debug().Msg("worker channel closing, shutting down")
return
}
b.workProcessed.Add(1)
// Synchronous path: a caller is blocking on resultCh
// waiting for a generated MapResponse (used by AddNode
// for the initial map). Always contains a single change.
if w.resultCh != nil {
var result workResult
if nc, exists := b.nodes.Load(w.nodeID); exists && nc != nil {
// Hold workMu so concurrent async work for this
// node waits until the initial map is sent.
nc.workMu.Lock()
var err error
result.mapResponse, err = generateMapResponse(nc, b.mapper, w.changes[0])
result.err = err
if result.err != nil {
b.workErrors.Add(1)
wlog.Error().Err(result.err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Str(zf.Reason, w.changes[0].Reason).
Msg("failed to generate map response for synchronous work")
} else if result.mapResponse != nil {
nc.updateSentPeers(result.mapResponse)
}
nc.workMu.Unlock()
} else {
result.err = fmt.Errorf("%w: %d", ErrNodeNotFoundMapper, w.nodeID)
b.workErrors.Add(1)
wlog.Error().Err(result.err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Msg("node not found for synchronous work")
}
select {
case w.resultCh <- result:
case <-b.done:
return
}
continue
}
// Async path: process all bundled changes sequentially.
// workMu ensures that if another worker picks up the next
// tick's bundle for the same node, it waits until we
// finish — preventing out-of-order delivery and races
// on lastSentPeers (Clear+Store vs Range).
if nc, exists := b.nodes.Load(w.nodeID); exists && nc != nil {
nc.workMu.Lock()
for _, ch := range w.changes {
err := nc.change(ch)
if err != nil {
b.workErrors.Add(1)
wlog.Error().Err(err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Str(zf.Reason, ch.Reason).
Msg("failed to apply change")
}
}
nc.workMu.Unlock()
}
case <-b.done:
wlog.Debug().Msg("batcher shutting down, exiting worker")
return
}
}
}
// queueWork safely queues work.
func (b *Batcher) queueWork(w work) {
b.workQueuedCount.Add(1)
select {
case b.workCh <- w:
// Successfully queued
case <-b.done:
// Batcher is shutting down
return
}
}
// addToBatch adds changes to the pending batch.
func (b *Batcher) addToBatch(changes ...change.Change) {
// Clean up any nodes being permanently removed from the system.
//
// This handles the case where a node is deleted from state but the batcher
// still has it registered. By cleaning up here, we prevent "node not found"
// errors when workers try to generate map responses for deleted nodes.
//
// Safety: change.Change.PeersRemoved is ONLY populated when nodes are actually
// deleted from the system (via change.NodeRemoved in state.DeleteNode). Policy
// changes that affect peer visibility do NOT use this field - they set
// RequiresRuntimePeerComputation=true and compute removed peers at runtime,
// putting them in tailcfg.MapResponse.PeersRemoved (a different struct).
// Therefore, this cleanup only removes nodes that are truly being deleted,
// not nodes that are still connected but have lost visibility of certain peers.
//
// See: https://github.com/juanfont/headscale/issues/2924
for _, ch := range changes {
for _, removedID := range ch.PeersRemoved {
if _, existed := b.nodes.LoadAndDelete(removedID); existed {
b.totalNodes.Add(-1)
log.Debug().
Uint64(zf.NodeID, removedID.Uint64()).
Msg("removed deleted node from batcher")
}
}
}
// Short circuit if any of the changes is a full update, which
// means we can skip sending individual changes.
if change.HasFull(changes) {
b.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
if nc == nil {
return true
}
nc.pendingMu.Lock()
nc.pending = []change.Change{change.FullUpdate()}
nc.pendingMu.Unlock()
return true
})
return
}
broadcast, targeted := change.SplitTargetedAndBroadcast(changes)
// Handle targeted changes - send only to the specific node
for _, ch := range targeted {
if nc, ok := b.nodes.Load(ch.TargetNode); ok && nc != nil {
nc.appendPending(ch)
}
}
// Handle broadcast changes - send to all nodes, filtering as needed
if len(broadcast) > 0 {
b.nodes.Range(func(nodeID types.NodeID, nc *multiChannelNodeConn) bool {
if nc == nil {
return true
}
filtered := change.FilterForNode(nodeID, broadcast)
if len(filtered) > 0 {
nc.appendPending(filtered...)
}
return true
})
}
}
// processBatchedChanges processes all pending batched changes.
func (b *Batcher) processBatchedChanges() {
b.nodes.Range(func(nodeID types.NodeID, nc *multiChannelNodeConn) bool {
if nc == nil {
return true
}
pending := nc.drainPending()
if len(pending) == 0 {
return true
}
// Queue a single work item containing all pending changes.
// One item per node ensures a single worker processes them
// sequentially, preventing out-of-order delivery.
b.queueWork(work{changes: pending, nodeID: nodeID, resultCh: nil})
return true
})
}
// cleanupOfflineNodes removes nodes that have been offline for too long to prevent memory leaks.
// Uses Compute() for atomic check-and-delete to prevent TOCTOU races where a node
// reconnects between the hasActiveConnections() check and the Delete() call.
func (b *Batcher) cleanupOfflineNodes() {
var nodesToCleanup []types.NodeID
// Find nodes that have been offline for too long by scanning b.nodes
// and checking each node's disconnectedAt timestamp.
b.nodes.Range(func(nodeID types.NodeID, nc *multiChannelNodeConn) bool {
if nc != nil && !nc.hasActiveConnections() && nc.offlineDuration() > offlineNodeCleanupThreshold {
nodesToCleanup = append(nodesToCleanup, nodeID)
}
return true
})
// Clean up the identified nodes using Compute() for atomic check-and-delete.
// This prevents a TOCTOU race where a node reconnects (adding an active
// connection) between the hasActiveConnections() check and the Delete() call.
cleaned := 0
for _, nodeID := range nodesToCleanup {
b.nodes.Compute(
nodeID,
func(conn *multiChannelNodeConn, loaded bool) (*multiChannelNodeConn, xsync.ComputeOp) {
if !loaded || conn == nil || conn.hasActiveConnections() {
return conn, xsync.CancelOp
}
// Perform all bookkeeping inside the Compute callback so
// that a concurrent AddNode (which calls LoadOrStore on
// b.nodes) cannot slip in between the delete and the
// counter update.
b.totalNodes.Add(-1)
cleaned++
log.Info().Uint64(zf.NodeID, nodeID.Uint64()).
Dur("offline_duration", offlineNodeCleanupThreshold).
Msg("cleaning up node that has been offline for too long")
return conn, xsync.DeleteOp
},
)
}
if cleaned > 0 {
log.Info().Int(zf.CleanedNodes, cleaned).
Msg("completed cleanup of long-offline nodes")
}
}
// IsConnected is a lock-free read that checks if a node is connected.
// A node is considered connected if it has active connections or has
// not been marked as disconnected.
func (b *Batcher) IsConnected(id types.NodeID) bool {
nodeConn, exists := b.nodes.Load(id)
if !exists || nodeConn == nil {
return false
}
return nodeConn.isConnected()
}
// ConnectedMap returns a lock-free map of all known nodes and their
// connection status (true = connected, false = disconnected).
func (b *Batcher) ConnectedMap() *xsync.Map[types.NodeID, bool] {
ret := xsync.NewMap[types.NodeID, bool]()
b.nodes.Range(func(id types.NodeID, nc *multiChannelNodeConn) bool {
if nc != nil {
ret.Store(id, nc.isConnected())
}
return true
})
return ret
}
// MapResponseFromChange queues work to generate a map response and waits for the result.
// This allows synchronous map generation using the same worker pool.
func (b *Batcher) MapResponseFromChange(id types.NodeID, ch change.Change) (*tailcfg.MapResponse, error) {
resultCh := make(chan workResult, 1)
// Queue the work with a result channel using the safe queueing method
b.queueWork(work{changes: []change.Change{ch}, nodeID: id, resultCh: resultCh})
// Wait for the result
select {
case result := <-resultCh:
return result.mapResponse, result.err
case <-b.done:
return nil, fmt.Errorf("%w while generating map response for node %d", ErrBatcherShuttingDown, id)
}
}
// DebugNodeInfo contains debug information about a node's connections.
type DebugNodeInfo struct {
Connected bool `json:"connected"`
ActiveConnections int `json:"active_connections"`
}
// Debug returns a pre-baked map of node debug information for the debug interface.
func (b *Batcher) Debug() map[types.NodeID]DebugNodeInfo {
result := make(map[types.NodeID]DebugNodeInfo)
b.nodes.Range(func(id types.NodeID, nc *multiChannelNodeConn) bool {
if nc == nil {
return true
}
result[id] = DebugNodeInfo{
Connected: nc.isConnected(),
ActiveConnections: nc.getActiveConnectionCount(),
}
return true
})
return result
}
func (b *Batcher) DebugMapResponses() (map[types.NodeID][]tailcfg.MapResponse, error) {
return b.mapper.debugMapResponses()
}
// WorkErrors returns the count of work errors encountered.
// This is primarily useful for testing and debugging.
func (b *Batcher) WorkErrors() int64 {
return b.workErrors.Load()
}
+742
View File
@@ -0,0 +1,742 @@
package mapper
// Benchmarks for batcher components and full pipeline.
//
// Organized into three tiers:
// - Component benchmarks: individual functions (connectionEntry.send, computePeerDiff, etc.)
// - System benchmarks: batching mechanics (addToBatch, processBatchedChanges, broadcast)
// - Full pipeline benchmarks: end-to-end with real DB (gated behind !testing.Short())
//
// All benchmarks use sub-benchmarks with 10/100/1000 node counts for scaling analysis.
import (
"fmt"
"sync"
"testing"
"time"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/change"
"github.com/puzpuzpuz/xsync/v4"
"tailscale.com/tailcfg"
)
// ============================================================================
// Component Benchmarks
// ============================================================================
// BenchmarkConnectionEntry_Send measures the throughput of sending a single
// MapResponse through a connectionEntry with a buffered channel.
func BenchmarkConnectionEntry_Send(b *testing.B) {
ch := make(chan *tailcfg.MapResponse, b.N+1)
entry := makeConnectionEntry("bench-conn", ch)
data := testMapResponse()
b.ResetTimer()
for range b.N {
_ = entry.send(data)
}
}
// BenchmarkMultiChannelSend measures broadcast throughput to multiple connections.
func BenchmarkMultiChannelSend(b *testing.B) {
for _, connCount := range []int{1, 3, 10} {
b.Run(fmt.Sprintf("%dconn", connCount), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
channels := make([]chan *tailcfg.MapResponse, connCount)
for i := range channels {
channels[i] = make(chan *tailcfg.MapResponse, b.N+1)
mc.addConnection(makeConnectionEntry(fmt.Sprintf("conn-%d", i), channels[i]))
}
data := testMapResponse()
b.ResetTimer()
for range b.N {
_ = mc.send(data)
}
})
}
}
// BenchmarkComputePeerDiff measures the cost of computing peer diffs at scale.
func BenchmarkComputePeerDiff(b *testing.B) {
for _, peerCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dpeers", peerCount), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
// Populate tracked peers: 1..peerCount
for i := 1; i <= peerCount; i++ {
mc.lastSentPeers.Store(tailcfg.NodeID(i), struct{}{})
}
// Current peers: remove ~10% (every 10th peer is missing)
current := make([]tailcfg.NodeID, 0, peerCount)
for i := 1; i <= peerCount; i++ {
if i%10 != 0 {
current = append(current, tailcfg.NodeID(i))
}
}
b.ResetTimer()
for range b.N {
_ = mc.computePeerDiff(current)
}
})
}
}
// BenchmarkUpdateSentPeers measures the cost of updating peer tracking state.
func BenchmarkUpdateSentPeers(b *testing.B) {
for _, peerCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dpeers_full", peerCount), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
// Pre-build response with full peer list
peerIDs := make([]tailcfg.NodeID, peerCount)
for i := range peerIDs {
peerIDs[i] = tailcfg.NodeID(i + 1)
}
resp := testMapResponseWithPeers(peerIDs...)
b.ResetTimer()
for range b.N {
mc.updateSentPeers(resp)
}
})
b.Run(fmt.Sprintf("%dpeers_incremental", peerCount), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
// Pre-populate with existing peers
for i := 1; i <= peerCount; i++ {
mc.lastSentPeers.Store(tailcfg.NodeID(i), struct{}{})
}
// Build incremental response: add 10% new peers
addCount := peerCount / 10
if addCount == 0 {
addCount = 1
}
resp := testMapResponse()
resp.PeersChanged = make([]*tailcfg.Node, addCount)
for i := range addCount {
resp.PeersChanged[i] = &tailcfg.Node{ID: tailcfg.NodeID(peerCount + i + 1)}
}
b.ResetTimer()
for range b.N {
mc.updateSentPeers(resp)
}
})
}
}
// ============================================================================
// System Benchmarks (no DB, batcher mechanics only)
// ============================================================================
// benchBatcher creates a lightweight batcher for benchmarks. Unlike the test
// helper, it doesn't register cleanup and suppresses logging.
func benchBatcher(nodeCount, bufferSize int) (*Batcher, map[types.NodeID]chan *tailcfg.MapResponse) {
b := &Batcher{
tick: time.NewTicker(1 * time.Hour), // never fires during bench
workers: 4,
workCh: make(chan work, 4*200),
nodes: xsync.NewMap[types.NodeID, *multiChannelNodeConn](),
done: make(chan struct{}),
}
channels := make(map[types.NodeID]chan *tailcfg.MapResponse, nodeCount)
for i := 1; i <= nodeCount; i++ {
id := types.NodeID(i) //nolint:gosec // benchmark with small controlled values
mc := newMultiChannelNodeConn(id, nil)
ch := make(chan *tailcfg.MapResponse, bufferSize)
entry := &connectionEntry{
id: fmt.Sprintf("conn-%d", i),
c: ch,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
b.nodes.Store(id, mc)
channels[id] = ch
}
b.totalNodes.Store(int64(nodeCount))
return b, channels
}
// BenchmarkAddToBatch_Broadcast measures the cost of broadcasting a change
// to all nodes via addToBatch (no worker processing, just queuing).
func BenchmarkAddToBatch_Broadcast(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
ch := change.DERPMap()
b.ResetTimer()
for range b.N {
batcher.addToBatch(ch)
// Clear pending to avoid unbounded growth
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
}
})
}
}
// BenchmarkAddToBatch_Targeted measures the cost of adding a targeted change
// to a single node.
func BenchmarkAddToBatch_Targeted(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
targetID := types.NodeID(1 + (i % nodeCount)) //nolint:gosec // benchmark
ch := change.Change{
Reason: "bench-targeted",
TargetNode: targetID,
PeerPatches: []*tailcfg.PeerChange{
{NodeID: tailcfg.NodeID(targetID)}, //nolint:gosec // benchmark
},
}
batcher.addToBatch(ch)
// Clear pending periodically to avoid growth
if i%100 == 99 {
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
}
}
})
}
}
// BenchmarkAddToBatch_FullUpdate measures the cost of a FullUpdate broadcast.
func BenchmarkAddToBatch_FullUpdate(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for range b.N {
batcher.addToBatch(change.FullUpdate())
}
})
}
}
// BenchmarkProcessBatchedChanges measures the cost of moving pending changes
// to the work queue.
func BenchmarkProcessBatchedChanges(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dpending", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
// Use a very large work channel to avoid blocking
batcher.workCh = make(chan work, nodeCount*b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for range b.N {
b.StopTimer()
// Seed pending changes
for i := 1; i <= nodeCount; i++ {
if nc, ok := batcher.nodes.Load(types.NodeID(i)); ok { //nolint:gosec // benchmark
nc.appendPending(change.DERPMap())
}
}
b.StartTimer()
batcher.processBatchedChanges()
}
})
}
}
// BenchmarkBroadcastToN measures end-to-end broadcast: addToBatch + processBatchedChanges
// to N nodes. Does NOT include worker processing (MapResponse generation).
func BenchmarkBroadcastToN(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
batcher.workCh = make(chan work, nodeCount*b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
ch := change.DERPMap()
b.ResetTimer()
for range b.N {
batcher.addToBatch(ch)
batcher.processBatchedChanges()
}
})
}
}
// BenchmarkMultiChannelBroadcast measures the cost of sending a MapResponse
// to N nodes each with varying connection counts.
func BenchmarkMultiChannelBroadcast(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
// Add extra connections to every 3rd node
for i := 1; i <= nodeCount; i++ {
if i%3 == 0 {
if mc, ok := batcher.nodes.Load(types.NodeID(i)); ok { //nolint:gosec // benchmark
for j := range 2 {
ch := make(chan *tailcfg.MapResponse, b.N+1)
entry := &connectionEntry{
id: fmt.Sprintf("extra-%d-%d", i, j),
c: ch,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
}
}
}
}
data := testMapResponse()
b.ResetTimer()
for range b.N {
batcher.nodes.Range(func(_ types.NodeID, mc *multiChannelNodeConn) bool {
_ = mc.send(data)
return true
})
}
})
}
}
// BenchmarkConcurrentAddToBatch measures addToBatch throughput under
// concurrent access from multiple goroutines.
func BenchmarkConcurrentAddToBatch(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
// Background goroutine to drain pending periodically
drainDone := make(chan struct{})
go func() {
defer close(drainDone)
for {
select {
case <-batcher.done:
return
default:
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
time.Sleep(time.Millisecond) //nolint:forbidigo // benchmark drain loop
}
}
}()
ch := change.DERPMap()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
batcher.addToBatch(ch)
}
})
b.StopTimer()
// Cleanup
close(batcher.done)
<-drainDone
// Re-open done so the defer doesn't double-close
batcher.done = make(chan struct{})
})
}
}
// BenchmarkIsConnected measures the read throughput of IsConnected checks.
func BenchmarkIsConnected(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, _ := benchBatcher(nodeCount, 1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
id := types.NodeID(1 + (i % nodeCount)) //nolint:gosec // benchmark
_ = batcher.IsConnected(id)
}
})
}
}
// BenchmarkConnectedMap measures the cost of building the full connected map.
func BenchmarkConnectedMap(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, channels := benchBatcher(nodeCount, 1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
// Disconnect 10% of nodes for a realistic mix
for i := 1; i <= nodeCount; i++ {
if i%10 == 0 {
id := types.NodeID(i) //nolint:gosec // benchmark
if mc, ok := batcher.nodes.Load(id); ok {
mc.removeConnectionByChannel(channels[id])
mc.markDisconnected()
}
}
}
b.ResetTimer()
for range b.N {
_ = batcher.ConnectedMap()
}
})
}
}
// BenchmarkConnectionChurn measures the cost of add/remove connection cycling
// which simulates client reconnection patterns.
func BenchmarkConnectionChurn(b *testing.B) {
for _, nodeCount := range []int{10, 100, 1000} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, channels := benchBatcher(nodeCount, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
id := types.NodeID(1 + (i % nodeCount)) //nolint:gosec // benchmark
mc, ok := batcher.nodes.Load(id)
if !ok {
continue
}
// Remove old connection
oldCh := channels[id]
mc.removeConnectionByChannel(oldCh)
// Add new connection
newCh := make(chan *tailcfg.MapResponse, 10)
entry := &connectionEntry{
id: fmt.Sprintf("churn-%d", i),
c: newCh,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
channels[id] = newCh
}
})
}
}
// BenchmarkConcurrentSendAndChurn measures the combined cost of sends happening
// concurrently with connection churn - the hot path in production.
func BenchmarkConcurrentSendAndChurn(b *testing.B) {
for _, nodeCount := range []int{10, 100} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
batcher, channels := benchBatcher(nodeCount, 100)
var mu sync.Mutex // protect channels map
stopChurn := make(chan struct{})
defer close(stopChurn)
// Background churn on 10% of nodes
go func() {
i := 0
for {
select {
case <-stopChurn:
return
default:
id := types.NodeID(1 + (i % nodeCount)) //nolint:gosec // benchmark
if i%10 == 0 { // only churn 10%
mc, ok := batcher.nodes.Load(id)
if ok {
mu.Lock()
oldCh := channels[id]
mu.Unlock()
mc.removeConnectionByChannel(oldCh)
newCh := make(chan *tailcfg.MapResponse, 100)
entry := &connectionEntry{
id: fmt.Sprintf("churn-%d", i),
c: newCh,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
mu.Lock()
channels[id] = newCh
mu.Unlock()
}
}
i++
}
}
}()
data := testMapResponse()
b.ResetTimer()
for range b.N {
batcher.nodes.Range(func(_ types.NodeID, mc *multiChannelNodeConn) bool {
_ = mc.send(data)
return true
})
}
})
}
}
// ============================================================================
// Full Pipeline Benchmarks (with DB)
// ============================================================================
// BenchmarkAddNode measures the cost of adding nodes to the batcher,
// including initial MapResponse generation from a real database.
func BenchmarkAddNode(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 100} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
// Start consumers
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
b.ResetTimer()
for range b.N {
// Connect all nodes (measuring AddNode cost)
for i := range allNodes {
node := &allNodes[i]
_ = batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
}
b.StopTimer()
// Disconnect for next iteration
for i := range allNodes {
node := &allNodes[i]
batcher.RemoveNode(node.n.ID, node.ch)
}
// Drain channels
for i := range allNodes {
for {
select {
case <-allNodes[i].ch:
default:
goto drained
}
}
drained:
}
b.StartTimer()
}
})
}
}
// BenchmarkFullPipeline measures the full pipeline cost: addToBatch → processBatchedChanges
// → worker → generateMapResponse → send, with real nodes from a database.
func BenchmarkFullPipeline(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 100} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
// Start consumers
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
// Connect all nodes first
for i := range allNodes {
node := &allNodes[i]
err := batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
if err != nil {
b.Fatalf("failed to add node %d: %v", i, err)
}
}
// Wait for initial maps to settle
time.Sleep(200 * time.Millisecond) //nolint:forbidigo // benchmark coordination
b.ResetTimer()
for range b.N {
batcher.AddWork(change.DERPMap())
// Allow workers to process (the batcher tick is what normally
// triggers processBatchedChanges, but for benchmarks we need
// to give the system time to process)
time.Sleep(20 * time.Millisecond) //nolint:forbidigo // benchmark coordination
}
})
}
}
// BenchmarkMapResponseFromChange measures the cost of synchronous
// MapResponse generation for individual nodes.
func BenchmarkMapResponseFromChange(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 100} {
b.Run(fmt.Sprintf("%dnodes", nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
// Start consumers
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
// Connect all nodes
for i := range allNodes {
node := &allNodes[i]
err := batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
if err != nil {
b.Fatalf("failed to add node %d: %v", i, err)
}
}
time.Sleep(200 * time.Millisecond) //nolint:forbidigo // benchmark coordination
ch := change.DERPMap()
b.ResetTimer()
for i := range b.N {
nodeIdx := i % len(allNodes)
_, _ = batcher.MapResponseFromChange(allNodes[nodeIdx].n.ID, ch)
}
})
}
}
File diff suppressed because it is too large Load Diff
-889
View File
@@ -1,889 +0,0 @@
package mapper
import (
"crypto/rand"
"encoding/hex"
"errors"
"fmt"
"sync"
"sync/atomic"
"time"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/change"
"github.com/juanfont/headscale/hscontrol/util/zlog/zf"
"github.com/puzpuzpuz/xsync/v4"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"tailscale.com/tailcfg"
)
// LockFreeBatcher errors.
var (
errConnectionClosed = errors.New("connection channel already closed")
ErrInitialMapSendTimeout = errors.New("sending initial map: timeout")
ErrBatcherShuttingDown = errors.New("batcher shutting down")
ErrConnectionSendTimeout = errors.New("timeout sending to channel (likely stale connection)")
)
// LockFreeBatcher uses atomic operations and concurrent maps to eliminate mutex contention.
type LockFreeBatcher struct {
tick *time.Ticker
mapper *mapper
workers int
nodes *xsync.Map[types.NodeID, *multiChannelNodeConn]
connected *xsync.Map[types.NodeID, *time.Time]
// Work queue channel
workCh chan work
workChOnce sync.Once // Ensures workCh is only closed once
done chan struct{}
doneOnce sync.Once // Ensures done is only closed once
// Batching state
pendingChanges *xsync.Map[types.NodeID, []change.Change]
// Metrics
totalNodes atomic.Int64
workQueuedCount atomic.Int64
workProcessed atomic.Int64
workErrors atomic.Int64
}
// AddNode registers a new node connection with the batcher and sends an initial map response.
// It creates or updates the node's connection data, validates the initial map generation,
// and notifies other nodes that this node has come online.
// The stop function tears down the owning session if this connection is later declared stale.
func (b *LockFreeBatcher) AddNode(
id types.NodeID,
c chan<- *tailcfg.MapResponse,
version tailcfg.CapabilityVersion,
stop func(),
) error {
addNodeStart := time.Now()
nlog := log.With().Uint64(zf.NodeID, id.Uint64()).Logger()
// Generate connection ID
connID := generateConnectionID()
// Create new connection entry
now := time.Now()
newEntry := &connectionEntry{
id: connID,
c: c,
version: version,
created: now,
stop: stop,
}
// Initialize last used timestamp
newEntry.lastUsed.Store(now.Unix())
// Get or create multiChannelNodeConn - this reuses existing offline nodes for rapid reconnection
nodeConn, loaded := b.nodes.LoadOrStore(id, newMultiChannelNodeConn(id, b.mapper))
if !loaded {
b.totalNodes.Add(1)
}
// Add connection to the list (lock-free)
nodeConn.addConnection(newEntry)
// Use the worker pool for controlled concurrency instead of direct generation
initialMap, err := b.MapResponseFromChange(id, change.FullSelf(id))
if err != nil {
nlog.Error().Err(err).Msg("initial map generation failed")
nodeConn.removeConnectionByChannel(c)
return fmt.Errorf("generating initial map for node %d: %w", id, err)
}
// Use a blocking send with timeout for initial map since the channel should be ready
// and we want to avoid the race condition where the receiver isn't ready yet
select {
case c <- initialMap:
// Success
case <-time.After(5 * time.Second): //nolint:mnd
nlog.Error().Err(ErrInitialMapSendTimeout).Msg("initial map send timeout")
nlog.Debug().Caller().Dur("timeout.duration", 5*time.Second). //nolint:mnd
Msg("initial map send timed out because channel was blocked or receiver not ready")
nodeConn.removeConnectionByChannel(c)
return fmt.Errorf("%w for node %d", ErrInitialMapSendTimeout, id)
}
// Update connection status
b.connected.Store(id, nil) // nil = connected
// Node will automatically receive updates through the normal flow
// The initial full map already contains all current state
nlog.Debug().Caller().Dur(zf.TotalDuration, time.Since(addNodeStart)).
Int("active.connections", nodeConn.getActiveConnectionCount()).
Msg("node connection established in batcher")
return nil
}
// RemoveNode disconnects a node from the batcher, marking it as offline and cleaning up its state.
// It validates the connection channel matches one of the current connections, closes that specific connection,
// and keeps the node entry alive for rapid reconnections instead of aggressive deletion.
// Reports if the node still has active connections after removal.
func (b *LockFreeBatcher) RemoveNode(id types.NodeID, c chan<- *tailcfg.MapResponse) bool {
nlog := log.With().Uint64(zf.NodeID, id.Uint64()).Logger()
nodeConn, exists := b.nodes.Load(id)
if !exists {
nlog.Debug().Caller().Msg("removeNode called for non-existent node")
return false
}
// Remove specific connection
removed := nodeConn.removeConnectionByChannel(c)
if !removed {
nlog.Debug().Caller().Msg("removeNode: channel not found, connection already removed or invalid")
}
// Check if node has any remaining active connections
if nodeConn.hasActiveConnections() {
nlog.Debug().Caller().
Int("active.connections", nodeConn.getActiveConnectionCount()).
Msg("node connection removed but keeping online, other connections remain")
return true // Node still has active connections
}
// No active connections - keep the node entry alive for rapid reconnections
// The node will get a fresh full map when it reconnects
nlog.Debug().Caller().Msg("node disconnected from batcher, keeping entry for rapid reconnection")
b.connected.Store(id, new(time.Now()))
return false
}
// AddWork queues a change to be processed by the batcher.
func (b *LockFreeBatcher) AddWork(r ...change.Change) {
b.addWork(r...)
}
func (b *LockFreeBatcher) Start() {
b.done = make(chan struct{})
go b.doWork()
}
func (b *LockFreeBatcher) Close() {
// Signal shutdown to all goroutines, only once
b.doneOnce.Do(func() {
if b.done != nil {
close(b.done)
}
})
// Only close workCh once using sync.Once to prevent races
b.workChOnce.Do(func() {
close(b.workCh)
})
// Close the underlying channels supplying the data to the clients.
b.nodes.Range(func(nodeID types.NodeID, conn *multiChannelNodeConn) bool {
conn.close()
return true
})
}
func (b *LockFreeBatcher) doWork() {
for i := range b.workers {
go b.worker(i + 1)
}
// Create a cleanup ticker for removing truly disconnected nodes
cleanupTicker := time.NewTicker(5 * time.Minute)
defer cleanupTicker.Stop()
for {
select {
case <-b.tick.C:
// Process batched changes
b.processBatchedChanges()
case <-cleanupTicker.C:
// Clean up nodes that have been offline for too long
b.cleanupOfflineNodes()
case <-b.done:
log.Info().Msg("batcher done channel closed, stopping to feed workers")
return
}
}
}
func (b *LockFreeBatcher) worker(workerID int) {
wlog := log.With().Int(zf.WorkerID, workerID).Logger()
for {
select {
case w, ok := <-b.workCh:
if !ok {
wlog.Debug().Msg("worker channel closing, shutting down")
return
}
b.workProcessed.Add(1)
// If the resultCh is set, it means that this is a work request
// where there is a blocking function waiting for the map that
// is being generated.
// This is used for synchronous map generation.
if w.resultCh != nil {
var result workResult
if nc, exists := b.nodes.Load(w.nodeID); exists {
var err error
result.mapResponse, err = generateMapResponse(nc, b.mapper, w.c)
result.err = err
if result.err != nil {
b.workErrors.Add(1)
wlog.Error().Err(result.err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Str(zf.Reason, w.c.Reason).
Msg("failed to generate map response for synchronous work")
} else if result.mapResponse != nil {
// Update peer tracking for synchronous responses too
nc.updateSentPeers(result.mapResponse)
}
} else {
result.err = fmt.Errorf("%w: %d", ErrNodeNotFoundMapper, w.nodeID)
b.workErrors.Add(1)
wlog.Error().Err(result.err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Msg("node not found for synchronous work")
}
// Send result
select {
case w.resultCh <- result:
case <-b.done:
return
}
continue
}
// If resultCh is nil, this is an asynchronous work request
// that should be processed and sent to the node instead of
// returned to the caller.
if nc, exists := b.nodes.Load(w.nodeID); exists {
// Apply change to node - this will handle offline nodes gracefully
// and queue work for when they reconnect
err := nc.change(w.c)
if err != nil {
b.workErrors.Add(1)
wlog.Error().Err(err).
Uint64(zf.NodeID, w.nodeID.Uint64()).
Str(zf.Reason, w.c.Reason).
Msg("failed to apply change")
}
}
case <-b.done:
wlog.Debug().Msg("batcher shutting down, exiting worker")
return
}
}
}
func (b *LockFreeBatcher) addWork(r ...change.Change) {
b.addToBatch(r...)
}
// queueWork safely queues work.
func (b *LockFreeBatcher) queueWork(w work) {
b.workQueuedCount.Add(1)
select {
case b.workCh <- w:
// Successfully queued
case <-b.done:
// Batcher is shutting down
return
}
}
// addToBatch adds changes to the pending batch.
func (b *LockFreeBatcher) addToBatch(changes ...change.Change) {
// Clean up any nodes being permanently removed from the system.
//
// This handles the case where a node is deleted from state but the batcher
// still has it registered. By cleaning up here, we prevent "node not found"
// errors when workers try to generate map responses for deleted nodes.
//
// Safety: change.Change.PeersRemoved is ONLY populated when nodes are actually
// deleted from the system (via change.NodeRemoved in state.DeleteNode). Policy
// changes that affect peer visibility do NOT use this field - they set
// RequiresRuntimePeerComputation=true and compute removed peers at runtime,
// putting them in tailcfg.MapResponse.PeersRemoved (a different struct).
// Therefore, this cleanup only removes nodes that are truly being deleted,
// not nodes that are still connected but have lost visibility of certain peers.
//
// See: https://github.com/juanfont/headscale/issues/2924
for _, ch := range changes {
for _, removedID := range ch.PeersRemoved {
if _, existed := b.nodes.LoadAndDelete(removedID); existed {
b.totalNodes.Add(-1)
log.Debug().
Uint64(zf.NodeID, removedID.Uint64()).
Msg("removed deleted node from batcher")
}
b.connected.Delete(removedID)
b.pendingChanges.Delete(removedID)
}
}
// Short circuit if any of the changes is a full update, which
// means we can skip sending individual changes.
if change.HasFull(changes) {
b.nodes.Range(func(nodeID types.NodeID, _ *multiChannelNodeConn) bool {
b.pendingChanges.Store(nodeID, []change.Change{change.FullUpdate()})
return true
})
return
}
broadcast, targeted := change.SplitTargetedAndBroadcast(changes)
// Handle targeted changes - send only to the specific node
for _, ch := range targeted {
pending, _ := b.pendingChanges.LoadOrStore(ch.TargetNode, []change.Change{})
pending = append(pending, ch)
b.pendingChanges.Store(ch.TargetNode, pending)
}
// Handle broadcast changes - send to all nodes, filtering as needed
if len(broadcast) > 0 {
b.nodes.Range(func(nodeID types.NodeID, _ *multiChannelNodeConn) bool {
filtered := change.FilterForNode(nodeID, broadcast)
if len(filtered) > 0 {
pending, _ := b.pendingChanges.LoadOrStore(nodeID, []change.Change{})
pending = append(pending, filtered...)
b.pendingChanges.Store(nodeID, pending)
}
return true
})
}
}
// processBatchedChanges processes all pending batched changes.
func (b *LockFreeBatcher) processBatchedChanges() {
if b.pendingChanges == nil {
return
}
// Process all pending changes
b.pendingChanges.Range(func(nodeID types.NodeID, pending []change.Change) bool {
if len(pending) == 0 {
return true
}
// Send all batched changes for this node
for _, ch := range pending {
b.queueWork(work{c: ch, nodeID: nodeID, resultCh: nil})
}
// Clear the pending changes for this node
b.pendingChanges.Delete(nodeID)
return true
})
}
// cleanupOfflineNodes removes nodes that have been offline for too long to prevent memory leaks.
// TODO(kradalby): reevaluate if we want to keep this.
func (b *LockFreeBatcher) cleanupOfflineNodes() {
cleanupThreshold := 15 * time.Minute
now := time.Now()
var nodesToCleanup []types.NodeID
// Find nodes that have been offline for too long
b.connected.Range(func(nodeID types.NodeID, disconnectTime *time.Time) bool {
if disconnectTime != nil && now.Sub(*disconnectTime) > cleanupThreshold {
// Double-check the node doesn't have active connections
if nodeConn, exists := b.nodes.Load(nodeID); exists {
if !nodeConn.hasActiveConnections() {
nodesToCleanup = append(nodesToCleanup, nodeID)
}
}
}
return true
})
// Clean up the identified nodes
for _, nodeID := range nodesToCleanup {
log.Info().Uint64(zf.NodeID, nodeID.Uint64()).
Dur("offline_duration", cleanupThreshold).
Msg("cleaning up node that has been offline for too long")
b.nodes.Delete(nodeID)
b.connected.Delete(nodeID)
b.totalNodes.Add(-1)
}
if len(nodesToCleanup) > 0 {
log.Info().Int(zf.CleanedNodes, len(nodesToCleanup)).
Msg("completed cleanup of long-offline nodes")
}
}
// IsConnected is lock-free read that checks if a node has any active connections.
func (b *LockFreeBatcher) IsConnected(id types.NodeID) bool {
// First check if we have active connections for this node
if nodeConn, exists := b.nodes.Load(id); exists {
if nodeConn.hasActiveConnections() {
return true
}
}
// Check disconnected timestamp with grace period
val, ok := b.connected.Load(id)
if !ok {
return false
}
// nil means connected
if val == nil {
return true
}
return false
}
// ConnectedMap returns a lock-free map of all connected nodes.
func (b *LockFreeBatcher) ConnectedMap() *xsync.Map[types.NodeID, bool] {
ret := xsync.NewMap[types.NodeID, bool]()
// First, add all nodes with active connections
b.nodes.Range(func(id types.NodeID, nodeConn *multiChannelNodeConn) bool {
if nodeConn.hasActiveConnections() {
ret.Store(id, true)
}
return true
})
// Then add all entries from the connected map
b.connected.Range(func(id types.NodeID, val *time.Time) bool {
// Only add if not already added as connected above
if _, exists := ret.Load(id); !exists {
if val == nil {
// nil means connected
ret.Store(id, true)
} else {
// timestamp means disconnected
ret.Store(id, false)
}
}
return true
})
return ret
}
// MapResponseFromChange queues work to generate a map response and waits for the result.
// This allows synchronous map generation using the same worker pool.
func (b *LockFreeBatcher) MapResponseFromChange(id types.NodeID, ch change.Change) (*tailcfg.MapResponse, error) {
resultCh := make(chan workResult, 1)
// Queue the work with a result channel using the safe queueing method
b.queueWork(work{c: ch, nodeID: id, resultCh: resultCh})
// Wait for the result
select {
case result := <-resultCh:
return result.mapResponse, result.err
case <-b.done:
return nil, fmt.Errorf("%w while generating map response for node %d", ErrBatcherShuttingDown, id)
}
}
// connectionEntry represents a single connection to a node.
type connectionEntry struct {
id string // unique connection ID
c chan<- *tailcfg.MapResponse
version tailcfg.CapabilityVersion
created time.Time
stop func()
lastUsed atomic.Int64 // Unix timestamp of last successful send
closed atomic.Bool // Indicates if this connection has been closed
}
// multiChannelNodeConn manages multiple concurrent connections for a single node.
type multiChannelNodeConn struct {
id types.NodeID
mapper *mapper
log zerolog.Logger
mutex sync.RWMutex
connections []*connectionEntry
updateCount atomic.Int64
// lastSentPeers tracks which peers were last sent to this node.
// This enables computing diffs for policy changes instead of sending
// full peer lists (which clients interpret as "no change" when empty).
// Using xsync.Map for lock-free concurrent access.
lastSentPeers *xsync.Map[tailcfg.NodeID, struct{}]
}
// generateConnectionID generates a unique connection identifier.
func generateConnectionID() string {
bytes := make([]byte, 8)
_, _ = rand.Read(bytes)
return hex.EncodeToString(bytes)
}
// newMultiChannelNodeConn creates a new multi-channel node connection.
func newMultiChannelNodeConn(id types.NodeID, mapper *mapper) *multiChannelNodeConn {
return &multiChannelNodeConn{
id: id,
mapper: mapper,
lastSentPeers: xsync.NewMap[tailcfg.NodeID, struct{}](),
log: log.With().Uint64(zf.NodeID, id.Uint64()).Logger(),
}
}
func (mc *multiChannelNodeConn) close() {
mc.mutex.Lock()
defer mc.mutex.Unlock()
for _, conn := range mc.connections {
mc.stopConnection(conn)
}
}
// stopConnection marks a connection as closed and tears down the owning session
// at most once, even if multiple cleanup paths race to remove it.
func (mc *multiChannelNodeConn) stopConnection(conn *connectionEntry) {
if conn.closed.CompareAndSwap(false, true) {
if conn.stop != nil {
conn.stop()
}
}
}
// removeConnectionAtIndexLocked removes the active connection at index.
// If stopConnection is true, it also stops that session.
// Caller must hold mc.mutex.
func (mc *multiChannelNodeConn) removeConnectionAtIndexLocked(i int, stopConnection bool) *connectionEntry {
conn := mc.connections[i]
mc.connections = append(mc.connections[:i], mc.connections[i+1:]...)
if stopConnection {
mc.stopConnection(conn)
}
return conn
}
// addConnection adds a new connection.
func (mc *multiChannelNodeConn) addConnection(entry *connectionEntry) {
mutexWaitStart := time.Now()
mc.log.Debug().Caller().Str(zf.Chan, fmt.Sprintf("%p", entry.c)).Str(zf.ConnID, entry.id).
Msg("addConnection: waiting for mutex - POTENTIAL CONTENTION POINT")
mc.mutex.Lock()
mutexWaitDur := time.Since(mutexWaitStart)
defer mc.mutex.Unlock()
mc.connections = append(mc.connections, entry)
mc.log.Debug().Caller().Str(zf.Chan, fmt.Sprintf("%p", entry.c)).Str(zf.ConnID, entry.id).
Int("total_connections", len(mc.connections)).
Dur("mutex_wait_time", mutexWaitDur).
Msg("successfully added connection after mutex wait")
}
// removeConnectionByChannel removes a connection by matching channel pointer.
func (mc *multiChannelNodeConn) removeConnectionByChannel(c chan<- *tailcfg.MapResponse) bool {
mc.mutex.Lock()
defer mc.mutex.Unlock()
for i, entry := range mc.connections {
if entry.c == c {
mc.removeConnectionAtIndexLocked(i, false)
mc.log.Debug().Caller().Str(zf.Chan, fmt.Sprintf("%p", c)).
Int("remaining_connections", len(mc.connections)).
Msg("successfully removed connection")
return true
}
}
return false
}
// hasActiveConnections checks if the node has any active connections.
func (mc *multiChannelNodeConn) hasActiveConnections() bool {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
return len(mc.connections) > 0
}
// getActiveConnectionCount returns the number of active connections.
func (mc *multiChannelNodeConn) getActiveConnectionCount() int {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
return len(mc.connections)
}
// send broadcasts data to all active connections for the node.
func (mc *multiChannelNodeConn) send(data *tailcfg.MapResponse) error {
if data == nil {
return nil
}
mc.mutex.Lock()
defer mc.mutex.Unlock()
if len(mc.connections) == 0 {
// During rapid reconnection, nodes may temporarily have no active connections
// This is not an error - the node will receive a full map when it reconnects
mc.log.Debug().Caller().
Msg("send: skipping send to node with no active connections (likely rapid reconnection)")
return nil // Return success instead of error
}
mc.log.Debug().Caller().
Int("total_connections", len(mc.connections)).
Msg("send: broadcasting to all connections")
var lastErr error
successCount := 0
var failedConnections []int // Track failed connections for removal
// Send to all connections
for i, conn := range mc.connections {
mc.log.Debug().Caller().Str(zf.Chan, fmt.Sprintf("%p", conn.c)).
Str(zf.ConnID, conn.id).Int(zf.ConnectionIndex, i).
Msg("send: attempting to send to connection")
err := conn.send(data)
if err != nil {
lastErr = err
failedConnections = append(failedConnections, i)
mc.log.Warn().Err(err).Str(zf.Chan, fmt.Sprintf("%p", conn.c)).
Str(zf.ConnID, conn.id).Int(zf.ConnectionIndex, i).
Msg("send: connection send failed")
} else {
successCount++
mc.log.Debug().Caller().Str(zf.Chan, fmt.Sprintf("%p", conn.c)).
Str(zf.ConnID, conn.id).Int(zf.ConnectionIndex, i).
Msg("send: successfully sent to connection")
}
}
// Remove failed connections (in reverse order to maintain indices)
for i := len(failedConnections) - 1; i >= 0; i-- {
idx := failedConnections[i]
entry := mc.removeConnectionAtIndexLocked(idx, true)
mc.log.Debug().Caller().
Str(zf.ConnID, entry.id).
Msg("send: removed failed connection")
}
mc.updateCount.Add(1)
mc.log.Debug().
Int("successful_sends", successCount).
Int("failed_connections", len(failedConnections)).
Int("remaining_connections", len(mc.connections)).
Msg("send: completed broadcast")
// Success if at least one send succeeded
if successCount > 0 {
return nil
}
return fmt.Errorf("node %d: all connections failed, last error: %w", mc.id, lastErr)
}
// send sends data to a single connection entry with timeout-based stale connection detection.
func (entry *connectionEntry) send(data *tailcfg.MapResponse) error {
if data == nil {
return nil
}
// Check if the connection has been closed to prevent send on closed channel panic.
// This can happen during shutdown when Close() is called while workers are still processing.
if entry.closed.Load() {
return fmt.Errorf("connection %s: %w", entry.id, errConnectionClosed)
}
// Use a short timeout to detect stale connections where the client isn't reading the channel.
// This is critical for detecting Docker containers that are forcefully terminated
// but still have channels that appear open.
select {
case entry.c <- data:
// Update last used timestamp on successful send
entry.lastUsed.Store(time.Now().Unix())
return nil
case <-time.After(50 * time.Millisecond):
// Connection is likely stale - client isn't reading from channel
// This catches the case where Docker containers are killed but channels remain open
return fmt.Errorf("connection %s: %w", entry.id, ErrConnectionSendTimeout)
}
}
// nodeID returns the node ID.
func (mc *multiChannelNodeConn) nodeID() types.NodeID {
return mc.id
}
// version returns the capability version from the first active connection.
// All connections for a node should have the same version in practice.
func (mc *multiChannelNodeConn) version() tailcfg.CapabilityVersion {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
if len(mc.connections) == 0 {
return 0
}
return mc.connections[0].version
}
// updateSentPeers updates the tracked peer state based on a sent MapResponse.
// This must be called after successfully sending a response to keep track of
// what the client knows about, enabling accurate diffs for future updates.
func (mc *multiChannelNodeConn) updateSentPeers(resp *tailcfg.MapResponse) {
if resp == nil {
return
}
// Full peer list replaces tracked state entirely
if resp.Peers != nil {
mc.lastSentPeers.Clear()
for _, peer := range resp.Peers {
mc.lastSentPeers.Store(peer.ID, struct{}{})
}
}
// Incremental additions
for _, peer := range resp.PeersChanged {
mc.lastSentPeers.Store(peer.ID, struct{}{})
}
// Incremental removals
for _, id := range resp.PeersRemoved {
mc.lastSentPeers.Delete(id)
}
}
// computePeerDiff compares the current peer list against what was last sent
// and returns the peers that were removed (in lastSentPeers but not in current).
func (mc *multiChannelNodeConn) computePeerDiff(currentPeers []tailcfg.NodeID) []tailcfg.NodeID {
currentSet := make(map[tailcfg.NodeID]struct{}, len(currentPeers))
for _, id := range currentPeers {
currentSet[id] = struct{}{}
}
var removed []tailcfg.NodeID
// Find removed: in lastSentPeers but not in current
mc.lastSentPeers.Range(func(id tailcfg.NodeID, _ struct{}) bool {
if _, exists := currentSet[id]; !exists {
removed = append(removed, id)
}
return true
})
return removed
}
// change applies a change to all active connections for the node.
func (mc *multiChannelNodeConn) change(r change.Change) error {
return handleNodeChange(mc, mc.mapper, r)
}
// DebugNodeInfo contains debug information about a node's connections.
type DebugNodeInfo struct {
Connected bool `json:"connected"`
ActiveConnections int `json:"active_connections"`
}
// Debug returns a pre-baked map of node debug information for the debug interface.
func (b *LockFreeBatcher) Debug() map[types.NodeID]DebugNodeInfo {
result := make(map[types.NodeID]DebugNodeInfo)
// Get all nodes with their connection status using immediate connection logic
// (no grace period) for debug purposes
b.nodes.Range(func(id types.NodeID, nodeConn *multiChannelNodeConn) bool {
nodeConn.mutex.RLock()
activeConnCount := len(nodeConn.connections)
nodeConn.mutex.RUnlock()
// Use immediate connection status: if active connections exist, node is connected
// If not, check the connected map for nil (connected) vs timestamp (disconnected)
connected := false
if activeConnCount > 0 {
connected = true
} else {
// Check connected map for immediate status
if val, ok := b.connected.Load(id); ok && val == nil {
connected = true
}
}
result[id] = DebugNodeInfo{
Connected: connected,
ActiveConnections: activeConnCount,
}
return true
})
// Add all entries from the connected map to capture both connected and disconnected nodes
b.connected.Range(func(id types.NodeID, val *time.Time) bool {
// Only add if not already processed above
if _, exists := result[id]; !exists {
// Use immediate connection status for debug (no grace period)
connected := (val == nil) // nil means connected, timestamp means disconnected
result[id] = DebugNodeInfo{
Connected: connected,
ActiveConnections: 0,
}
}
return true
})
return result
}
func (b *LockFreeBatcher) DebugMapResponses() (map[types.NodeID][]tailcfg.MapResponse, error) {
return b.mapper.debugMapResponses()
}
// WorkErrors returns the count of work errors encountered.
// This is primarily useful for testing and debugging.
func (b *LockFreeBatcher) WorkErrors() int64 {
return b.workErrors.Load()
}
@@ -0,0 +1,887 @@
package mapper
// Scale benchmarks for the batcher system.
//
// These benchmarks systematically increase node counts to find scaling limits
// and identify bottlenecks. Organized into tiers:
//
// Tier 1 - O(1) operations: should stay flat regardless of node count
// Tier 2 - O(N) lightweight: batch queuing and processing (no MapResponse generation)
// Tier 3 - O(N) heavier: map building, peer diff, peer tracking
// Tier 4 - Concurrent contention: multi-goroutine access under load
//
// Node count progression: 100, 500, 1000, 2000, 5000, 10000, 20000, 50000
import (
"fmt"
"strconv"
"sync"
"testing"
"time"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/change"
"tailscale.com/tailcfg"
)
// scaleCounts defines the node counts used across all scaling benchmarks.
// Tier 1 (O(1)) tests up to 50k; Tier 2-4 test up to 10k-20k.
var (
scaleCountsO1 = []int{100, 500, 1000, 2000, 5000, 10000, 20000, 50000}
scaleCountsLinear = []int{100, 500, 1000, 2000, 5000, 10000}
scaleCountsHeavy = []int{100, 500, 1000, 2000, 5000, 10000}
scaleCountsConc = []int{100, 500, 1000, 2000, 5000}
)
// ============================================================================
// Tier 1: O(1) Operations — should scale flat
// ============================================================================
// BenchmarkScale_IsConnected tests single-node lookup at increasing map sizes.
func BenchmarkScale_IsConnected(b *testing.B) {
for _, n := range scaleCountsO1 {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
id := types.NodeID(1 + (i % n)) //nolint:gosec
_ = batcher.IsConnected(id)
}
})
}
}
// BenchmarkScale_AddToBatch_Targeted tests single-node targeted change at
// increasing map sizes. The map size should not affect per-operation cost.
func BenchmarkScale_AddToBatch_Targeted(b *testing.B) {
for _, n := range scaleCountsO1 {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
targetID := types.NodeID(1 + (i % n)) //nolint:gosec
ch := change.Change{
Reason: "scale-targeted",
TargetNode: targetID,
PeerPatches: []*tailcfg.PeerChange{
{NodeID: tailcfg.NodeID(targetID)}, //nolint:gosec
},
}
batcher.addToBatch(ch)
// Drain every 100 ops to avoid unbounded growth
if i%100 == 99 {
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
}
}
})
}
}
// BenchmarkScale_ConnectionChurn tests add/remove connection cycle.
// The map size should not affect per-operation cost for a single node.
func BenchmarkScale_ConnectionChurn(b *testing.B) {
for _, n := range scaleCountsO1 {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, channels := benchBatcher(n, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for i := range b.N {
id := types.NodeID(1 + (i % n)) //nolint:gosec
mc, ok := batcher.nodes.Load(id)
if !ok {
continue
}
oldCh := channels[id]
mc.removeConnectionByChannel(oldCh)
newCh := make(chan *tailcfg.MapResponse, 10)
entry := &connectionEntry{
id: fmt.Sprintf("sc-%d", i),
c: newCh,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
channels[id] = newCh
}
})
}
}
// ============================================================================
// Tier 2: O(N) Lightweight — batch mechanics without MapResponse generation
// ============================================================================
// BenchmarkScale_AddToBatch_Broadcast tests broadcasting a change to ALL nodes.
// Cost should scale linearly with node count.
func BenchmarkScale_AddToBatch_Broadcast(b *testing.B) {
for _, n := range scaleCountsLinear {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
ch := change.DERPMap()
b.ResetTimer()
for range b.N {
batcher.addToBatch(ch)
// Drain to avoid unbounded growth
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
}
})
}
}
// BenchmarkScale_AddToBatch_FullUpdate tests FullUpdate broadcast cost.
func BenchmarkScale_AddToBatch_FullUpdate(b *testing.B) {
for _, n := range scaleCountsLinear {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for range b.N {
batcher.addToBatch(change.FullUpdate())
}
})
}
}
// BenchmarkScale_ProcessBatchedChanges tests draining pending changes into work queue.
func BenchmarkScale_ProcessBatchedChanges(b *testing.B) {
for _, n := range scaleCountsLinear {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
batcher.workCh = make(chan work, n*b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
b.ResetTimer()
for range b.N {
b.StopTimer()
for i := 1; i <= n; i++ {
if nc, ok := batcher.nodes.Load(types.NodeID(i)); ok { //nolint:gosec
nc.appendPending(change.DERPMap())
}
}
b.StartTimer()
batcher.processBatchedChanges()
}
})
}
}
// BenchmarkScale_BroadcastToN tests end-to-end: addToBatch + processBatchedChanges.
func BenchmarkScale_BroadcastToN(b *testing.B) {
for _, n := range scaleCountsLinear {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
batcher.workCh = make(chan work, n*b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
ch := change.DERPMap()
b.ResetTimer()
for range b.N {
batcher.addToBatch(ch)
batcher.processBatchedChanges()
}
})
}
}
// BenchmarkScale_SendToAll tests raw channel send cost to N nodes (no batching).
// This isolates the multiChannelNodeConn.send() cost.
// Uses large buffered channels to avoid goroutine drain overhead.
func BenchmarkScale_SendToAll(b *testing.B) {
for _, n := range scaleCountsLinear {
b.Run(strconv.Itoa(n), func(b *testing.B) {
// b.N+1 buffer so sends never block
batcher, _ := benchBatcher(n, b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
data := testMapResponse()
b.ResetTimer()
for range b.N {
batcher.nodes.Range(func(_ types.NodeID, mc *multiChannelNodeConn) bool {
_ = mc.send(data)
return true
})
}
})
}
}
// ============================================================================
// Tier 3: O(N) Heavier — map building, peer diff, peer tracking
// ============================================================================
// BenchmarkScale_ConnectedMap tests building the full connected/disconnected map.
func BenchmarkScale_ConnectedMap(b *testing.B) {
for _, n := range scaleCountsHeavy {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, channels := benchBatcher(n, 1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
// 10% disconnected for realism
for i := 1; i <= n; i++ {
if i%10 == 0 {
id := types.NodeID(i) //nolint:gosec
if mc, ok := batcher.nodes.Load(id); ok {
mc.removeConnectionByChannel(channels[id])
mc.markDisconnected()
}
}
}
b.ResetTimer()
for range b.N {
_ = batcher.ConnectedMap()
}
})
}
}
// BenchmarkScale_ComputePeerDiff tests peer diff computation at scale.
// Each node tracks N-1 peers, with 10% removed.
func BenchmarkScale_ComputePeerDiff(b *testing.B) {
for _, n := range scaleCountsHeavy {
b.Run(strconv.Itoa(n), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
// Track N peers
for i := 1; i <= n; i++ {
mc.lastSentPeers.Store(tailcfg.NodeID(i), struct{}{})
}
// Current: 90% present (every 10th missing)
current := make([]tailcfg.NodeID, 0, n)
for i := 1; i <= n; i++ {
if i%10 != 0 {
current = append(current, tailcfg.NodeID(i))
}
}
b.ResetTimer()
for range b.N {
_ = mc.computePeerDiff(current)
}
})
}
}
// BenchmarkScale_UpdateSentPeers_Full tests full peer list update.
func BenchmarkScale_UpdateSentPeers_Full(b *testing.B) {
for _, n := range scaleCountsHeavy {
b.Run(strconv.Itoa(n), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
peerIDs := make([]tailcfg.NodeID, n)
for i := range peerIDs {
peerIDs[i] = tailcfg.NodeID(i + 1)
}
resp := testMapResponseWithPeers(peerIDs...)
b.ResetTimer()
for range b.N {
mc.updateSentPeers(resp)
}
})
}
}
// BenchmarkScale_UpdateSentPeers_Incremental tests incremental peer updates (10% new).
func BenchmarkScale_UpdateSentPeers_Incremental(b *testing.B) {
for _, n := range scaleCountsHeavy {
b.Run(strconv.Itoa(n), func(b *testing.B) {
mc := newMultiChannelNodeConn(1, nil)
// Pre-populate
for i := 1; i <= n; i++ {
mc.lastSentPeers.Store(tailcfg.NodeID(i), struct{}{})
}
addCount := n / 10
if addCount == 0 {
addCount = 1
}
resp := testMapResponse()
resp.PeersChanged = make([]*tailcfg.Node, addCount)
for i := range addCount {
resp.PeersChanged[i] = &tailcfg.Node{ID: tailcfg.NodeID(n + i + 1)}
}
b.ResetTimer()
for range b.N {
mc.updateSentPeers(resp)
}
})
}
}
// BenchmarkScale_MultiChannelBroadcast tests sending to N nodes, each with
// ~1.6 connections on average (every 3rd node has 3 connections).
// Uses large buffered channels to avoid goroutine drain overhead.
func BenchmarkScale_MultiChannelBroadcast(b *testing.B) {
for _, n := range scaleCountsHeavy {
b.Run(strconv.Itoa(n), func(b *testing.B) {
// Use b.N+1 buffer so sends never block
batcher, _ := benchBatcher(n, b.N+1)
defer func() {
close(batcher.done)
batcher.tick.Stop()
}()
// Add extra connections to every 3rd node (also buffered)
for i := 1; i <= n; i++ {
if i%3 == 0 {
if mc, ok := batcher.nodes.Load(types.NodeID(i)); ok { //nolint:gosec
for j := range 2 {
ch := make(chan *tailcfg.MapResponse, b.N+1)
entry := &connectionEntry{
id: fmt.Sprintf("extra-%d-%d", i, j),
c: ch,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
}
}
}
}
data := testMapResponse()
b.ResetTimer()
for range b.N {
batcher.nodes.Range(func(_ types.NodeID, mc *multiChannelNodeConn) bool {
_ = mc.send(data)
return true
})
}
})
}
}
// ============================================================================
// Tier 4: Concurrent Contention — multi-goroutine access
// ============================================================================
// BenchmarkScale_ConcurrentAddToBatch tests parallel addToBatch throughput.
func BenchmarkScale_ConcurrentAddToBatch(b *testing.B) {
for _, n := range scaleCountsConc {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, _ := benchBatcher(n, 10)
drainDone := make(chan struct{})
go func() {
defer close(drainDone)
for {
select {
case <-batcher.done:
return
default:
batcher.nodes.Range(func(_ types.NodeID, nc *multiChannelNodeConn) bool {
nc.drainPending()
return true
})
time.Sleep(time.Millisecond) //nolint:forbidigo
}
}
}()
ch := change.DERPMap()
b.ResetTimer()
b.RunParallel(func(pb *testing.PB) {
for pb.Next() {
batcher.addToBatch(ch)
}
})
b.StopTimer()
close(batcher.done)
<-drainDone
batcher.done = make(chan struct{})
batcher.tick.Stop()
})
}
}
// BenchmarkScale_ConcurrentSendAndChurn tests the production hot path:
// sending to all nodes while 10% of connections are churning concurrently.
// Uses large buffered channels to avoid goroutine drain overhead.
func BenchmarkScale_ConcurrentSendAndChurn(b *testing.B) {
for _, n := range scaleCountsConc {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, channels := benchBatcher(n, b.N+1)
var mu sync.Mutex
stopChurn := make(chan struct{})
go func() {
i := 0
for {
select {
case <-stopChurn:
return
default:
id := types.NodeID(1 + (i % n)) //nolint:gosec
if i%10 == 0 {
mc, ok := batcher.nodes.Load(id)
if ok {
mu.Lock()
oldCh := channels[id]
mu.Unlock()
mc.removeConnectionByChannel(oldCh)
newCh := make(chan *tailcfg.MapResponse, b.N+1)
entry := &connectionEntry{
id: fmt.Sprintf("sc-churn-%d", i),
c: newCh,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
mu.Lock()
channels[id] = newCh
mu.Unlock()
}
}
i++
}
}
}()
data := testMapResponse()
b.ResetTimer()
for range b.N {
batcher.nodes.Range(func(_ types.NodeID, mc *multiChannelNodeConn) bool {
_ = mc.send(data)
return true
})
}
b.StopTimer()
close(stopChurn)
close(batcher.done)
batcher.tick.Stop()
})
}
}
// BenchmarkScale_MixedWorkload simulates a realistic production workload:
// - 70% targeted changes (single node updates)
// - 20% DERP map changes (broadcast)
// - 10% full updates (broadcast with full map)
// All while 10% of connections are churning.
func BenchmarkScale_MixedWorkload(b *testing.B) {
for _, n := range scaleCountsConc {
b.Run(strconv.Itoa(n), func(b *testing.B) {
batcher, channels := benchBatcher(n, 10)
batcher.workCh = make(chan work, n*100+1)
var mu sync.Mutex
stopChurn := make(chan struct{})
// Background churn on 10% of nodes
go func() {
i := 0
for {
select {
case <-stopChurn:
return
default:
id := types.NodeID(1 + (i % n)) //nolint:gosec
if i%10 == 0 {
mc, ok := batcher.nodes.Load(id)
if ok {
mu.Lock()
oldCh := channels[id]
mu.Unlock()
mc.removeConnectionByChannel(oldCh)
newCh := make(chan *tailcfg.MapResponse, 10)
entry := &connectionEntry{
id: fmt.Sprintf("mix-churn-%d", i),
c: newCh,
version: tailcfg.CapabilityVersion(100),
created: time.Now(),
}
entry.lastUsed.Store(time.Now().Unix())
mc.addConnection(entry)
mu.Lock()
channels[id] = newCh
mu.Unlock()
}
}
i++
}
}
}()
// Background batch processor
stopProc := make(chan struct{})
go func() {
for {
select {
case <-stopProc:
return
default:
batcher.processBatchedChanges()
time.Sleep(time.Millisecond) //nolint:forbidigo
}
}
}()
// Background work channel consumer (simulates workers)
stopWorkers := make(chan struct{})
go func() {
for {
select {
case <-batcher.workCh:
case <-stopWorkers:
return
}
}
}()
b.ResetTimer()
for i := range b.N {
switch {
case i%10 < 7: // 70% targeted
targetID := types.NodeID(1 + (i % n)) //nolint:gosec
batcher.addToBatch(change.Change{
Reason: "mixed-targeted",
TargetNode: targetID,
PeerPatches: []*tailcfg.PeerChange{
{NodeID: tailcfg.NodeID(targetID)}, //nolint:gosec
},
})
case i%10 < 9: // 20% DERP map broadcast
batcher.addToBatch(change.DERPMap())
default: // 10% full update
batcher.addToBatch(change.FullUpdate())
}
}
b.StopTimer()
close(stopChurn)
close(stopProc)
close(stopWorkers)
close(batcher.done)
batcher.tick.Stop()
})
}
}
// ============================================================================
// Tier 5: DB-dependent — AddNode with real MapResponse generation
// ============================================================================
// BenchmarkScale_AddAllNodes measures the cost of connecting ALL N nodes
// to a batcher backed by a real database. Each AddNode generates an initial
// MapResponse containing all peer data, so cost is O(N) per node, O(N²) total.
func BenchmarkScale_AddAllNodes(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 50, 100, 200, 500} {
b.Run(strconv.Itoa(nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
b.ResetTimer()
for range b.N {
for i := range allNodes {
node := &allNodes[i]
_ = batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
}
b.StopTimer()
for i := range allNodes {
node := &allNodes[i]
batcher.RemoveNode(node.n.ID, node.ch)
}
for i := range allNodes {
for {
select {
case <-allNodes[i].ch:
default:
goto drained
}
}
drained:
}
b.StartTimer()
}
})
}
}
// BenchmarkScale_SingleAddNode measures the cost of adding ONE node to an
// already-populated batcher. This is the real production scenario: a new node
// joins an existing network. The cost should scale with the number of existing
// peers since the initial MapResponse includes all peer data.
func BenchmarkScale_SingleAddNode(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 50, 100, 200, 500, 1000} {
b.Run(strconv.Itoa(nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
// Connect all nodes except the last one
for i := range len(allNodes) - 1 {
node := &allNodes[i]
err := batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
if err != nil {
b.Fatalf("failed to add node %d: %v", i, err)
}
}
time.Sleep(200 * time.Millisecond) //nolint:forbidigo
// Benchmark: repeatedly add and remove the last node
lastNode := &allNodes[len(allNodes)-1]
b.ResetTimer()
for range b.N {
_ = batcher.AddNode(lastNode.n.ID, lastNode.ch, tailcfg.CapabilityVersion(100), nil)
b.StopTimer()
batcher.RemoveNode(lastNode.n.ID, lastNode.ch)
for {
select {
case <-lastNode.ch:
default:
goto drainDone
}
}
drainDone:
b.StartTimer()
}
})
}
}
// BenchmarkScale_MapResponse_DERPMap measures MapResponse generation for a
// DERPMap change. This is a lightweight change that doesn't touch peers.
func BenchmarkScale_MapResponse_DERPMap(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 50, 100, 200, 500} {
b.Run(strconv.Itoa(nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
for i := range allNodes {
node := &allNodes[i]
err := batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
if err != nil {
b.Fatalf("failed to add node %d: %v", i, err)
}
}
time.Sleep(200 * time.Millisecond) //nolint:forbidigo
ch := change.DERPMap()
b.ResetTimer()
for i := range b.N {
nodeIdx := i % len(allNodes)
_, _ = batcher.MapResponseFromChange(allNodes[nodeIdx].n.ID, ch)
}
})
}
}
// BenchmarkScale_MapResponse_FullUpdate measures MapResponse generation for a
// FullUpdate change. This forces full peer serialization — the primary bottleneck
// for large networks.
func BenchmarkScale_MapResponse_FullUpdate(b *testing.B) {
if testing.Short() {
b.Skip("skipping full pipeline benchmark in short mode")
}
for _, nodeCount := range []int{10, 50, 100, 200, 500} {
b.Run(strconv.Itoa(nodeCount), func(b *testing.B) {
testData, cleanup := setupBatcherWithTestData(b, NewBatcherAndMapper, 1, nodeCount, largeBufferSize)
defer cleanup()
batcher := testData.Batcher
allNodes := testData.Nodes
for i := range allNodes {
allNodes[i].start()
}
defer func() {
for i := range allNodes {
allNodes[i].cleanup()
}
}()
for i := range allNodes {
node := &allNodes[i]
err := batcher.AddNode(node.n.ID, node.ch, tailcfg.CapabilityVersion(100), nil)
if err != nil {
b.Fatalf("failed to add node %d: %v", i, err)
}
}
time.Sleep(200 * time.Millisecond) //nolint:forbidigo
ch := change.FullUpdate()
b.ResetTimer()
for i := range b.N {
nodeIdx := i % len(allNodes)
_, _ = batcher.MapResponseFromChange(allNodes[nodeIdx].n.ID, ch)
}
})
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+17 -9
View File
@@ -2,6 +2,7 @@ package mapper
import (
"net/netip"
"slices"
"sort"
"time"
@@ -78,7 +79,10 @@ func (b *MapResponseBuilder) WithSelfNode() *MapResponseBuilder {
tailnode, err := nv.TailNode(
b.capVer,
func(id types.NodeID) []netip.Prefix {
return policy.ReduceRoutes(nv, b.mapper.state.GetNodePrimaryRoutes(id), matchers)
// Self node: include own primaries + exit routes (no via steering for self).
primaries := policy.ReduceRoutes(nv, b.mapper.state.GetNodePrimaryRoutes(id), matchers)
return slices.Concat(primaries, nv.ExitRoutes())
},
b.mapper.cfg)
if err != nil {
@@ -251,14 +255,18 @@ func (b *MapResponseBuilder) buildTailPeers(peers views.Slice[types.NodeView]) (
changedViews = peers
}
tailPeers, err := types.TailNodes(
changedViews, b.capVer,
func(id types.NodeID) []netip.Prefix {
return policy.ReduceRoutes(node, b.mapper.state.GetNodePrimaryRoutes(id), matchers)
},
b.mapper.cfg)
if err != nil {
return nil, err
// Build tail nodes with per-peer via-aware route function.
tailPeers := make([]*tailcfg.Node, 0, changedViews.Len())
for _, peer := range changedViews.All() {
tn, err := peer.TailNode(b.capVer, func(_ types.NodeID) []netip.Prefix {
return b.mapper.state.RoutesForPeer(node, peer, matchers)
}, b.mapper.cfg)
if err != nil {
return nil, err
}
tailPeers = append(tailPeers, tn)
}
// Peers is always returned sorted by Node.ID.
+1 -1
View File
@@ -44,7 +44,7 @@ type mapper struct {
// Configuration
state *state.State
cfg *types.Config
batcher Batcher
batcher *Batcher
created time.Time
}
+438
View File
@@ -0,0 +1,438 @@
package mapper
import (
"fmt"
"strconv"
"sync"
"sync/atomic"
"time"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/change"
"github.com/juanfont/headscale/hscontrol/util/zlog/zf"
"github.com/puzpuzpuz/xsync/v4"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"tailscale.com/tailcfg"
)
// connectionEntry represents a single connection to a node.
type connectionEntry struct {
id string // unique connection ID
c chan<- *tailcfg.MapResponse
version tailcfg.CapabilityVersion
created time.Time
stop func()
lastUsed atomic.Int64 // Unix timestamp of last successful send
closed atomic.Bool // Indicates if this connection has been closed
}
// multiChannelNodeConn manages multiple concurrent connections for a single node.
type multiChannelNodeConn struct {
id types.NodeID
mapper *mapper
log zerolog.Logger
mutex sync.RWMutex
connections []*connectionEntry
// pendingMu protects pending changes independently of the connection mutex.
// This avoids contention between addToBatch (which appends changes) and
// send() (which sends data to connections).
pendingMu sync.Mutex
pending []change.Change
// workMu serializes change processing for this node across batch ticks.
// Without this, two workers could process consecutive ticks' bundles
// concurrently, causing out-of-order MapResponse delivery and races
// on lastSentPeers (Clear+Store in updateSentPeers vs Range in
// computePeerDiff).
workMu sync.Mutex
closeOnce sync.Once
updateCount atomic.Int64
// disconnectedAt records when the last connection was removed.
// nil means the node is considered connected (or newly created);
// non-nil means the node disconnected at the stored timestamp.
// Used by cleanupOfflineNodes to evict stale entries.
disconnectedAt atomic.Pointer[time.Time]
// lastSentPeers tracks which peers were last sent to this node.
// This enables computing diffs for policy changes instead of sending
// full peer lists (which clients interpret as "no change" when empty).
// Using xsync.Map for lock-free concurrent access.
lastSentPeers *xsync.Map[tailcfg.NodeID, struct{}]
}
// connIDCounter is a monotonically increasing counter used to generate
// unique connection identifiers without the overhead of crypto/rand.
// Connection IDs are process-local and need not be cryptographically random.
var connIDCounter atomic.Uint64
// generateConnectionID generates a unique connection identifier.
func generateConnectionID() string {
return strconv.FormatUint(connIDCounter.Add(1), 10)
}
// newMultiChannelNodeConn creates a new multi-channel node connection.
func newMultiChannelNodeConn(id types.NodeID, mapper *mapper) *multiChannelNodeConn {
return &multiChannelNodeConn{
id: id,
mapper: mapper,
lastSentPeers: xsync.NewMap[tailcfg.NodeID, struct{}](),
log: log.With().Uint64(zf.NodeID, id.Uint64()).Logger(),
}
}
func (mc *multiChannelNodeConn) close() {
mc.closeOnce.Do(func() {
mc.mutex.Lock()
defer mc.mutex.Unlock()
for _, conn := range mc.connections {
mc.stopConnection(conn)
}
})
}
// stopConnection marks a connection as closed and tears down the owning session
// at most once, even if multiple cleanup paths race to remove it.
func (mc *multiChannelNodeConn) stopConnection(conn *connectionEntry) {
if conn.closed.CompareAndSwap(false, true) {
if conn.stop != nil {
conn.stop()
}
}
}
// removeConnectionAtIndexLocked removes the active connection at index.
// If stopConnection is true, it also stops that session.
// Caller must hold mc.mutex.
func (mc *multiChannelNodeConn) removeConnectionAtIndexLocked(i int, stopConnection bool) *connectionEntry {
conn := mc.connections[i]
copy(mc.connections[i:], mc.connections[i+1:])
mc.connections[len(mc.connections)-1] = nil // release pointer for GC
mc.connections = mc.connections[:len(mc.connections)-1]
if stopConnection {
mc.stopConnection(conn)
}
return conn
}
// addConnection adds a new connection.
func (mc *multiChannelNodeConn) addConnection(entry *connectionEntry) {
mc.mutex.Lock()
defer mc.mutex.Unlock()
mc.connections = append(mc.connections, entry)
mc.log.Debug().Str(zf.ConnID, entry.id).
Int("total_connections", len(mc.connections)).
Msg("connection added")
}
// removeConnectionByChannel removes a connection by matching channel pointer.
func (mc *multiChannelNodeConn) removeConnectionByChannel(c chan<- *tailcfg.MapResponse) bool {
mc.mutex.Lock()
defer mc.mutex.Unlock()
for i, entry := range mc.connections {
if entry.c == c {
mc.removeConnectionAtIndexLocked(i, false)
mc.log.Debug().Str(zf.ConnID, entry.id).
Int("remaining_connections", len(mc.connections)).
Msg("connection removed")
return true
}
}
return false
}
// hasActiveConnections checks if the node has any active connections.
func (mc *multiChannelNodeConn) hasActiveConnections() bool {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
return len(mc.connections) > 0
}
// getActiveConnectionCount returns the number of active connections.
func (mc *multiChannelNodeConn) getActiveConnectionCount() int {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
return len(mc.connections)
}
// markConnected clears the disconnect timestamp, indicating the node
// has an active connection.
func (mc *multiChannelNodeConn) markConnected() {
mc.disconnectedAt.Store(nil)
}
// markDisconnected records the current time as the moment the node
// lost its last connection. Used by cleanupOfflineNodes to determine
// how long the node has been offline.
func (mc *multiChannelNodeConn) markDisconnected() {
now := time.Now()
mc.disconnectedAt.Store(&now)
}
// isConnected returns true if the node has active connections or has
// not been marked as disconnected.
func (mc *multiChannelNodeConn) isConnected() bool {
if mc.hasActiveConnections() {
return true
}
return mc.disconnectedAt.Load() == nil
}
// offlineDuration returns how long the node has been disconnected.
// Returns 0 if the node is connected or has never been marked as disconnected.
func (mc *multiChannelNodeConn) offlineDuration() time.Duration {
t := mc.disconnectedAt.Load()
if t == nil {
return 0
}
return time.Since(*t)
}
// appendPending appends changes to this node's pending change list.
// Thread-safe via pendingMu; does not contend with the connection mutex.
func (mc *multiChannelNodeConn) appendPending(changes ...change.Change) {
mc.pendingMu.Lock()
mc.pending = append(mc.pending, changes...)
mc.pendingMu.Unlock()
}
// drainPending atomically removes and returns all pending changes.
// Returns nil if there are no pending changes.
func (mc *multiChannelNodeConn) drainPending() []change.Change {
mc.pendingMu.Lock()
p := mc.pending
mc.pending = nil
mc.pendingMu.Unlock()
return p
}
// send broadcasts data to all active connections for the node.
//
// To avoid holding the write lock during potentially slow sends (each stale
// connection can block for up to 50ms), the method snapshots connections under
// a read lock, sends without any lock held, then write-locks only to remove
// failures. New connections added between the snapshot and cleanup are safe:
// they receive a full initial map via AddNode, so missing this update causes
// no data loss.
func (mc *multiChannelNodeConn) send(data *tailcfg.MapResponse) error {
if data == nil {
return nil
}
// Snapshot connections under read lock.
mc.mutex.RLock()
if len(mc.connections) == 0 {
mc.mutex.RUnlock()
mc.log.Trace().
Msg("send: no active connections, skipping")
return nil
}
// Copy the slice so we can release the read lock before sending.
snapshot := make([]*connectionEntry, len(mc.connections))
copy(snapshot, mc.connections)
mc.mutex.RUnlock()
mc.log.Trace().
Int("total_connections", len(snapshot)).
Msg("send: broadcasting")
// Send to all connections without holding any lock.
// Stale connection timeouts (50ms each) happen here without blocking
// other goroutines that need the mutex.
var (
lastErr error
successCount int
failed []*connectionEntry
)
for _, conn := range snapshot {
err := conn.send(data)
if err != nil {
lastErr = err
failed = append(failed, conn)
mc.log.Warn().Err(err).
Str(zf.ConnID, conn.id).
Msg("send: connection failed")
} else {
successCount++
}
}
// Write-lock only to remove failed connections.
if len(failed) > 0 {
mc.mutex.Lock()
// Remove by pointer identity: only remove entries that still exist
// in the current connections slice and match a failed pointer.
// New connections added since the snapshot are not affected.
failedSet := make(map[*connectionEntry]struct{}, len(failed))
for _, f := range failed {
failedSet[f] = struct{}{}
}
clean := mc.connections[:0]
for _, conn := range mc.connections {
if _, isFailed := failedSet[conn]; !isFailed {
clean = append(clean, conn)
} else {
mc.log.Debug().
Str(zf.ConnID, conn.id).
Msg("send: removing failed connection")
// Tear down the owning session so the old serveLongPoll
// goroutine exits instead of lingering as a stale session.
mc.stopConnection(conn)
}
}
// Nil out trailing slots so removed *connectionEntry values
// are not retained by the backing array.
for i := len(clean); i < len(mc.connections); i++ {
mc.connections[i] = nil
}
mc.connections = clean
mc.mutex.Unlock()
}
mc.updateCount.Add(1)
mc.log.Trace().
Int("successful_sends", successCount).
Int("failed_connections", len(failed)).
Msg("send: broadcast complete")
// Success if at least one send succeeded
if successCount > 0 {
return nil
}
return fmt.Errorf("node %d: all connections failed, last error: %w", mc.id, lastErr)
}
// send sends data to a single connection entry with timeout-based stale connection detection.
func (entry *connectionEntry) send(data *tailcfg.MapResponse) error {
if data == nil {
return nil
}
// Check if the connection has been closed to prevent send on closed channel panic.
// This can happen during shutdown when Close() is called while workers are still processing.
if entry.closed.Load() {
return fmt.Errorf("connection %s: %w", entry.id, errConnectionClosed)
}
// Use a short timeout to detect stale connections where the client isn't reading the channel.
// This is critical for detecting Docker containers that are forcefully terminated
// but still have channels that appear open.
//
// We use time.NewTimer + Stop instead of time.After to avoid leaking timers.
// time.After creates a timer that lives in the runtime's timer heap until it fires,
// even when the send succeeds immediately. On the hot path (1000+ nodes per tick),
// this leaks thousands of timers per second.
timer := time.NewTimer(50 * time.Millisecond) //nolint:mnd
defer timer.Stop()
select {
case entry.c <- data:
// Update last used timestamp on successful send
entry.lastUsed.Store(time.Now().Unix())
return nil
case <-timer.C:
// Connection is likely stale - client isn't reading from channel
// This catches the case where Docker containers are killed but channels remain open
return fmt.Errorf("connection %s: %w", entry.id, ErrConnectionSendTimeout)
}
}
// nodeID returns the node ID.
func (mc *multiChannelNodeConn) nodeID() types.NodeID {
return mc.id
}
// version returns the capability version from the first active connection.
// All connections for a node should have the same version in practice.
func (mc *multiChannelNodeConn) version() tailcfg.CapabilityVersion {
mc.mutex.RLock()
defer mc.mutex.RUnlock()
if len(mc.connections) == 0 {
return 0
}
return mc.connections[0].version
}
// updateSentPeers updates the tracked peer state based on a sent MapResponse.
// This must be called after successfully sending a response to keep track of
// what the client knows about, enabling accurate diffs for future updates.
func (mc *multiChannelNodeConn) updateSentPeers(resp *tailcfg.MapResponse) {
if resp == nil {
return
}
// Full peer list replaces tracked state entirely
if resp.Peers != nil {
mc.lastSentPeers.Clear()
for _, peer := range resp.Peers {
mc.lastSentPeers.Store(peer.ID, struct{}{})
}
}
// Incremental additions
for _, peer := range resp.PeersChanged {
mc.lastSentPeers.Store(peer.ID, struct{}{})
}
// Incremental removals
for _, id := range resp.PeersRemoved {
mc.lastSentPeers.Delete(id)
}
}
// computePeerDiff compares the current peer list against what was last sent
// and returns the peers that were removed (in lastSentPeers but not in current).
func (mc *multiChannelNodeConn) computePeerDiff(currentPeers []tailcfg.NodeID) []tailcfg.NodeID {
currentSet := make(map[tailcfg.NodeID]struct{}, len(currentPeers))
for _, id := range currentPeers {
currentSet[id] = struct{}{}
}
var removed []tailcfg.NodeID
// Find removed: in lastSentPeers but not in current
mc.lastSentPeers.Range(func(id tailcfg.NodeID, _ struct{}) bool {
if _, exists := currentSet[id]; !exists {
removed = append(removed, id)
}
return true
})
return removed
}
// change applies a change to all active connections for the node.
func (mc *multiChannelNodeConn) change(r change.Change) error {
return handleNodeChange(mc, mc.mapper, r)
}
+21 -11
View File
@@ -3,6 +3,7 @@ package mapper
import (
"encoding/json"
"net/netip"
"slices"
"testing"
"time"
@@ -74,9 +75,11 @@ func TestTailNode(t *testing.T) {
MachineAuthorized: true,
CapMap: tailcfg.NodeCapMap{
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveShare: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveAccess: []tailcfg.RawMessage{},
},
},
wantErr: false,
@@ -163,9 +166,11 @@ func TestTailNode(t *testing.T) {
MachineAuthorized: true,
CapMap: tailcfg.NodeCapMap{
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveShare: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveAccess: []tailcfg.RawMessage{},
},
},
wantErr: false,
@@ -188,9 +193,11 @@ func TestTailNode(t *testing.T) {
MachineAuthorized: true,
CapMap: tailcfg.NodeCapMap{
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.CapabilityFileSharing: []tailcfg.RawMessage{},
tailcfg.CapabilityAdmin: []tailcfg.RawMessage{},
tailcfg.CapabilitySSH: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveShare: []tailcfg.RawMessage{},
tailcfg.NodeAttrsTaildriveAccess: []tailcfg.RawMessage{},
},
},
wantErr: false,
@@ -214,10 +221,13 @@ func TestTailNode(t *testing.T) {
// This is a hack to avoid having a second node to test the primary route.
// This should be baked into the test case proper if it is extended in the future.
_ = primary.SetRoutes(2, netip.MustParsePrefix("192.168.0.0/24"))
got, err := tt.node.View().TailNode(
nv := tt.node.View()
got, err := nv.TailNode(
0,
func(id types.NodeID) []netip.Prefix {
return primary.PrimaryRoutes(id)
// Route function returns primaries + exit routes
// (matching the real caller contract).
return slices.Concat(primary.PrimaryRoutes(id), nv.ExitRoutes())
},
cfg,
)
+24 -11
View File
@@ -46,6 +46,12 @@ const (
// of length. Then that many bytes of JSON-encoded tailcfg.EarlyNoise.
// The early payload is optional. Some servers may not send it... But we do!
earlyPayloadMagic = "\xff\xff\xffTS"
// noiseBodyLimit is the maximum allowed request body size for Noise protocol
// handlers. This prevents unauthenticated OOM attacks via unbounded io.ReadAll.
// No legitimate Noise request (MapRequest, RegisterRequest, etc.) comes close
// to this limit; typical payloads are a few KB.
noiseBodyLimit int64 = 1048576 // 1 MiB
)
type noiseServer struct {
@@ -110,6 +116,17 @@ func (h *Headscale) NoiseUpgradeHandler(
// a single hijacked connection from /ts2021, using netutil.NewOneConnListener
r := chi.NewRouter()
// Limit request body size to prevent unauthenticated OOM attacks.
// The Noise handshake accepts any machine key without checking
// registration, so all endpoints behind this router are reachable
// without credentials.
r.Use(func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, noiseBodyLimit)
next.ServeHTTP(w, r)
})
})
r.Use(metrics.Collector(metrics.CollectorOpts{
Host: false,
Proto: true,
@@ -251,8 +268,7 @@ func rejectUnsupported(
}
func (ns *noiseServer) NotImplementedHandler(writer http.ResponseWriter, req *http.Request) {
d, _ := io.ReadAll(req.Body)
log.Trace().Caller().Str("path", req.URL.String()).Bytes("body", d).Msgf("not implemented handler hit")
log.Trace().Caller().Str("path", req.URL.String()).Msg("not implemented handler hit")
http.Error(writer, "Not implemented yet", http.StatusNotImplemented)
}
@@ -535,10 +551,10 @@ func (ns *noiseServer) PollNetMapHandler(
writer http.ResponseWriter,
req *http.Request,
) {
body, _ := io.ReadAll(req.Body)
var mapRequest tailcfg.MapRequest
if err := json.Unmarshal(body, &mapRequest); err != nil { //nolint:noinlineerr
err := json.NewDecoder(req.Body).Decode(&mapRequest)
if err != nil {
httpError(writer, err)
return
}
@@ -584,13 +600,10 @@ func (ns *noiseServer) RegistrationHandler(
registerRequest, registerResponse := func() (*tailcfg.RegisterRequest, *tailcfg.RegisterResponse) { //nolint:contextcheck
var resp *tailcfg.RegisterResponse
body, err := io.ReadAll(req.Body)
if err != nil {
return &tailcfg.RegisterRequest{}, regErr(err)
}
var regReq tailcfg.RegisterRequest
if err := json.Unmarshal(body, &regReq); err != nil { //nolint:noinlineerr
err := json.NewDecoder(req.Body).Decode(&regReq)
if err != nil {
return &regReq, regErr(err)
}
+195
View File
@@ -0,0 +1,195 @@
package hscontrol
import (
"bytes"
"context"
"encoding/json"
"io"
"net/http"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"tailscale.com/tailcfg"
)
// newNoiseRouterWithBodyLimit builds a chi router with the same body-limit
// middleware used in the real Noise router but wired to a test handler that
// captures the io.ReadAll result. This lets us verify the limit without
// needing a full Headscale instance.
func newNoiseRouterWithBodyLimit(readBody *[]byte, readErr *error) http.Handler {
r := chi.NewRouter()
r.Use(func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Body = http.MaxBytesReader(w, r.Body, noiseBodyLimit)
next.ServeHTTP(w, r)
})
})
handler := func(w http.ResponseWriter, r *http.Request) {
*readBody, *readErr = io.ReadAll(r.Body)
if *readErr != nil {
http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
return
}
w.WriteHeader(http.StatusOK)
}
r.Post("/machine/map", handler)
r.Post("/machine/register", handler)
return r
}
func TestNoiseBodyLimit_MapEndpoint(t *testing.T) {
t.Parallel()
t.Run("normal_map_request", func(t *testing.T) {
t.Parallel()
var body []byte
var readErr error
router := newNoiseRouterWithBodyLimit(&body, &readErr)
mapReq := tailcfg.MapRequest{Version: 100, Stream: true}
payload, err := json.Marshal(mapReq)
require.NoError(t, err)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/map", bytes.NewReader(payload))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
require.NoError(t, readErr)
assert.Equal(t, http.StatusOK, rec.Code)
assert.Len(t, body, len(payload))
})
t.Run("oversized_body_rejected", func(t *testing.T) {
t.Parallel()
var body []byte
var readErr error
router := newNoiseRouterWithBodyLimit(&body, &readErr)
oversized := bytes.Repeat([]byte("x"), int(noiseBodyLimit)+1)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/map", bytes.NewReader(oversized))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
require.Error(t, readErr)
assert.Equal(t, http.StatusRequestEntityTooLarge, rec.Code)
assert.LessOrEqual(t, len(body), int(noiseBodyLimit))
})
}
func TestNoiseBodyLimit_RegisterEndpoint(t *testing.T) {
t.Parallel()
t.Run("normal_register_request", func(t *testing.T) {
t.Parallel()
var body []byte
var readErr error
router := newNoiseRouterWithBodyLimit(&body, &readErr)
regReq := tailcfg.RegisterRequest{Version: 100}
payload, err := json.Marshal(regReq)
require.NoError(t, err)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/register", bytes.NewReader(payload))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
require.NoError(t, readErr)
assert.Equal(t, http.StatusOK, rec.Code)
assert.Len(t, body, len(payload))
})
t.Run("oversized_body_rejected", func(t *testing.T) {
t.Parallel()
var body []byte
var readErr error
router := newNoiseRouterWithBodyLimit(&body, &readErr)
oversized := bytes.Repeat([]byte("x"), int(noiseBodyLimit)+1)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/register", bytes.NewReader(oversized))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
require.Error(t, readErr)
assert.Equal(t, http.StatusRequestEntityTooLarge, rec.Code)
assert.LessOrEqual(t, len(body), int(noiseBodyLimit))
})
}
func TestNoiseBodyLimit_AtExactLimit(t *testing.T) {
t.Parallel()
var body []byte
var readErr error
router := newNoiseRouterWithBodyLimit(&body, &readErr)
payload := bytes.Repeat([]byte("a"), int(noiseBodyLimit))
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/map", bytes.NewReader(payload))
rec := httptest.NewRecorder()
router.ServeHTTP(rec, req)
require.NoError(t, readErr)
assert.Equal(t, http.StatusOK, rec.Code)
assert.Len(t, body, int(noiseBodyLimit))
}
// TestPollNetMapHandler_OversizedBody calls the real handler with a
// MaxBytesReader-wrapped body to verify it fails gracefully (json decode
// error on truncated data) rather than consuming unbounded memory.
func TestPollNetMapHandler_OversizedBody(t *testing.T) {
t.Parallel()
ns := &noiseServer{}
oversized := bytes.Repeat([]byte("x"), int(noiseBodyLimit)+1)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/map", bytes.NewReader(oversized))
rec := httptest.NewRecorder()
req.Body = http.MaxBytesReader(rec, req.Body, noiseBodyLimit)
ns.PollNetMapHandler(rec, req)
// Body is truncated → json.Decode fails → httpError returns 500.
assert.Equal(t, http.StatusInternalServerError, rec.Code)
}
// TestRegistrationHandler_OversizedBody calls the real handler with a
// MaxBytesReader-wrapped body to verify it returns an error response
// rather than consuming unbounded memory.
func TestRegistrationHandler_OversizedBody(t *testing.T) {
t.Parallel()
ns := &noiseServer{}
oversized := bytes.Repeat([]byte("x"), int(noiseBodyLimit)+1)
req := httptest.NewRequestWithContext(context.Background(), http.MethodPost, "/machine/register", bytes.NewReader(oversized))
rec := httptest.NewRecorder()
req.Body = http.MaxBytesReader(rec, req.Body, noiseBodyLimit)
ns.RegistrationHandler(rec, req)
// json.Decode returns MaxBytesError → regErr wraps it → handler writes
// a RegisterResponse with the error and then rejectUnsupported kicks in
// for version 0 → returns 400.
assert.Equal(t, http.StatusBadRequest, rec.Code)
}
+430
View File
@@ -1 +1,431 @@
package matcher
import (
"net/netip"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"tailscale.com/tailcfg"
)
func TestMatchFromStrings(t *testing.T) {
t.Parallel()
tests := []struct {
name string
srcs []string
dsts []string
wantSrc netip.Addr
wantDst netip.Addr
srcIn bool
dstIn bool
}{
{
name: "basic CIDR match",
srcs: []string{"10.0.0.0/8"},
dsts: []string{"192.168.1.0/24"},
wantSrc: netip.MustParseAddr("10.1.2.3"),
wantDst: netip.MustParseAddr("192.168.1.100"),
srcIn: true,
dstIn: true,
},
{
name: "basic CIDR no match",
srcs: []string{"10.0.0.0/8"},
dsts: []string{"192.168.1.0/24"},
wantSrc: netip.MustParseAddr("172.16.0.1"),
wantDst: netip.MustParseAddr("10.0.0.1"),
srcIn: false,
dstIn: false,
},
{
name: "wildcard matches everything",
srcs: []string{"*"},
dsts: []string{"*"},
wantSrc: netip.MustParseAddr("8.8.8.8"),
wantDst: netip.MustParseAddr("1.1.1.1"),
srcIn: true,
dstIn: true,
},
{
name: "wildcard matches IPv6",
srcs: []string{"*"},
dsts: []string{"*"},
wantSrc: netip.MustParseAddr("2001:db8::1"),
wantDst: netip.MustParseAddr("fd7a:115c:a1e0::1"),
srcIn: true,
dstIn: true,
},
{
name: "single IP source",
srcs: []string{"100.64.0.1"},
dsts: []string{"10.0.0.0/8"},
wantSrc: netip.MustParseAddr("100.64.0.1"),
wantDst: netip.MustParseAddr("10.33.0.1"),
srcIn: true,
dstIn: true,
},
{
name: "single IP source no match",
srcs: []string{"100.64.0.1"},
dsts: []string{"10.0.0.0/8"},
wantSrc: netip.MustParseAddr("100.64.0.2"),
wantDst: netip.MustParseAddr("10.33.0.1"),
srcIn: false,
dstIn: true,
},
{
name: "multiple CIDRs",
srcs: []string{"10.0.0.0/8", "172.16.0.0/12"},
dsts: []string{"192.168.0.0/16", "100.64.0.0/10"},
wantSrc: netip.MustParseAddr("172.20.0.1"),
wantDst: netip.MustParseAddr("100.100.0.1"),
srcIn: true,
dstIn: true,
},
{
name: "IPv6 CIDR",
srcs: []string{"fd7a:115c:a1e0::/48"},
dsts: []string{"2001:db8::/32"},
wantSrc: netip.MustParseAddr("fd7a:115c:a1e0::1"),
wantDst: netip.MustParseAddr("2001:db8::1"),
srcIn: true,
dstIn: true,
},
{
name: "empty sources and destinations",
srcs: []string{},
dsts: []string{},
wantSrc: netip.MustParseAddr("10.0.0.1"),
wantDst: netip.MustParseAddr("10.0.0.1"),
srcIn: false,
dstIn: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
m := MatchFromStrings(tt.srcs, tt.dsts)
assert.Equal(t, tt.srcIn, m.SrcsContainsIPs(tt.wantSrc),
"SrcsContainsIPs(%s)", tt.wantSrc)
assert.Equal(t, tt.dstIn, m.DestsContainsIP(tt.wantDst),
"DestsContainsIP(%s)", tt.wantDst)
})
}
}
func TestMatchFromFilterRule(t *testing.T) {
t.Parallel()
tests := []struct {
name string
rule tailcfg.FilterRule
checkSrc netip.Addr
checkDst netip.Addr
srcMatch bool
dstMatch bool
}{
{
name: "standard rule with port range",
rule: tailcfg.FilterRule{
SrcIPs: []string{"100.64.0.1", "fd7a:115c:a1e0::1"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.33.0.0/16", Ports: tailcfg.PortRange{First: 0, Last: 65535}},
},
},
checkSrc: netip.MustParseAddr("100.64.0.1"),
checkDst: netip.MustParseAddr("10.33.0.50"),
srcMatch: true,
dstMatch: true,
},
{
name: "wildcard destination",
rule: tailcfg.FilterRule{
SrcIPs: []string{"10.0.0.0/8"},
DstPorts: []tailcfg.NetPortRange{
{IP: "*"},
},
},
checkSrc: netip.MustParseAddr("10.1.1.1"),
checkDst: netip.MustParseAddr("8.8.8.8"),
srcMatch: true,
dstMatch: true,
},
{
name: "multiple DstPorts entries",
rule: tailcfg.FilterRule{
SrcIPs: []string{"100.64.0.1"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.33.0.0/16"},
{IP: "192.168.1.0/24"},
},
},
checkSrc: netip.MustParseAddr("100.64.0.1"),
checkDst: netip.MustParseAddr("192.168.1.50"),
srcMatch: true,
dstMatch: true,
},
{
name: "empty DstPorts",
rule: tailcfg.FilterRule{
SrcIPs: []string{"100.64.0.1"},
DstPorts: nil,
},
checkSrc: netip.MustParseAddr("100.64.0.1"),
checkDst: netip.MustParseAddr("10.0.0.1"),
srcMatch: true,
dstMatch: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
m := MatchFromFilterRule(tt.rule)
assert.Equal(t, tt.srcMatch, m.SrcsContainsIPs(tt.checkSrc),
"SrcsContainsIPs(%s)", tt.checkSrc)
assert.Equal(t, tt.dstMatch, m.DestsContainsIP(tt.checkDst),
"DestsContainsIP(%s)", tt.checkDst)
})
}
}
func TestMatchesFromFilterRules(t *testing.T) {
t.Parallel()
rules := []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
DstPorts: []tailcfg.NetPortRange{{IP: "192.168.1.0/24"}},
},
{
SrcIPs: []string{"172.16.0.0/12"},
DstPorts: []tailcfg.NetPortRange{{IP: "10.33.0.0/16"}},
},
}
matches := MatchesFromFilterRules(rules)
require.Len(t, matches, 2)
// First matcher: 10.0.0.0/8 -> 192.168.1.0/24
assert.True(t, matches[0].SrcsContainsIPs(netip.MustParseAddr("10.1.2.3")))
assert.False(t, matches[0].SrcsContainsIPs(netip.MustParseAddr("172.16.0.1")))
assert.True(t, matches[0].DestsContainsIP(netip.MustParseAddr("192.168.1.100")))
// Second matcher: 172.16.0.0/12 -> 10.33.0.0/16
assert.True(t, matches[1].SrcsContainsIPs(netip.MustParseAddr("172.20.0.1")))
assert.True(t, matches[1].DestsContainsIP(netip.MustParseAddr("10.33.0.1")))
assert.False(t, matches[1].DestsContainsIP(netip.MustParseAddr("192.168.1.1")))
}
func TestSrcsOverlapsPrefixes(t *testing.T) {
t.Parallel()
tests := []struct {
name string
srcs []string
prefixes []netip.Prefix
want bool
}{
{
name: "exact match",
srcs: []string{"10.33.0.0/16"},
prefixes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
want: true,
},
{
name: "parent contains child",
srcs: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
want: true,
},
{
name: "child overlaps parent",
srcs: []string{"10.33.0.0/16"},
prefixes: []netip.Prefix{netip.MustParsePrefix("10.0.0.0/8")},
want: true,
},
{
name: "no overlap",
srcs: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{netip.MustParsePrefix("192.168.1.0/24")},
want: false,
},
{
name: "multiple prefixes one overlaps",
srcs: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{
netip.MustParsePrefix("192.168.1.0/24"),
netip.MustParsePrefix("10.33.0.0/16"),
},
want: true,
},
{
name: "IPv6 overlap",
srcs: []string{"fd7a:115c:a1e0::/48"},
prefixes: []netip.Prefix{netip.MustParsePrefix("fd7a:115c:a1e0:ab12::/64")},
want: true,
},
{
name: "empty prefixes",
srcs: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{},
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
m := MatchFromStrings(tt.srcs, nil)
got := m.SrcsOverlapsPrefixes(tt.prefixes...)
assert.Equal(t, tt.want, got)
})
}
}
func TestDestsOverlapsPrefixes(t *testing.T) {
t.Parallel()
tests := []struct {
name string
dsts []string
prefixes []netip.Prefix
want bool
}{
{
name: "exact match",
dsts: []string{"10.33.0.0/16"},
prefixes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
want: true,
},
{
name: "parent contains child",
dsts: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
want: true,
},
{
name: "no overlap",
dsts: []string{"10.0.0.0/8"},
prefixes: []netip.Prefix{netip.MustParsePrefix("192.168.0.0/16")},
want: false,
},
{
name: "wildcard overlaps everything",
dsts: []string{"*"},
prefixes: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
},
want: true,
},
{
name: "wildcard overlaps exit route",
dsts: []string{"*"},
prefixes: []netip.Prefix{netip.MustParsePrefix("::/0")},
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
m := MatchFromStrings(nil, tt.dsts)
got := m.DestsOverlapsPrefixes(tt.prefixes...)
assert.Equal(t, tt.want, got)
})
}
}
func TestDestsIsTheInternet(t *testing.T) {
t.Parallel()
tests := []struct {
name string
dsts []string
want bool
}{
{
name: "all IPv4 is the internet",
dsts: []string{"0.0.0.0/0"},
want: true,
},
{
name: "all IPv6 is the internet",
dsts: []string{"::/0"},
want: true,
},
{
name: "wildcard is the internet",
dsts: []string{"*"},
want: true,
},
{
name: "private range is not the internet",
dsts: []string{"10.0.0.0/8"},
want: false,
},
{
name: "CGNAT range is not the internet",
dsts: []string{"100.64.0.0/10"},
want: false,
},
{
name: "single public IP is not the internet",
dsts: []string{"8.8.8.8"},
want: false,
},
{
name: "empty dests is not the internet",
dsts: []string{},
want: false,
},
{
name: "multiple private ranges are not the internet",
dsts: []string{
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16",
},
want: false,
},
{
name: "all IPv4 combined with subnet is the internet",
dsts: []string{"0.0.0.0/0", "10.33.0.0/16"},
want: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
t.Parallel()
m := MatchFromStrings(nil, tt.dsts)
got := m.DestsIsTheInternet()
assert.Equal(t, tt.want, got,
"DestsIsTheInternet() for dsts=%v", tt.dsts)
})
}
}
func TestDebugString(t *testing.T) {
t.Parallel()
m := MatchFromStrings(
[]string{"10.0.0.0/8"},
[]string{"192.168.1.0/24"},
)
s := m.DebugString()
assert.Contains(t, s, "Match:")
assert.Contains(t, s, "Sources:")
assert.Contains(t, s, "Destinations:")
assert.Contains(t, s, "10.0.0.0/8")
assert.Contains(t, s, "192.168.1.0/24")
}
+6
View File
@@ -36,6 +36,12 @@ type PolicyManager interface {
// NodeCanApproveRoute reports whether the given node can approve the given route.
NodeCanApproveRoute(node types.NodeView, route netip.Prefix) bool
// ViaRoutesForPeer computes via grant effects for a viewer-peer pair.
// It returns which routes should be included (peer is via-designated for viewer)
// and excluded (steered to a different peer). When no via grants apply,
// both fields are empty and the caller falls back to existing behavior.
ViaRoutesForPeer(viewer, peer types.NodeView) types.ViaRouteResult
Version() int
DebugString() string
}
+2 -2
View File
@@ -19,11 +19,11 @@ import (
func TestApproveRoutesWithPolicy_NeverRemovesApprovedRoutes(t *testing.T) {
user1 := types.User{
Model: gorm.Model{ID: 1},
Name: "testuser@",
Name: "testuser",
}
user2 := types.User{
Model: gorm.Model{ID: 2},
Name: "otheruser@",
Name: "otheruser",
}
users := []types.User{user1, user2}
+180
View File
@@ -768,6 +768,65 @@ func TestReduceNodes(t *testing.T) {
},
},
},
// Subnet-to-subnet: routers must see each other when ACL
// uses only subnet CIDRs. Issue #3157.
{
name: "subnet-to-subnet-routers-see-each-other-3157",
args: args{
nodes: []*types.Node{
{
ID: 1,
IPv4: ap("100.64.0.1"),
Hostname: "router-a",
User: &types.User{Name: "router-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.88.8.0/24")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.88.8.0/24")},
},
{
ID: 2,
IPv4: ap("100.64.0.2"),
Hostname: "router-b",
User: &types.User{Name: "router-b"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.99.9.0/24")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.99.9.0/24")},
},
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.88.8.0/24"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.99.9.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
node: &types.Node{
ID: 1,
IPv4: ap("100.64.0.1"),
Hostname: "router-a",
User: &types.User{Name: "router-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.88.8.0/24")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.88.8.0/24")},
},
},
want: []*types.Node{
{
ID: 2,
IPv4: ap("100.64.0.2"),
Hostname: "router-b",
User: &types.User{Name: "router-b"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.99.9.0/24")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.99.9.0/24")},
},
},
},
}
for _, tt := range tests {
@@ -2230,6 +2289,127 @@ func TestReduceRoutes(t *testing.T) {
netip.MustParsePrefix("192.168.1.0/14"),
},
},
// Subnet-to-subnet tests for issue #3157.
// When an ACL references subnet CIDRs as both source and destination,
// the subnet routers for those subnets must receive routes to each
// other's subnets.
{
name: "subnet-to-subnet-src-router-gets-dst-route-3157",
args: args{
node: &types.Node{
ID: 1,
IPv4: ap("100.64.0.1"),
User: &types.User{Name: "router-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
},
},
routes: []netip.Prefix{
netip.MustParsePrefix("10.99.9.0/24"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.88.8.0/24"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.99.9.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
},
want: []netip.Prefix{
netip.MustParsePrefix("10.99.9.0/24"),
},
},
{
name: "subnet-to-subnet-dst-router-gets-src-route-3157",
args: args{
node: &types.Node{
ID: 2,
IPv4: ap("100.64.0.2"),
User: &types.User{Name: "router-b"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.99.9.0/24"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.99.9.0/24"),
},
},
routes: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.88.8.0/24"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.99.9.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
},
want: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
},
},
{
name: "subnet-to-subnet-regular-node-no-route-leak-3157",
args: args{
node: &types.Node{
ID: 3,
IPv4: ap("100.64.0.3"),
User: &types.User{Name: "regular-node"},
},
routes: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
netip.MustParsePrefix("10.99.9.0/24"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.88.8.0/24"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.99.9.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
},
want: nil,
},
{
name: "subnet-to-subnet-unrelated-router-no-route-leak-3157",
args: args{
node: &types.Node{
ID: 4,
IPv4: ap("100.64.0.4"),
User: &types.User{Name: "router-c"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("172.16.0.0/24"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("172.16.0.0/24"),
},
},
routes: []netip.Prefix{
netip.MustParsePrefix("10.88.8.0/24"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.88.8.0/24"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.99.9.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
},
want: nil,
},
}
for _, tt := range tests {
+108 -13
View File
@@ -1,8 +1,12 @@
package policyutil
import (
"net/netip"
"slices"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/util"
"tailscale.com/net/tsaddr"
"tailscale.com/tailcfg"
)
@@ -16,6 +20,17 @@ func ReduceFilterRules(node types.NodeView, rules []tailcfg.FilterRule) []tailcf
ret := []tailcfg.FilterRule{}
for _, rule := range rules {
// Handle CapGrant rules separately — they use CapGrant[].Dsts
// instead of DstPorts for destination matching.
if len(rule.CapGrant) > 0 {
reduced := reduceCapGrantRule(node, rule)
if reduced != nil {
ret = append(ret, *reduced)
}
continue
}
// record if the rule is actually relevant for the given node.
var dests []tailcfg.NetPortRange
@@ -33,12 +48,28 @@ func ReduceFilterRules(node types.NodeView, rules []tailcfg.FilterRule) []tailcf
continue DEST_LOOP
}
// If the node exposes routes, ensure they are note removed
// when the filters are reduced.
// If the node advertises routes, ensure filter rules
// targeting those routes are preserved. Exit routes
// (0.0.0.0/0, ::/0) are skipped because exit nodes
// handle traffic via AllowedIPs/routing, not packet
// filter rules. This matches Tailscale SaaS behavior.
//
// NOTE: This uses RoutableIPs (advertised routes)
// rather than SubnetRoutes (approved routes). The two
// sets are identical in all 98 golden file captures
// from Tailscale SaaS, so we cannot determine from
// captured data which set Tailscale actually checks.
// RoutableIPs is a superset of SubnetRoutes (which
// further filters by approval), so this is the more
// permissive choice. See MISSING_SAAS_DATA.md for
// the capture needed to resolve this ambiguity.
if node.Hostinfo().Valid() {
routableIPs := node.Hostinfo().RoutableIPs()
if routableIPs.Len() > 0 {
for _, routableIP := range routableIPs.All() {
if tsaddr.IsExitRoute(routableIP) {
continue
}
if expanded.OverlapsPrefix(routableIP) {
dests = append(dests, dest)
continue DEST_LOOP
@@ -46,17 +77,6 @@ func ReduceFilterRules(node types.NodeView, rules []tailcfg.FilterRule) []tailcf
}
}
}
// Also check approved subnet routes - nodes should have access
// to subnets they're approved to route traffic for.
subnetRoutes := node.SubnetRoutes()
for _, subnetRoute := range subnetRoutes {
if expanded.OverlapsPrefix(subnetRoute) {
dests = append(dests, dest)
continue DEST_LOOP
}
}
}
if len(dests) > 0 {
@@ -70,3 +90,78 @@ func ReduceFilterRules(node types.NodeView, rules []tailcfg.FilterRule) []tailcf
return ret
}
// reduceCapGrantRule filters a CapGrant rule to only include CapGrant
// entries whose Dsts match the given node's IPs. When a broad prefix
// (e.g. 100.64.0.0/10 from dst:*) contains a node's IP, it is
// narrowed to the node's specific /32 or /128 prefix. Returns nil if
// no CapGrant entries are relevant to this node.
func reduceCapGrantRule(
node types.NodeView,
rule tailcfg.FilterRule,
) *tailcfg.FilterRule {
var capGrants []tailcfg.CapGrant
nodeIPs := node.IPs()
for _, cg := range rule.CapGrant {
// Collect the node's IPs that fall within any of this
// CapGrant's Dsts. Broad prefixes are narrowed to specific
// /32 and /128 entries for the node.
var matchingDsts []netip.Prefix
for _, dst := range cg.Dsts {
if dst.IsSingleIP() {
// Already a specific IP — keep it if it matches
// any of the node's IPs.
if slices.Contains(nodeIPs, dst.Addr()) {
matchingDsts = append(matchingDsts, dst)
}
} else {
// Broad prefix — narrow to node's specific IPs.
for _, ip := range nodeIPs {
if dst.Contains(ip) {
matchingDsts = append(matchingDsts, netip.PrefixFrom(ip, ip.BitLen()))
}
}
}
}
// Also check routable IPs (subnet routes) — nodes that
// advertise routes should receive CapGrant rules for
// destinations that overlap their routes.
if node.Hostinfo().Valid() {
routableIPs := node.Hostinfo().RoutableIPs()
if routableIPs.Len() > 0 {
for _, dst := range cg.Dsts {
for _, routableIP := range routableIPs.All() {
if tsaddr.IsExitRoute(routableIP) {
continue
}
if dst.Overlaps(routableIP) {
// For route overlaps, keep the original prefix.
matchingDsts = append(matchingDsts, dst)
}
}
}
}
}
if len(matchingDsts) > 0 {
capGrants = append(capGrants, tailcfg.CapGrant{
Dsts: matchingDsts,
CapMap: cg.CapMap,
})
}
}
if len(capGrants) == 0 {
return nil
}
return &tailcfg.FilterRule{
SrcIPs: rule.SrcIPs,
CapGrant: capGrants,
}
}
+486 -75
View File
@@ -9,7 +9,6 @@ import (
"github.com/google/go-cmp/cmp"
"github.com/juanfont/headscale/hscontrol/policy"
"github.com/juanfont/headscale/hscontrol/policy/policyutil"
v2 "github.com/juanfont/headscale/hscontrol/policy/v2"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/util"
"github.com/rs/zerolog/log"
@@ -206,21 +205,19 @@ func TestReduceFilterRules(t *testing.T) {
},
},
want: []tailcfg.FilterRule{
// Merged: Both ACL rules combined (same SrcIPs and IPProto)
// Merged: Both ACL rules combined (same SrcIPs)
{
SrcIPs: []string{
"100.64.0.1/32",
"100.64.0.2/32",
"fd7a:115c:a1e0::1/128",
"fd7a:115c:a1e0::2/128",
"100.64.0.1-100.64.0.2",
"fd7a:115c:a1e0::1-fd7a:115c:a1e0::2",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.1/32",
IP: "100.64.0.1",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::1/128",
IP: "fd7a:115c:a1e0::1",
Ports: tailcfg.PortRangeAny,
},
{
@@ -228,7 +225,6 @@ func TestReduceFilterRules(t *testing.T) {
Ports: tailcfg.PortRangeAny,
},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -356,18 +352,16 @@ func TestReduceFilterRules(t *testing.T) {
// autogroup:internet does NOT generate packet filters - it's handled
// by exit node routing via AllowedIPs, not by packet filtering.
{
SrcIPs: []string{"100.64.0.1/32", "100.64.0.2/32", "fd7a:115c:a1e0::1/128", "fd7a:115c:a1e0::2/128"},
SrcIPs: []string{
"100.64.0.1-100.64.0.2",
"fd7a:115c:a1e0::1-fd7a:115c:a1e0::2",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.100/32",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::100/128",
IP: "100.64.0.100",
Ports: tailcfg.PortRangeAny,
},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -459,50 +453,22 @@ func TestReduceFilterRules(t *testing.T) {
},
},
want: []tailcfg.FilterRule{
// Merged: Both ACL rules combined (same SrcIPs and IPProto)
// Exit routes (0.0.0.0/0, ::/0) are skipped when checking RoutableIPs
// overlap, matching Tailscale SaaS behavior. Only destinations that
// contain the node's own Tailscale IP (via InIPSet) are kept.
// Here, 64.0.0.0/2 contains 100.64.0.100 (CGNAT range), so it matches.
{
SrcIPs: []string{"100.64.0.1/32", "100.64.0.2/32", "fd7a:115c:a1e0::1/128", "fd7a:115c:a1e0::2/128"},
SrcIPs: []string{
"100.64.0.1-100.64.0.2",
"fd7a:115c:a1e0::1-fd7a:115c:a1e0::2",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.100/32",
IP: "100.64.0.100",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::100/128",
Ports: tailcfg.PortRangeAny,
},
{IP: "0.0.0.0/5", Ports: tailcfg.PortRangeAny},
{IP: "8.0.0.0/7", Ports: tailcfg.PortRangeAny},
{IP: "11.0.0.0/8", Ports: tailcfg.PortRangeAny},
{IP: "12.0.0.0/6", Ports: tailcfg.PortRangeAny},
{IP: "16.0.0.0/4", Ports: tailcfg.PortRangeAny},
{IP: "32.0.0.0/3", Ports: tailcfg.PortRangeAny},
{IP: "64.0.0.0/2", Ports: tailcfg.PortRangeAny},
{IP: "128.0.0.0/3", Ports: tailcfg.PortRangeAny},
{IP: "160.0.0.0/5", Ports: tailcfg.PortRangeAny},
{IP: "168.0.0.0/6", Ports: tailcfg.PortRangeAny},
{IP: "172.0.0.0/12", Ports: tailcfg.PortRangeAny},
{IP: "172.32.0.0/11", Ports: tailcfg.PortRangeAny},
{IP: "172.64.0.0/10", Ports: tailcfg.PortRangeAny},
{IP: "172.128.0.0/9", Ports: tailcfg.PortRangeAny},
{IP: "173.0.0.0/8", Ports: tailcfg.PortRangeAny},
{IP: "174.0.0.0/7", Ports: tailcfg.PortRangeAny},
{IP: "176.0.0.0/4", Ports: tailcfg.PortRangeAny},
{IP: "192.0.0.0/9", Ports: tailcfg.PortRangeAny},
{IP: "192.128.0.0/11", Ports: tailcfg.PortRangeAny},
{IP: "192.160.0.0/13", Ports: tailcfg.PortRangeAny},
{IP: "192.169.0.0/16", Ports: tailcfg.PortRangeAny},
{IP: "192.170.0.0/15", Ports: tailcfg.PortRangeAny},
{IP: "192.172.0.0/14", Ports: tailcfg.PortRangeAny},
{IP: "192.176.0.0/12", Ports: tailcfg.PortRangeAny},
{IP: "192.192.0.0/10", Ports: tailcfg.PortRangeAny},
{IP: "193.0.0.0/8", Ports: tailcfg.PortRangeAny},
{IP: "194.0.0.0/7", Ports: tailcfg.PortRangeAny},
{IP: "196.0.0.0/6", Ports: tailcfg.PortRangeAny},
{IP: "200.0.0.0/5", Ports: tailcfg.PortRangeAny},
{IP: "208.0.0.0/4", Ports: tailcfg.PortRangeAny},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -566,16 +532,15 @@ func TestReduceFilterRules(t *testing.T) {
},
},
want: []tailcfg.FilterRule{
// Merged: Both ACL rules combined (same SrcIPs and IPProto)
// Merged: Both ACL rules combined (same SrcIPs)
{
SrcIPs: []string{"100.64.0.1/32", "100.64.0.2/32", "fd7a:115c:a1e0::1/128", "fd7a:115c:a1e0::2/128"},
SrcIPs: []string{
"100.64.0.1-100.64.0.2",
"fd7a:115c:a1e0::1-fd7a:115c:a1e0::2",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.100/32",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::100/128",
IP: "100.64.0.100",
Ports: tailcfg.PortRangeAny,
},
{
@@ -587,7 +552,6 @@ func TestReduceFilterRules(t *testing.T) {
Ports: tailcfg.PortRangeAny,
},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -651,16 +615,15 @@ func TestReduceFilterRules(t *testing.T) {
},
},
want: []tailcfg.FilterRule{
// Merged: Both ACL rules combined (same SrcIPs and IPProto)
// Merged: Both ACL rules combined (same SrcIPs)
{
SrcIPs: []string{"100.64.0.1/32", "100.64.0.2/32", "fd7a:115c:a1e0::1/128", "fd7a:115c:a1e0::2/128"},
SrcIPs: []string{
"100.64.0.1-100.64.0.2",
"fd7a:115c:a1e0::1-fd7a:115c:a1e0::2",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.100/32",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::100/128",
IP: "100.64.0.100",
Ports: tailcfg.PortRangeAny,
},
{
@@ -672,7 +635,6 @@ func TestReduceFilterRules(t *testing.T) {
Ports: tailcfg.PortRangeAny,
},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -725,22 +687,24 @@ func TestReduceFilterRules(t *testing.T) {
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"100.64.0.1/32", "fd7a:115c:a1e0::1/128"},
SrcIPs: []string{
"100.64.0.1",
"fd7a:115c:a1e0::1",
},
DstPorts: []tailcfg.NetPortRange{
{
IP: "100.64.0.100/32",
IP: "100.64.0.100",
Ports: tailcfg.PortRangeAny,
},
{
IP: "fd7a:115c:a1e0::100/128",
IP: "fd7a:115c:a1e0::100",
Ports: tailcfg.PortRangeAny,
},
{
IP: "172.16.0.21/32",
IP: "172.16.0.21",
Ports: tailcfg.PortRangeAny,
},
},
IPProto: []int{v2.ProtocolTCP, v2.ProtocolUDP, v2.ProtocolICMP, v2.ProtocolIPv6ICMP},
},
},
},
@@ -795,7 +759,9 @@ func TestReduceFilterRules(t *testing.T) {
}
for _, tt := range tests {
for idx, pmf := range policy.PolicyManagerFuncsForTest([]byte(tt.pol)) {
for idx, pmf := range policy.PolicyManagerFuncsForTest(
[]byte(tt.pol),
) {
t.Run(fmt.Sprintf("%s-index%d", tt.name, idx), func(t *testing.T) {
var (
pm policy.PolicyManager
@@ -817,3 +783,448 @@ func TestReduceFilterRules(t *testing.T) {
}
}
}
// TestReduceFilterRulesPartialApproval documents the behavior of
// ReduceFilterRules when RoutableIPs (advertised) and SubnetRoutes
// (approved) differ. In production, SubnetRoutes is always a subset
// of RoutableIPs, but the two code paths in ReduceFilterRules handle
// them independently:
// - Lines 56-68: Check RoutableIPs (advertised, may be unapproved)
// - Lines 73-79: Check SubnetRoutes (approved only)
//
// This test documents that ReduceFilterRules includes filter rules
// for advertised-but-unapproved routes via the RoutableIPs check.
func TestReduceFilterRulesPartialApproval(t *testing.T) {
tests := []struct {
name string
node *types.Node
rules []tailcfg.FilterRule
wantCount int
wantRoutes []string
}{
{
name: "approved-route-included",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
DstPorts: []tailcfg.NetPortRange{
{IP: "10.33.0.0/16", Ports: tailcfg.PortRangeAny},
},
},
},
wantCount: 1,
wantRoutes: []string{"10.33.0.0/16"},
},
{
name: "unapproved-route-still-included-via-routableips",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
// Advertised but NOT approved:
netip.MustParsePrefix("172.16.0.0/24"),
},
},
ApprovedRoutes: []netip.Prefix{
// Only 10.33.0.0/16 approved
netip.MustParsePrefix("10.33.0.0/16"),
},
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
DstPorts: []tailcfg.NetPortRange{
// Targets the unapproved route
{IP: "172.16.0.0/24", Ports: tailcfg.PortRangeAny},
},
},
},
// The RoutableIPs check (line 63) matches because
// 172.16.0.0/24 IS in RoutableIPs. The rule is
// kept even though the route is not approved.
wantCount: 1,
wantRoutes: []string{"172.16.0.0/24"},
},
{
name: "neither-advertised-nor-approved-excluded",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
DstPorts: []tailcfg.NetPortRange{
// Not advertised, not approved
{IP: "192.168.0.0/16", Ports: tailcfg.PortRangeAny},
},
},
},
wantCount: 0,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := policyutil.ReduceFilterRules(
tt.node.View(), tt.rules,
)
require.Len(t, got, tt.wantCount,
"rule count mismatch")
if tt.wantCount > 0 {
var gotRoutes []string
for _, dp := range got[0].DstPorts {
gotRoutes = append(gotRoutes, dp.IP)
}
require.Equal(t, tt.wantRoutes, gotRoutes)
}
})
}
}
// TestReduceFilterRulesCapGrant tests the CapGrant branch of
// ReduceFilterRules, which was previously untested. All existing
// test cases use ACL-only policies with DstPorts rules.
func TestReduceFilterRulesCapGrant(t *testing.T) {
tests := []struct {
name string
node *types.Node
rules []tailcfg.FilterRule
want []tailcfg.FilterRule
}{
{
name: "capgrant-matches-node-ip-narrowed",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// Broad IPv4 prefix containing node IP
netip.MustParsePrefix("100.64.0.0/10"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// Only IPv4 narrowed (IPv6 not in /10)
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
},
{
name: "capgrant-no-match-filtered-out",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// Different IP — doesn't match this node
netip.MustParsePrefix("100.64.0.99/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{},
},
{
name: "capgrant-with-subnet-route-overlap",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
},
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// Subnet route overlap
netip.MustParsePrefix("10.0.0.0/8"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/relay-target": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// 10.0.0.0/8 doesn't contain node's
// CGNAT IP (100.64.0.1), so no IP
// narrowing. Only route overlap kept
// as original prefix.
netip.MustParsePrefix("10.0.0.0/8"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/relay-target": nil,
},
},
},
},
},
},
{
name: "capgrant-exit-route-skipped",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: tsaddr.ExitRoutes(),
},
ApprovedRoutes: tsaddr.ExitRoutes(),
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// 0.0.0.0/0 overlaps the exit route
// but exit routes should be skipped
netip.MustParsePrefix("0.0.0.0/0"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"*"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
// Node IP narrowed (0.0.0.0/0
// contains the node's IP)
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
},
{
name: "mixed-dstports-and-capgrant-rules",
node: &types.Node{
IPv4: ap("100.64.0.1"),
IPv6: ap("fd7a:115c:a1e0::1"),
},
rules: []tailcfg.FilterRule{
{
// DstPorts rule that matches
SrcIPs: []string{"10.0.0.0/8"},
DstPorts: []tailcfg.NetPortRange{
{IP: "100.64.0.1", Ports: tailcfg.PortRangeAny},
},
},
{
// CapGrant rule that matches
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
{
// CapGrant rule that doesn't match
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.99/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
DstPorts: []tailcfg.NetPortRange{
{IP: "100.64.0.1", Ports: tailcfg.PortRangeAny},
},
},
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
// Third rule filtered out — doesn't match node
},
},
{
name: "capgrant-ipv4-only-node",
node: &types.Node{
IPv4: ap("100.64.0.1"),
// IPv6 is nil, single-IP node
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
},
{
name: "capgrant-zero-ip-node-no-panic",
node: &types.Node{
// Both IPv4 and IPv6 are nil, zero IPs.
// Must not panic.
},
rules: []tailcfg.FilterRule{
{
SrcIPs: []string{"10.0.0.0/8"},
CapGrant: []tailcfg.CapGrant{
{
Dsts: []netip.Prefix{
netip.MustParsePrefix("100.64.0.1/32"),
},
CapMap: tailcfg.PeerCapMap{
"tailscale.com/cap/drive-sharer": nil,
},
},
},
},
},
want: []tailcfg.FilterRule{},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := policyutil.ReduceFilterRules(tt.node.View(), tt.rules)
require.Len(t, got, len(tt.want),
"rule count mismatch")
for i := range tt.want {
require.Equal(t, tt.want[i].SrcIPs, got[i].SrcIPs,
"rule[%d] SrcIPs", i)
require.Len(t, got[i].DstPorts, len(tt.want[i].DstPorts),
"rule[%d] DstPorts count", i)
require.Len(t, got[i].CapGrant, len(tt.want[i].CapGrant),
"rule[%d] CapGrant count", i)
for j := range tt.want[i].CapGrant {
require.ElementsMatch(t,
tt.want[i].CapGrant[j].Dsts,
got[i].CapGrant[j].Dsts,
"rule[%d].CapGrant[%d] Dsts", i, j,
)
}
}
})
}
}
+725
View File
@@ -0,0 +1,725 @@
package v2
import (
"net/netip"
"slices"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/rs/zerolog/log"
"go4.org/netipx"
"tailscale.com/tailcfg"
"tailscale.com/types/views"
)
// grantCategory classifies a grant by what per-node work it needs.
type grantCategory int
const (
// grantCategoryRegular requires no per-node work. The pre-compiled
// rules are complete and only need ReduceFilterRules.
grantCategoryRegular grantCategory = iota
// grantCategorySelf has autogroup:self destinations that must be
// expanded per-node to same-user untagged device IPs.
grantCategorySelf
// grantCategoryVia has Via tags that route rules to specific
// nodes based on their tags and advertised routes.
grantCategoryVia
)
// compiledGrant is a grant with its sources already resolved to IP
// addresses. The expensive work (alias → IP resolution) is done once
// here. Extracting rules for a specific node reads from pre-resolved
// data without re-resolving.
type compiledGrant struct {
category grantCategory
// srcIPStrings is the final SrcIPs for non-self rules, with
// nonWildcardSrcs appended to match Tailscale SaaS behavior.
srcIPStrings []string
hasWildcard bool
hasDangerAll bool
// rules are the pre-compiled filter rules for non-self, non-via
// destinations. For regular grants this is the complete output.
// For self grants with mixed destinations (self + other), this
// is the non-self portion only.
rules []tailcfg.FilterRule
// self is non-nil when the grant has autogroup:self destinations.
self *selfGrantData
// via is non-nil when the grant has Via tags.
via *viaGrantData
}
// selfGrantData holds data needed for per-node autogroup:self
// compilation. Sources are already resolved.
type selfGrantData struct {
resolvedSrcs []ResolvedAddresses
internetProtocols []ProtocolPort
app tailcfg.PeerCapMap
}
// viaGrantData holds data needed for per-node via-grant compilation.
// Sources are already resolved into srcIPStrings.
type viaGrantData struct {
viaTags []Tag
destinations Aliases
internetProtocols []ProtocolPort
srcIPStrings []string
}
// userNodeIndex maps user IDs to their untagged nodes. Precomputed
// once per node-set change, used by compileAutogroupSelf to avoid
// O(N) scans per self-grant per node.
type userNodeIndex map[uint][]types.NodeView
func buildUserNodeIndex(
nodes views.Slice[types.NodeView],
) userNodeIndex {
idx := make(userNodeIndex)
for _, n := range nodes.All() {
if !n.IsTagged() && n.User().Valid() {
uid := n.User().ID()
idx[uid] = append(idx[uid], n)
}
}
return idx
}
// compileGrants resolves all policy grants into compiledGrant structs.
// Source resolution and non-self destination resolution happens once
// here. This is the single resolution path that replaces the
// duplicated work in compileFilterRules and compileGrantWithAutogroupSelf.
func (pol *Policy) compileGrants(
users types.Users,
nodes views.Slice[types.NodeView],
) []compiledGrant {
if pol == nil || (pol.ACLs == nil && len(pol.Grants) == 0) {
return nil
}
grants := pol.Grants
for _, acl := range pol.ACLs {
grants = append(grants, aclToGrants(acl)...)
}
compiled := make([]compiledGrant, 0, len(grants))
for _, grant := range grants {
cg, err := pol.compileOneGrant(grant, users, nodes)
if err != nil {
log.Trace().Err(err).Msg("compiling grant")
continue
}
if cg != nil {
compiled = append(compiled, *cg)
}
}
return compiled
}
// compileOneGrant resolves a single grant into a compiledGrant.
// All source resolution happens here. Non-self, non-via destination
// resolution also happens here. Per-node data (self dests, via
// matching) is stored for deferred compilation.
//
//nolint:gocyclo,cyclop
func (pol *Policy) compileOneGrant(
grant Grant,
users types.Users,
nodes views.Slice[types.NodeView],
) (*compiledGrant, error) {
// Via grants: resolve sources, store deferred data.
if len(grant.Via) > 0 {
return pol.compileOneViaGrant(grant, users, nodes)
}
// Split destinations into self vs other.
var autogroupSelfDests, otherDests []Alias
for _, dest := range grant.Destinations {
if ag, ok := dest.(*AutoGroup); ok && ag.Is(AutoGroupSelf) {
autogroupSelfDests = append(autogroupSelfDests, dest)
} else {
otherDests = append(otherDests, dest)
}
}
// Resolve sources per-alias, tracking non-wildcard sources
// separately so we can preserve their IPs alongside the
// wildcard CGNAT ranges (matching Tailscale SaaS behavior).
resolvedSrcs, nonWildcardSrcs, err := resolveSources(
pol, grant.Sources, users, nodes,
)
if err != nil {
return nil, err
}
// Literally empty src=[] or dst=[] produces no rules.
if len(grant.Sources) == 0 || len(grant.Destinations) == 0 {
return nil, nil //nolint:nilnil
}
if len(resolvedSrcs) == 0 && grant.App == nil {
return nil, nil //nolint:nilnil
}
hasWildcard := sourcesHaveWildcard(grant.Sources)
hasDangerAll := sourcesHaveDangerAll(grant.Sources)
srcIPStrings := buildSrcIPStrings(
resolvedSrcs, nonWildcardSrcs,
hasWildcard, hasDangerAll, nodes,
)
cg := &compiledGrant{
srcIPStrings: srcIPStrings,
hasWildcard: hasWildcard,
hasDangerAll: hasDangerAll,
}
// Compile non-self destination rules (done once, shared).
if len(otherDests) > 0 {
cg.rules = pol.compileOtherDests(
users, nodes, grant, otherDests,
resolvedSrcs, srcIPStrings,
)
}
// Classify and store deferred self data.
switch {
case len(autogroupSelfDests) > 0:
cg.category = grantCategorySelf
cg.self = &selfGrantData{
resolvedSrcs: resolvedSrcs,
internetProtocols: grant.InternetProtocols,
app: grant.App,
}
default:
cg.category = grantCategoryRegular
}
return cg, nil
}
// compileOneViaGrant resolves sources for a via grant and stores the
// deferred per-node data. The actual via-node matching and route
// intersection happens in compileViaForNode.
func (pol *Policy) compileOneViaGrant(
grant Grant,
users types.Users,
nodes views.Slice[types.NodeView],
) (*compiledGrant, error) {
if len(grant.InternetProtocols) == 0 {
return nil, nil //nolint:nilnil
}
resolvedSrcs, _, err := resolveSources(
pol, grant.Sources, users, nodes,
)
if err != nil {
return nil, err
}
if len(resolvedSrcs) == 0 {
return nil, nil //nolint:nilnil
}
// Build merged SrcIPs.
var srcIPs netipx.IPSetBuilder
for _, ips := range resolvedSrcs {
for _, pref := range ips.Prefixes() {
srcIPs.AddPrefix(pref)
}
}
srcResolved, err := newResolved(&srcIPs)
if err != nil {
return nil, err
}
if srcResolved.Empty() {
return nil, nil //nolint:nilnil
}
hasWildcard := sourcesHaveWildcard(grant.Sources)
hasDangerAll := sourcesHaveDangerAll(grant.Sources)
return &compiledGrant{
category: grantCategoryVia,
via: &viaGrantData{
viaTags: grant.Via,
destinations: grant.Destinations,
internetProtocols: grant.InternetProtocols,
srcIPStrings: srcIPsWithRoutes(
srcResolved, hasWildcard, hasDangerAll, nodes,
),
},
}, nil
}
// resolveSources resolves grant sources per-alias, returning the
// resolved addresses and a separate slice of non-wildcard sources.
// This is the shared source resolution used by all grant categories.
func resolveSources(
pol *Policy,
sources Aliases,
users types.Users,
nodes views.Slice[types.NodeView],
) ([]ResolvedAddresses, []ResolvedAddresses, error) {
var all, nonWild []ResolvedAddresses
for i, src := range sources {
if ag, ok := src.(*AutoGroup); ok && ag.Is(AutoGroupSelf) {
return nil, nil, errSelfInSources
}
ips, err := src.Resolve(pol, users, nodes)
if err != nil {
log.Trace().Caller().Err(err).
Msg("resolving source ips")
}
if ips != nil {
all = append(all, ips)
if _, isWildcard := sources[i].(Asterix); !isWildcard {
nonWild = append(nonWild, ips)
}
}
}
return all, nonWild, nil
}
// buildSrcIPStrings builds the final SrcIPs string slice from
// resolved sources, preserving non-wildcard IPs alongside wildcard
// CGNAT ranges to match Tailscale SaaS behavior.
func buildSrcIPStrings(
resolvedSrcs, nonWildcardSrcs []ResolvedAddresses,
hasWildcard, hasDangerAll bool,
nodes views.Slice[types.NodeView],
) []string {
var merged netipx.IPSetBuilder
for _, ips := range resolvedSrcs {
for _, pref := range ips.Prefixes() {
merged.AddPrefix(pref)
}
}
srcResolved, err := newResolved(&merged)
if err != nil {
return nil
}
if srcResolved.Empty() {
return nil
}
srcIPStrs := srcIPsWithRoutes(
srcResolved, hasWildcard, hasDangerAll, nodes,
)
// When sources include a wildcard (*) alongside explicit
// sources (tags, groups, etc.), Tailscale preserves the
// individual IPs from non-wildcard sources alongside the
// merged CGNAT ranges rather than absorbing them.
if hasWildcard && len(nonWildcardSrcs) > 0 {
seen := make(map[string]bool, len(srcIPStrs))
for _, s := range srcIPStrs {
seen[s] = true
}
for _, ips := range nonWildcardSrcs {
for _, s := range ips.Strings() {
if !seen[s] {
seen[s] = true
srcIPStrs = append(srcIPStrs, s)
}
}
}
}
return srcIPStrs
}
// compileOtherDests compiles filter rules for non-self, non-via
// destinations. This produces both DstPorts rules (from
// InternetProtocols) and CapGrant rules (from App).
func (pol *Policy) compileOtherDests(
users types.Users,
nodes views.Slice[types.NodeView],
grant Grant,
otherDests Aliases,
resolvedSrcs []ResolvedAddresses,
srcIPStrings []string,
) []tailcfg.FilterRule {
var rules []tailcfg.FilterRule
// DstPorts rules from InternetProtocols.
for _, ipp := range grant.InternetProtocols {
destPorts := pol.destinationsToNetPortRange(
users, nodes, otherDests, ipp.Ports,
)
if len(destPorts) > 0 && len(srcIPStrings) > 0 {
rules = append(rules, tailcfg.FilterRule{
SrcIPs: srcIPStrings,
DstPorts: destPorts,
IPProto: ipp.Protocol.toIANAProtocolNumbers(),
})
}
}
// CapGrant rules from App.
if grant.App != nil {
capSrcIPStrs := srcIPStrings
// When sources resolved to empty but App is set,
// Tailscale still produces the CapGrant rule with
// empty SrcIPs.
if capSrcIPStrs == nil {
capSrcIPStrs = []string{}
}
var (
capGrants []tailcfg.CapGrant
dstIPStrings []string
)
for _, dst := range otherDests {
ips, err := dst.Resolve(pol, users, nodes)
if err != nil {
continue
}
capGrants = append(capGrants, tailcfg.CapGrant{
Dsts: ips.Prefixes(),
CapMap: grant.App,
})
dstIPStrings = append(dstIPStrings, ips.Strings()...)
}
if len(capGrants) > 0 {
srcPrefixes := make([]netip.Prefix, 0, len(resolvedSrcs)*2)
for _, ips := range resolvedSrcs {
srcPrefixes = append(
srcPrefixes, ips.Prefixes()...,
)
}
rules = append(rules, tailcfg.FilterRule{
SrcIPs: capSrcIPStrs,
CapGrant: capGrants,
})
dstsHaveWildcard := sourcesHaveWildcard(otherDests)
if dstsHaveWildcard {
dstIPStrings = append(
dstIPStrings,
approvedSubnetRoutes(nodes)...,
)
}
rules = append(
rules,
companionCapGrantRules(
dstIPStrings, srcPrefixes, grant.App,
)...,
)
}
}
return rules
}
// hasPerNodeGrants reports whether any compiled grant requires
// per-node filter compilation (via grants or autogroup:self).
func hasPerNodeGrants(grants []compiledGrant) bool {
for i := range grants {
if grants[i].category != grantCategoryRegular {
return true
}
}
return false
}
// globalFilterRules extracts global filter rules from compiled
// grants. Only includes pre-compiled rules from non-via grants.
// Via grants produce no global rules (they are per-node only).
func globalFilterRules(grants []compiledGrant) []tailcfg.FilterRule {
var rules []tailcfg.FilterRule
for i := range grants {
if grants[i].category == grantCategoryVia {
continue
}
rules = append(rules, grants[i].rules...)
}
return mergeFilterRules(rules)
}
// filterRulesForNode produces unreduced filter rules for a specific
// node by combining pre-compiled global rules with per-node self and
// via rules. Regular grants emit their pre-compiled rules as-is.
// Self grants add autogroup:self expansion. Via grants add
// tag-matched, route-intersected rules.
func filterRulesForNode(
grants []compiledGrant,
node types.NodeView,
userIdx userNodeIndex,
) []tailcfg.FilterRule {
var rules []tailcfg.FilterRule
for i := range grants {
cg := &grants[i]
// Pre-compiled rules apply to all grant categories
// (empty for via-only grants).
rules = append(rules, cg.rules...)
switch cg.category {
case grantCategoryRegular:
// Nothing more to do.
case grantCategorySelf:
rules = append(
rules,
compileAutogroupSelf(cg, node, userIdx)...,
)
case grantCategoryVia:
rules = append(
rules,
compileViaForNode(cg, node)...,
)
}
}
return mergeFilterRules(rules)
}
// compileAutogroupSelf produces filter rules for autogroup:self
// destinations for a specific node. Only called for grants with
// self destinations and only produces rules for untagged nodes.
func compileAutogroupSelf(
cg *compiledGrant,
node types.NodeView,
userIdx userNodeIndex,
) []tailcfg.FilterRule {
if node.IsTagged() || cg.self == nil {
return nil
}
if !node.User().Valid() {
return nil
}
sameUserNodes := userIdx[node.User().ID()]
if len(sameUserNodes) == 0 {
return nil
}
var rules []tailcfg.FilterRule
// Filter sources to only same-user untagged devices.
srcResolved := filterSourcesToSameUser(
cg.self.resolvedSrcs, sameUserNodes,
)
if srcResolved == nil || srcResolved.Empty() {
return nil
}
// DstPorts rules from InternetProtocols.
for _, ipp := range cg.self.internetProtocols {
var destPorts []tailcfg.NetPortRange
for _, n := range sameUserNodes {
for _, port := range ipp.Ports {
for _, ip := range n.IPs() {
destPorts = append(
destPorts,
tailcfg.NetPortRange{
IP: ip.String(),
Ports: port,
},
)
}
}
}
if len(destPorts) > 0 {
rules = append(rules, tailcfg.FilterRule{
SrcIPs: srcResolved.Strings(),
DstPorts: destPorts,
IPProto: ipp.Protocol.toIANAProtocolNumbers(),
})
}
}
// CapGrant rules from App.
if cg.self.app != nil {
var (
capGrants []tailcfg.CapGrant
dstIPStrings []string
)
for _, n := range sameUserNodes {
var dsts []netip.Prefix
for _, ip := range n.IPs() {
dsts = append(
dsts,
netip.PrefixFrom(ip, ip.BitLen()),
)
dstIPStrings = append(
dstIPStrings, ip.String(),
)
}
capGrants = append(capGrants, tailcfg.CapGrant{
Dsts: dsts,
CapMap: cg.self.app,
})
}
if len(capGrants) > 0 {
rules = append(rules, tailcfg.FilterRule{
SrcIPs: srcResolved.Strings(),
CapGrant: capGrants,
})
rules = append(
rules,
companionCapGrantRules(
dstIPStrings,
srcResolved.Prefixes(),
cg.self.app,
)...,
)
}
}
return rules
}
// filterSourcesToSameUser intersects resolved source addresses with
// same-user untagged device IPs, returning only the addresses that
// belong to those devices.
func filterSourcesToSameUser(
resolvedSrcs []ResolvedAddresses,
sameUserNodes []types.NodeView,
) ResolvedAddresses {
var srcIPs netipx.IPSetBuilder
for _, ips := range resolvedSrcs {
for _, n := range sameUserNodes {
if slices.ContainsFunc(n.IPs(), ips.Contains) {
n.AppendToIPSet(&srcIPs)
}
}
}
srcResolved, err := newResolved(&srcIPs)
if err != nil {
return nil
}
return srcResolved
}
// compileViaForNode produces via-grant filter rules for a specific
// node. Only produces rules when the node matches one of the via
// tags and advertises routes that match the grant destinations.
func compileViaForNode(
cg *compiledGrant,
node types.NodeView,
) []tailcfg.FilterRule {
if cg.via == nil {
return nil
}
// Check if node matches any via tag.
matchesVia := false
for _, viaTag := range cg.via.viaTags {
if node.HasTag(string(viaTag)) {
matchesVia = true
break
}
}
if !matchesVia {
return nil
}
// Find matching destination prefixes.
nodeSubnetRoutes := node.SubnetRoutes()
if len(nodeSubnetRoutes) == 0 {
return nil
}
var viaDstPrefixes []netip.Prefix
for _, dst := range cg.via.destinations {
switch d := dst.(type) {
case *Prefix:
dstPrefix := netip.Prefix(*d)
if slices.Contains(nodeSubnetRoutes, dstPrefix) {
viaDstPrefixes = append(
viaDstPrefixes, dstPrefix,
)
}
case *AutoGroup:
// autogroup:internet via grants do not produce
// PacketFilter rules on exit nodes.
}
}
if len(viaDstPrefixes) == 0 {
return nil
}
// Build rules using pre-resolved srcIPStrings.
var rules []tailcfg.FilterRule
for _, ipp := range cg.via.internetProtocols {
var destPorts []tailcfg.NetPortRange
for _, prefix := range viaDstPrefixes {
for _, port := range ipp.Ports {
destPorts = append(
destPorts,
tailcfg.NetPortRange{
IP: prefix.String(),
Ports: port,
},
)
}
}
if len(destPorts) > 0 {
rules = append(rules, tailcfg.FilterRule{
SrcIPs: cg.via.srcIPStrings,
DstPorts: destPorts,
IPProto: ipp.Protocol.toIANAProtocolNumbers(),
})
}
}
return rules
}
+189 -261
View File
@@ -22,90 +22,184 @@ var (
errSelfInSources = errors.New("autogroup:self cannot be used in sources")
)
// companionCaps maps certain well-known Tailscale capabilities to
// their companion capability. When a grant includes one of these
// capabilities, Tailscale automatically generates an additional
// FilterRule with the companion capability and a nil CapMap value.
var companionCaps = map[tailcfg.PeerCapability]tailcfg.PeerCapability{
tailcfg.PeerCapabilityTaildrive: tailcfg.PeerCapabilityTaildriveSharer,
tailcfg.PeerCapabilityRelay: tailcfg.PeerCapabilityRelayTarget,
}
// companionCapGrantRules returns additional FilterRules for any
// well-known capabilities that have companion caps. Companion rules
// are **reversed**: SrcIPs come from the original destinations and
// CapGrant Dsts come from the original sources. This allows
// ReduceFilterRules to distribute companion rules to source nodes
// (e.g. drive-sharer goes to the member nodes, not the destination).
// Rules are ordered by the original capability name.
//
// dstIPStrings are the resolved destination IPs as strings (used as
// companion SrcIPs). srcPrefixes are the resolved source IPs as
// netip.Prefix (used as companion CapGrant Dsts).
func companionCapGrantRules(
dstIPStrings []string,
srcPrefixes []netip.Prefix,
capMap tailcfg.PeerCapMap,
) []tailcfg.FilterRule {
// Process in deterministic order by original capability name.
type pair struct {
original tailcfg.PeerCapability
companion tailcfg.PeerCapability
}
var pairs []pair
for cap, companion := range companionCaps {
if _, ok := capMap[cap]; ok {
pairs = append(pairs, pair{cap, companion})
}
}
slices.SortFunc(pairs, func(a, b pair) int {
return strings.Compare(string(a.original), string(b.original))
})
companions := make([]tailcfg.FilterRule, 0, len(pairs))
for _, p := range pairs {
companions = append(companions, tailcfg.FilterRule{
SrcIPs: dstIPStrings,
CapGrant: []tailcfg.CapGrant{
{
Dsts: srcPrefixes,
CapMap: tailcfg.PeerCapMap{
p.companion: nil,
},
},
},
})
}
return companions
}
// sourcesHaveWildcard returns true if any of the source aliases is
// a wildcard (*). Used to determine whether approved subnet routes
// should be appended to SrcIPs.
func sourcesHaveWildcard(srcs Aliases) bool {
for _, src := range srcs {
if _, ok := src.(Asterix); ok {
return true
}
}
return false
}
// sourcesHaveDangerAll returns true if any of the source aliases is
// autogroup:danger-all. When present, SrcIPs should be ["*"] to
// represent all IP addresses including non-Tailscale addresses.
func sourcesHaveDangerAll(srcs Aliases) bool {
for _, src := range srcs {
if ag, ok := src.(*AutoGroup); ok && ag.Is(AutoGroupDangerAll) {
return true
}
}
return false
}
// srcIPsWithRoutes returns the SrcIPs string slice, appending
// approved subnet routes when the sources include a wildcard.
// When hasDangerAll is true, returns ["*"] to represent all IPs.
func srcIPsWithRoutes(
resolved ResolvedAddresses,
hasWildcard bool,
hasDangerAll bool,
nodes views.Slice[types.NodeView],
) []string {
if hasDangerAll {
return []string{"*"}
}
ips := resolved.Strings()
if hasWildcard {
ips = append(ips, approvedSubnetRoutes(nodes)...)
}
return ips
}
// compileFilterRules takes a set of nodes and an ACLPolicy and generates a
// set of Tailscale compatible FilterRules used to allow traffic on clients.
func (pol *Policy) compileFilterRules(
users types.Users,
nodes views.Slice[types.NodeView],
) ([]tailcfg.FilterRule, error) {
if pol == nil || pol.ACLs == nil {
if pol == nil || (pol.ACLs == nil && len(pol.Grants) == 0) {
return tailcfg.FilterAllowAll, nil
}
var rules []tailcfg.FilterRule
return globalFilterRules(pol.compileGrants(users, nodes)), nil
}
for _, acl := range pol.ACLs {
if acl.Action != ActionAccept {
return nil, ErrInvalidAction
func (pol *Policy) destinationsToNetPortRange(
users types.Users,
nodes views.Slice[types.NodeView],
dests Aliases,
ports []tailcfg.PortRange,
) []tailcfg.NetPortRange {
var ret []tailcfg.NetPortRange
for _, dest := range dests {
// Check if destination is a wildcard - use "*" directly instead of expanding
if _, isWildcard := dest.(Asterix); isWildcard {
for _, port := range ports {
ret = append(ret, tailcfg.NetPortRange{
IP: "*",
Ports: port,
})
}
continue
}
srcIPs, err := acl.Sources.Resolve(pol, users, nodes)
// autogroup:internet does not generate packet filters - it's handled
// by exit node routing via AllowedIPs, not by packet filtering.
if ag, isAutoGroup := dest.(*AutoGroup); isAutoGroup && ag.Is(AutoGroupInternet) {
continue
}
ips, err := dest.Resolve(pol, users, nodes)
if err != nil {
log.Trace().Caller().Err(err).Msgf("resolving source ips")
log.Trace().Caller().Err(err).Msgf("resolving destination ips")
}
if srcIPs == nil || len(srcIPs.Prefixes()) == 0 {
if ips == nil {
log.Debug().Caller().Msgf("destination resolved to nil ips: %v", dest)
continue
}
protocols := acl.Protocol.parseProtocol()
prefixes := ips.Prefixes()
var destPorts []tailcfg.NetPortRange
for _, dest := range acl.Destinations {
// Check if destination is a wildcard - use "*" directly instead of expanding
if _, isWildcard := dest.Alias.(Asterix); isWildcard {
for _, port := range dest.Ports {
destPorts = append(destPorts, tailcfg.NetPortRange{
IP: "*",
Ports: port,
})
for _, pref := range prefixes {
for _, port := range ports {
pr := tailcfg.NetPortRange{
IP: pref.String(),
Ports: port,
}
// Drop the prefix bits if its a single IP.
if pref.IsSingleIP() {
pr.IP = pref.Addr().String()
}
continue
}
// autogroup:internet does not generate packet filters - it's handled
// by exit node routing via AllowedIPs, not by packet filtering.
if ag, isAutoGroup := dest.Alias.(*AutoGroup); isAutoGroup && ag.Is(AutoGroupInternet) {
continue
}
ips, err := dest.Resolve(pol, users, nodes)
if err != nil {
log.Trace().Caller().Err(err).Msgf("resolving destination ips")
}
if ips == nil {
log.Debug().Caller().Msgf("destination resolved to nil ips: %v", dest)
continue
}
prefixes := ips.Prefixes()
for _, pref := range prefixes {
for _, port := range dest.Ports {
pr := tailcfg.NetPortRange{
IP: pref.String(),
Ports: port,
}
destPorts = append(destPorts, pr)
}
ret = append(ret, pr)
}
}
if len(destPorts) == 0 {
continue
}
rules = append(rules, tailcfg.FilterRule{
SrcIPs: ipSetToPrefixStringList(srcIPs),
DstPorts: destPorts,
IPProto: protocols,
})
}
return mergeFilterRules(rules), nil
return ret
}
// compileFilterRulesForNode compiles filter rules for a specific node.
@@ -118,205 +212,10 @@ func (pol *Policy) compileFilterRulesForNode(
return tailcfg.FilterAllowAll, nil
}
var rules []tailcfg.FilterRule
grants := pol.compileGrants(users, nodes)
userIdx := buildUserNodeIndex(nodes)
for _, acl := range pol.ACLs {
if acl.Action != ActionAccept {
return nil, ErrInvalidAction
}
aclRules, err := pol.compileACLWithAutogroupSelf(acl, users, node, nodes)
if err != nil {
log.Trace().Err(err).Msgf("compiling ACL")
continue
}
for _, rule := range aclRules {
if rule != nil {
rules = append(rules, *rule)
}
}
}
return mergeFilterRules(rules), nil
}
// compileACLWithAutogroupSelf compiles a single ACL rule, handling
// autogroup:self per-node while supporting all other alias types normally.
// It returns a slice of filter rules because when an ACL has both autogroup:self
// and other destinations, they need to be split into separate rules with different
// source filtering logic.
//
//nolint:gocyclo // complex ACL compilation logic
func (pol *Policy) compileACLWithAutogroupSelf(
acl ACL,
users types.Users,
node types.NodeView,
nodes views.Slice[types.NodeView],
) ([]*tailcfg.FilterRule, error) {
var (
autogroupSelfDests []AliasWithPorts
otherDests []AliasWithPorts
)
for _, dest := range acl.Destinations {
if ag, ok := dest.Alias.(*AutoGroup); ok && ag.Is(AutoGroupSelf) {
autogroupSelfDests = append(autogroupSelfDests, dest)
} else {
otherDests = append(otherDests, dest)
}
}
protocols := acl.Protocol.parseProtocol()
var rules []*tailcfg.FilterRule
var resolvedSrcIPs []*netipx.IPSet
for _, src := range acl.Sources {
if ag, ok := src.(*AutoGroup); ok && ag.Is(AutoGroupSelf) {
return nil, errSelfInSources
}
ips, err := src.Resolve(pol, users, nodes)
if err != nil {
log.Trace().Caller().Err(err).Msgf("resolving source ips")
}
if ips != nil {
resolvedSrcIPs = append(resolvedSrcIPs, ips)
}
}
if len(resolvedSrcIPs) == 0 {
return rules, nil
}
// Handle autogroup:self destinations (if any)
// Tagged nodes don't participate in autogroup:self (identity is tag-based, not user-based)
if len(autogroupSelfDests) > 0 && !node.IsTagged() {
// Pre-filter to same-user untagged devices once - reuse for both sources and destinations
sameUserNodes := make([]types.NodeView, 0)
for _, n := range nodes.All() {
if !n.IsTagged() && n.User().ID() == node.User().ID() {
sameUserNodes = append(sameUserNodes, n)
}
}
if len(sameUserNodes) > 0 {
// Filter sources to only same-user untagged devices
var srcIPs netipx.IPSetBuilder
for _, ips := range resolvedSrcIPs {
for _, n := range sameUserNodes {
// Check if any of this node's IPs are in the source set
if slices.ContainsFunc(n.IPs(), ips.Contains) {
n.AppendToIPSet(&srcIPs)
}
}
}
srcSet, err := srcIPs.IPSet()
if err != nil {
return nil, err
}
if srcSet != nil && len(srcSet.Prefixes()) > 0 {
var destPorts []tailcfg.NetPortRange
for _, dest := range autogroupSelfDests {
for _, n := range sameUserNodes {
for _, port := range dest.Ports {
for _, ip := range n.IPs() {
destPorts = append(destPorts, tailcfg.NetPortRange{
IP: netip.PrefixFrom(ip, ip.BitLen()).String(),
Ports: port,
})
}
}
}
}
if len(destPorts) > 0 {
rules = append(rules, &tailcfg.FilterRule{
SrcIPs: ipSetToPrefixStringList(srcSet),
DstPorts: destPorts,
IPProto: protocols,
})
}
}
}
}
if len(otherDests) > 0 {
var srcIPs netipx.IPSetBuilder
for _, ips := range resolvedSrcIPs {
srcIPs.AddSet(ips)
}
srcSet, err := srcIPs.IPSet()
if err != nil {
return nil, err
}
if srcSet != nil && len(srcSet.Prefixes()) > 0 {
var destPorts []tailcfg.NetPortRange
for _, dest := range otherDests {
// Check if destination is a wildcard - use "*" directly instead of expanding
if _, isWildcard := dest.Alias.(Asterix); isWildcard {
for _, port := range dest.Ports {
destPorts = append(destPorts, tailcfg.NetPortRange{
IP: "*",
Ports: port,
})
}
continue
}
// autogroup:internet does not generate packet filters - it's handled
// by exit node routing via AllowedIPs, not by packet filtering.
if ag, isAutoGroup := dest.Alias.(*AutoGroup); isAutoGroup && ag.Is(AutoGroupInternet) {
continue
}
ips, err := dest.Resolve(pol, users, nodes)
if err != nil {
log.Trace().Caller().Err(err).Msgf("resolving destination ips")
}
if ips == nil {
log.Debug().Caller().Msgf("destination resolved to nil ips: %v", dest)
continue
}
prefixes := ips.Prefixes()
for _, pref := range prefixes {
for _, port := range dest.Ports {
pr := tailcfg.NetPortRange{
IP: pref.String(),
Ports: port,
}
destPorts = append(destPorts, pr)
}
}
}
if len(destPorts) > 0 {
rules = append(rules, &tailcfg.FilterRule{
SrcIPs: ipSetToPrefixStringList(srcSet),
DstPorts: destPorts,
IPProto: protocols,
})
}
}
}
return rules, nil
return filterRulesForNode(grants, node, userIdx), nil
}
var sshAccept = tailcfg.SSHAction{
@@ -490,7 +389,9 @@ func (pol *Policy) compileSSHPolicy(
}
if ips != nil {
dest.AddSet(ips)
for _, pref := range ips.Prefixes() {
dest.AddPrefix(pref)
}
}
}
@@ -513,7 +414,7 @@ func (pol *Policy) compileSSHPolicy(
appendRules(taggedPrincipals, 0, false)
}
} else {
if principals := ipSetToPrincipals(srcIPs); len(principals) > 0 {
if principals := resolvedAddrsToPrincipals(srcIPs); len(principals) > 0 {
rules = append(rules, &tailcfg.SSHRule{
Principals: principals,
SSHUsers: baseUserMap,
@@ -522,7 +423,7 @@ func (pol *Policy) compileSSHPolicy(
})
}
}
} else if hasLocalpart && node.InIPSet(srcIPs) {
} else if hasLocalpart && slices.ContainsFunc(node.IPs(), srcIPs.Contains) {
// Self-access: source node not in destination set
// receives rules scoped to its own user.
if node.IsTagged() {
@@ -568,6 +469,23 @@ func (pol *Policy) compileSSHPolicy(
}, nil
}
// resolvedAddrsToPrincipals converts ResolvedAddresses into SSH principals, one per address.
func resolvedAddrsToPrincipals(addrs ResolvedAddresses) []*tailcfg.SSHPrincipal {
if addrs == nil {
return nil
}
var principals []*tailcfg.SSHPrincipal
for addr := range addrs.Iter() {
principals = append(principals, &tailcfg.SSHPrincipal{
NodeIP: addr.String(),
})
}
return principals
}
// ipSetToPrincipals converts an IPSet into SSH principals, one per address.
func ipSetToPrincipals(ipSet *netipx.IPSet) []*tailcfg.SSHPrincipal {
if ipSet == nil {
@@ -635,7 +553,7 @@ func resolveLocalparts(
// Only includes nodes whose IPs are in the srcIPs set.
func groupSourcesByUser(
nodes views.Slice[types.NodeView],
srcIPs *netipx.IPSet,
srcIPs ResolvedAddresses,
) ([]uint, map[uint][]*tailcfg.SSHPrincipal, []*tailcfg.SSHPrincipal) {
userIPSets := make(map[uint]*netipx.IPSetBuilder)
@@ -727,6 +645,8 @@ func filterRuleKey(rule tailcfg.FilterRule) string {
// mergeFilterRules merges rules with identical SrcIPs and IPProto by combining
// their DstPorts. DstPorts are NOT deduplicated to match Tailscale behavior.
// CapGrant rules (which have no DstPorts) are passed through without merging
// since CapGrant and DstPorts are mutually exclusive in a FilterRule.
func mergeFilterRules(rules []tailcfg.FilterRule) []tailcfg.FilterRule {
if len(rules) <= 1 {
return rules
@@ -736,6 +656,14 @@ func mergeFilterRules(rules []tailcfg.FilterRule) []tailcfg.FilterRule {
result := make([]tailcfg.FilterRule, 0, len(rules))
for _, rule := range rules {
// CapGrant rules are not merged — they are structurally
// different from DstPorts rules and passed through as-is.
if len(rule.CapGrant) > 0 {
result = append(result, rule)
continue
}
key := filterRuleKey(rule)
if idx, exists := keyToIdx[key]; exists {
File diff suppressed because it is too large Load Diff
+25
View File
@@ -0,0 +1,25 @@
package v2
import (
"os"
"path/filepath"
"runtime"
"testing"
)
// TestMain ensures the working directory is set to the package source directory
// so that relative testdata/ paths resolve correctly when the test binary is
// executed from an arbitrary location (e.g., via "go tool stress").
func TestMain(m *testing.M) {
_, filename, _, ok := runtime.Caller(0)
if !ok {
panic("could not determine test source directory")
}
err := os.Chdir(filepath.Dir(filename))
if err != nil {
panic("could not chdir to test source directory: " + err.Error())
}
os.Exit(m.Run())
}
+200 -116
View File
@@ -46,11 +46,18 @@ type PolicyManager struct {
// Lazy map of SSH policies
sshPolicyMap map[types.NodeID]*tailcfg.SSHPolicy
// Lazy map of per-node compiled filter rules (unreduced, for autogroup:self)
compiledFilterRulesMap map[types.NodeID][]tailcfg.FilterRule
// compiledGrants are the grants with sources pre-resolved.
// The single source of truth for filter compilation. Both
// global and per-node filter rules are derived from these.
compiledGrants []compiledGrant
userNodeIdx userNodeIndex
// Lazy map of per-node filter rules (reduced, for packet filters)
filterRulesMap map[types.NodeID][]tailcfg.FilterRule
usesAutogroupSelf bool
filterRulesMap map[types.NodeID][]tailcfg.FilterRule
// needsPerNodeFilter is true when any compiled grant requires
// per-node work (autogroup:self or via grants).
needsPerNodeFilter bool
}
// filterAndPolicy combines the compiled filter rules with policy content for hashing.
@@ -71,13 +78,11 @@ func NewPolicyManager(b []byte, users []types.User, nodes views.Slice[types.Node
}
pm := PolicyManager{
pol: policy,
users: users,
nodes: nodes,
sshPolicyMap: make(map[types.NodeID]*tailcfg.SSHPolicy, nodes.Len()),
compiledFilterRulesMap: make(map[types.NodeID][]tailcfg.FilterRule, nodes.Len()),
filterRulesMap: make(map[types.NodeID][]tailcfg.FilterRule, nodes.Len()),
usesAutogroupSelf: policy.usesAutogroupSelf(),
pol: policy,
users: users,
nodes: nodes,
sshPolicyMap: make(map[types.NodeID]*tailcfg.SSHPolicy, nodes.Len()),
filterRulesMap: make(map[types.NodeID][]tailcfg.FilterRule, nodes.Len()),
}
_, err = pm.updateLocked()
@@ -91,17 +96,17 @@ func NewPolicyManager(b []byte, users []types.User, nodes views.Slice[types.Node
// updateLocked updates the filter rules based on the current policy and nodes.
// It must be called with the lock held.
func (pm *PolicyManager) updateLocked() (bool, error) {
// Check if policy uses autogroup:self
pm.usesAutogroupSelf = pm.pol.usesAutogroupSelf()
// Compile all grants once. Both global and per-node filter
// rules are derived from these compiled grants.
pm.compiledGrants = pm.pol.compileGrants(pm.users, pm.nodes)
pm.userNodeIdx = buildUserNodeIndex(pm.nodes)
pm.needsPerNodeFilter = hasPerNodeGrants(pm.compiledGrants)
var filter []tailcfg.FilterRule
var err error
// Standard compilation for all policies
filter, err = pm.pol.compileFilterRules(pm.users, pm.nodes)
if err != nil {
return false, fmt.Errorf("compiling filter rules: %w", err)
if pm.pol == nil || (pm.pol.ACLs == nil && len(pm.pol.Grants) == 0) {
filter = tailcfg.FilterAllowAll
} else {
filter = globalFilterRules(pm.compiledGrants)
}
// Hash both the compiled filter AND the policy content together.
@@ -201,7 +206,6 @@ func (pm *PolicyManager) updateLocked() (bool, error) {
// policies for nodes that have changed. Particularly if the only difference is
// that nodes has been added or removed.
clear(pm.sshPolicyMap)
clear(pm.compiledFilterRulesMap)
clear(pm.filterRulesMap)
}
@@ -371,8 +375,10 @@ func (pm *PolicyManager) BuildPeerMap(nodes views.Slice[types.NodeView]) map[typ
pm.mu.Lock()
defer pm.mu.Unlock()
// If we have a global filter, use it for all nodes (normal case)
if !pm.usesAutogroupSelf {
// If we have a global filter, use it for all nodes (normal case).
// Via grants require the per-node path because the global filter
// skips via grants (compileFilterRules: if len(grant.Via) > 0 { continue }).
if !pm.needsPerNodeFilter {
ret := make(map[types.NodeID][]types.NodeView, nodes.Len())
// Build the map of all peers according to the matchers.
@@ -395,7 +401,7 @@ func (pm *PolicyManager) BuildPeerMap(nodes views.Slice[types.NodeView]) map[typ
return ret
}
// For autogroup:self (empty global filter), build per-node peer relationships
// For autogroup:self or via grants, build per-node peer relationships
ret := make(map[types.NodeID][]types.NodeView, nodes.Len())
// Pre-compute per-node matchers using unreduced compiled rules
@@ -404,16 +410,8 @@ func (pm *PolicyManager) BuildPeerMap(nodes views.Slice[types.NodeView]) map[typ
// but peer relationships require the full bidirectional access rules.
nodeMatchers := make(map[types.NodeID][]matcher.Match, nodes.Len())
for _, node := range nodes.All() {
filter, err := pm.compileFilterRulesForNodeLocked(node)
if err != nil {
continue
}
// Include all nodes in nodeMatchers, even those with empty filters.
// Empty filters result in empty matchers where CanAccess() returns false,
// but the node still needs to be in the map so hasFilterX is true.
// This ensures symmetric visibility works correctly: if node A can access
// node B, both should see each other regardless of B's filter rules.
nodeMatchers[node.ID()] = matcher.MatchesFromFilterRules(filter)
unreduced := pm.filterRulesForNodeLocked(node)
nodeMatchers[node.ID()] = matcher.MatchesFromFilterRules(unreduced)
}
// Check each node pair for peer relationships.
@@ -430,15 +428,21 @@ func (pm *PolicyManager) BuildPeerMap(nodes views.Slice[types.NodeView]) map[typ
nodeJ := nodes.At(j)
matchersJ, hasFilterJ := nodeMatchers[nodeJ.ID()]
// If either node can access the other, both should see each other as peers.
// This symmetric visibility is required for proper network operation:
// - Admin with *:* rule should see tagged servers (even if servers
// can't access admin)
// - Servers should see admin so they can respond to admin's connections
// Check all access directions for symmetric peer visibility.
// For via grants, filter rules exist on the via-designated node
// (e.g., router-a) with sources being the client (group-a).
// We need to check BOTH:
// 1. nodeI.CanAccess(matchersI, nodeJ) — can nodeI reach nodeJ?
// 2. nodeJ.CanAccess(matchersI, nodeI) — can nodeJ reach nodeI
// using nodeI's matchers? (reverse direction: the matchers
// on the via node accept traffic FROM the source)
// Same for matchersJ in both directions.
canIAccessJ := hasFilterI && nodeI.CanAccess(matchersI, nodeJ)
canJAccessI := hasFilterJ && nodeJ.CanAccess(matchersJ, nodeI)
canJReachI := hasFilterI && nodeJ.CanAccess(matchersI, nodeI)
canIReachJ := hasFilterJ && nodeI.CanAccess(matchersJ, nodeJ)
if canIAccessJ || canJAccessI {
if canIAccessJ || canJAccessI || canJReachI || canIReachJ {
ret[nodeI.ID()] = append(ret[nodeI.ID()], nodeJ)
ret[nodeJ.ID()] = append(ret[nodeJ.ID()], nodeI)
}
@@ -448,74 +452,50 @@ func (pm *PolicyManager) BuildPeerMap(nodes views.Slice[types.NodeView]) map[typ
return ret
}
// compileFilterRulesForNodeLocked returns the unreduced compiled filter rules for a node
// when using autogroup:self. This is used by BuildPeerMap to determine peer relationships.
// For packet filters sent to nodes, use filterForNodeLocked which returns reduced rules.
func (pm *PolicyManager) compileFilterRulesForNodeLocked(node types.NodeView) ([]tailcfg.FilterRule, error) {
if pm == nil {
return nil, nil
}
// Check if we have cached compiled rules
if rules, ok := pm.compiledFilterRulesMap[node.ID()]; ok {
return rules, nil
}
// Compile per-node rules with autogroup:self expanded
rules, err := pm.pol.compileFilterRulesForNode(pm.users, node, pm.nodes)
if err != nil {
return nil, fmt.Errorf("compiling filter rules for node: %w", err)
}
// Cache the unreduced compiled rules
pm.compiledFilterRulesMap[node.ID()] = rules
return rules, nil
// filterRulesForNodeLocked returns the unreduced compiled filter rules
// for a node, combining pre-compiled global rules with per-node self
// and via rules from the stored compiled grants.
func (pm *PolicyManager) filterRulesForNodeLocked(
node types.NodeView,
) []tailcfg.FilterRule {
return filterRulesForNode(
pm.compiledGrants, node, pm.userNodeIdx,
)
}
// filterForNodeLocked returns the filter rules for a specific node, already reduced
// to only include rules relevant to that node.
// This is a lock-free version of FilterForNode for internal use when the lock is already held.
// BuildPeerMap already holds the lock, so we need a version that doesn't re-acquire it.
func (pm *PolicyManager) filterForNodeLocked(node types.NodeView) ([]tailcfg.FilterRule, error) {
// filterForNodeLocked returns the filter rules for a specific node,
// already reduced to only include rules relevant to that node.
//
// Fast path (!needsPerNodeFilter): reduces global filter per-node.
// Slow path (needsPerNodeFilter): combines global + self + via rules
// from the stored compiled grants, then reduces.
//
// Both paths derive from the same compiledGrants, ensuring there is
// no divergence between global and per-node filter output.
//
// Lock-free version for internal use when the lock is already held.
func (pm *PolicyManager) filterForNodeLocked(
node types.NodeView,
) []tailcfg.FilterRule {
if pm == nil {
return nil, nil
return nil
}
if !pm.usesAutogroupSelf {
// For global filters, reduce to only rules relevant to this node.
// Cache the reduced filter per node for efficiency.
if rules, ok := pm.filterRulesMap[node.ID()]; ok {
return rules, nil
}
// Use policyutil.ReduceFilterRules for global filter reduction.
reducedFilter := policyutil.ReduceFilterRules(node, pm.filter)
pm.filterRulesMap[node.ID()] = reducedFilter
return reducedFilter, nil
}
// For autogroup:self, compile per-node rules then reduce them.
// Check if we have cached reduced rules for this node.
if rules, ok := pm.filterRulesMap[node.ID()]; ok {
return rules, nil
return rules
}
// Get unreduced compiled rules
compiledRules, err := pm.compileFilterRulesForNodeLocked(node)
if err != nil {
return nil, err
var unreduced []tailcfg.FilterRule
if !pm.needsPerNodeFilter {
unreduced = pm.filter
} else {
unreduced = pm.filterRulesForNodeLocked(node)
}
// Reduce the compiled rules to only destinations relevant to this node
reducedFilter := policyutil.ReduceFilterRules(node, compiledRules)
reduced := policyutil.ReduceFilterRules(node, unreduced)
pm.filterRulesMap[node.ID()] = reduced
// Cache the reduced filter
pm.filterRulesMap[node.ID()] = reducedFilter
return reducedFilter, nil
return reduced
}
// FilterForNode returns the filter rules for a specific node, already reduced
@@ -530,7 +510,7 @@ func (pm *PolicyManager) FilterForNode(node types.NodeView) ([]tailcfg.FilterRul
pm.mu.Lock()
defer pm.mu.Unlock()
return pm.filterForNodeLocked(node)
return pm.filterForNodeLocked(node), nil
}
// MatchersForNode returns the matchers for peer relationship determination for a specific node.
@@ -547,19 +527,18 @@ func (pm *PolicyManager) MatchersForNode(node types.NodeView) ([]matcher.Match,
pm.mu.Lock()
defer pm.mu.Unlock()
// For global policies, return the shared global matchers
if !pm.usesAutogroupSelf {
// For global policies, return the shared global matchers.
// Via grants require per-node matchers because the global matchers
// are empty for via-grant-only policies.
if !pm.needsPerNodeFilter {
return pm.matchers, nil
}
// For autogroup:self, get unreduced compiled rules and create matchers
compiledRules, err := pm.compileFilterRulesForNodeLocked(node)
if err != nil {
return nil, err
}
// For autogroup:self or via grants, derive matchers from
// the stored compiled grants for this specific node.
unreduced := pm.filterRulesForNodeLocked(node)
// Create matchers from unreduced rules for peer relationship determination
return matcher.MatchesFromFilterRules(compiledRules), nil
return matcher.MatchesFromFilterRules(unreduced), nil
}
// SetUsers updates the users in the policy manager and updates the filter rules.
@@ -630,7 +609,6 @@ func (pm *PolicyManager) SetNodes(nodes views.Slice[types.NodeView]) (bool, erro
if !needsUpdate {
// This ensures fresh filter rules are generated for all nodes
clear(pm.sshPolicyMap)
clear(pm.compiledFilterRulesMap)
clear(pm.filterRulesMap)
}
// Always return true when nodes changed, even if filter hash didn't change
@@ -660,6 +638,14 @@ func (pm *PolicyManager) nodesHavePolicyAffectingChanges(newNodes views.Slice[ty
if newNode.HasPolicyChange(oldNode) {
return true
}
// Via grants and autogroup:self compile filter rules per-node
// that depend on the node's route state (SubnetRoutes, ExitRoutes).
// Route changes are policy-affecting in this context because they
// alter which filter rules get generated for the via-designated node.
if pm.needsPerNodeFilter && newNode.HasNetworkChanges(oldNode) {
return true
}
}
return false
@@ -821,6 +807,101 @@ func (pm *PolicyManager) NodeCanApproveRoute(node types.NodeView, route netip.Pr
return false
}
// ViaRoutesForPeer computes via grant effects for a viewer-peer pair.
// For each via grant where the viewer matches the source, it checks whether the
// peer advertises any of the grant's destination prefixes. If the peer has the
// via tag, those prefixes go into Include; otherwise into Exclude.
func (pm *PolicyManager) ViaRoutesForPeer(viewer, peer types.NodeView) types.ViaRouteResult {
var result types.ViaRouteResult
if pm == nil || pm.pol == nil {
return result
}
pm.mu.Lock()
defer pm.mu.Unlock()
// Self-steering doesn't apply.
if viewer.ID() == peer.ID() {
return result
}
grants := pm.pol.Grants
for _, acl := range pm.pol.ACLs {
grants = append(grants, aclToGrants(acl)...)
}
for _, grant := range grants {
if len(grant.Via) == 0 {
continue
}
// Check if viewer matches any grant source.
viewerMatches := false
for _, src := range grant.Sources {
ips, err := src.Resolve(pm.pol, pm.users, pm.nodes)
if err != nil {
continue
}
if ips != nil && slices.ContainsFunc(viewer.IPs(), ips.Contains) {
viewerMatches = true
break
}
}
if !viewerMatches {
continue
}
// Collect destination prefixes that the peer actually advertises.
peerSubnetRoutes := peer.SubnetRoutes()
var matchedPrefixes []netip.Prefix
for _, dst := range grant.Destinations {
switch d := dst.(type) {
case *Prefix:
dstPrefix := netip.Prefix(*d)
if slices.Contains(peerSubnetRoutes, dstPrefix) {
matchedPrefixes = append(matchedPrefixes, dstPrefix)
}
case *AutoGroup:
// autogroup:internet via grants do NOT affect AllowedIPs or
// route steering for exit nodes. Tailscale SaaS handles exit
// traffic forwarding through the client's exit node selection
// mechanism, not through AllowedIPs. Verified by golden
// captures GRANT-V14 through GRANT-V36.
}
}
if len(matchedPrefixes) == 0 {
continue
}
// Check if peer has any of the via tags.
peerHasVia := false
for _, viaTag := range grant.Via {
if peer.HasTag(string(viaTag)) {
peerHasVia = true
break
}
}
if peerHasVia {
result.Include = append(result.Include, matchedPrefixes...)
} else {
result.Exclude = append(result.Exclude, matchedPrefixes...)
}
}
return result
}
func (pm *PolicyManager) Version() int {
return 2
}
@@ -1029,12 +1110,10 @@ func (pm *PolicyManager) invalidateAutogroupSelfCache(oldNodes, newNodes views.S
// If we found the user and they're affected, clear this cache entry
if found {
if _, affected := affectedUsers[nodeUserID]; affected {
delete(pm.compiledFilterRulesMap, nodeID)
delete(pm.filterRulesMap, nodeID)
}
} else {
// Node not found in either old or new list, clear it
delete(pm.compiledFilterRulesMap, nodeID)
delete(pm.filterRulesMap, nodeID)
}
}
@@ -1049,13 +1128,14 @@ func (pm *PolicyManager) invalidateAutogroupSelfCache(oldNodes, newNodes views.S
// invalidateNodeCache invalidates cache entries based on what changed.
func (pm *PolicyManager) invalidateNodeCache(newNodes views.Slice[types.NodeView]) {
if pm.usesAutogroupSelf {
// For autogroup:self, a node's filter depends on its peers (same user).
// When any node in a user changes, all nodes for that user need invalidation.
if pm.needsPerNodeFilter {
// For autogroup:self or via grants, a node's filter depends
// on its peers. When any node changes, invalidate affected
// users' caches.
pm.invalidateAutogroupSelfCache(pm.nodes, newNodes)
} else {
// For global policies, a node's filter depends only on its own properties.
// Only invalidate nodes whose properties actually changed.
// For global policies, a node's filter depends only on its
// own properties. Only invalidate changed nodes.
pm.invalidateGlobalPolicyCache(newNodes)
}
}
@@ -1198,7 +1278,11 @@ func resolveTagOwners(p *Policy, users types.Users, nodes views.Slice[types.Node
case Alias:
// If it does not resolve, that means the tag is not associated with any IP addresses.
resolved, _ := o.Resolve(p, users, nodes)
ips.AddSet(resolved)
if resolved != nil {
for _, pref := range resolved.Prefixes() {
ips.AddPrefix(pref)
}
}
default:
// Should never happen - after flattening, all owners should be Alias types
+458 -5
View File
@@ -355,9 +355,8 @@ func TestInvalidateGlobalPolicyCache(t *testing.T) {
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
pm := &PolicyManager{
nodes: tt.oldNodes.ViewSlice(),
filterRulesMap: tt.initialCache,
usesAutogroupSelf: false,
nodes: tt.oldNodes.ViewSlice(),
filterRulesMap: tt.initialCache,
}
pm.invalidateGlobalPolicyCache(tt.newNodes.ViewSlice())
@@ -399,7 +398,7 @@ func TestAutogroupSelfReducedVsUnreducedRules(t *testing.T) {
pm, err := NewPolicyManager([]byte(policyStr), users, nodes.ViewSlice())
require.NoError(t, err)
require.True(t, pm.usesAutogroupSelf, "policy should use autogroup:self")
require.True(t, pm.needsPerNodeFilter, "policy should need per-node filter")
// Test FilterForNode returns reduced rules
// For node1: should have rules where node1 is in destinations (its own IP)
@@ -572,7 +571,7 @@ func TestAutogroupSelfPolicyUpdateTriggersMapResponse(t *testing.T) {
pm, err := NewPolicyManager([]byte(initialPolicy), users, nodes.ViewSlice())
require.NoError(t, err)
require.True(t, pm.usesAutogroupSelf, "policy should use autogroup:self")
require.True(t, pm.needsPerNodeFilter, "policy should need per-node filter")
// Get initial filter rules for test-1 (should be cached)
rules1, err := pm.FilterForNode(test1Node.View())
@@ -1339,3 +1338,457 @@ func TestIssue2990SameUserTaggedDevice(t *testing.T) {
t.Logf(" rule %d: SrcIPs=%v DstPorts=%v", i, rule.SrcIPs, rule.DstPorts)
}
}
func TestViaRoutesForPeer(t *testing.T) {
t.Parallel()
users := types.Users{
{Model: gorm.Model{ID: 1}, Name: "user1", Email: "user1@"},
{Model: gorm.Model{ID: 2}, Name: "user2", Email: "user2@"},
}
t.Run("self_returns_empty", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "router",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{mp("10.0.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("10.0.0.0/24")},
},
}
//nolint:goconst
pol := `{
"tagOwners": {
"tag:router": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[0].View())
require.Empty(t, result.Include)
require.Empty(t, result.Exclude)
})
t.Run("viewer_not_in_source", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[1]),
UserID: new(users[1].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{mp("10.0.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("10.0.0.0/24")},
},
}
//nolint:goconst
pol := `{
"tagOwners": {
"tag:router": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
// user2 is not in the grant source (user1@), so result should be empty.
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
require.Empty(t, result.Include)
require.Empty(t, result.Exclude)
})
t.Run("peer_does_not_advertise_destination", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
// Advertises 192.168.0.0/24, not 10.0.0.0/24.
RoutableIPs: []netip.Prefix{mp("192.168.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("192.168.0.0/24")},
},
}
pol := `{
"tagOwners": {
"tag:router": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
require.Empty(t, result.Include)
require.Empty(t, result.Exclude)
})
t.Run("peer_with_via_tag_include", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{mp("10.0.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("10.0.0.0/24")},
},
}
pol := `{
"tagOwners": {
"tag:router": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
require.Equal(t, []netip.Prefix{mp("10.0.0.0/24")}, result.Include)
require.Empty(t, result.Exclude)
})
t.Run("peer_without_via_tag_exclude", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "other-router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:other"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{mp("10.0.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("10.0.0.0/24")},
},
}
pol := `{
"tagOwners": {
"tag:router": ["user1@"],
"tag:other": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
// Peer has tag:other, not tag:router, so route goes to Exclude.
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
require.Empty(t, result.Include)
require.Equal(t, []netip.Prefix{mp("10.0.0.0/24")}, result.Exclude)
})
t.Run("mixed_prefix_and_autogroup_internet", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
mp("10.0.0.0/24"),
mp("0.0.0.0/0"),
mp("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
mp("10.0.0.0/24"),
mp("0.0.0.0/0"),
mp("::/0"),
},
},
}
pol := `{
"tagOwners": {
"tag:router": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["10.0.0.0/24", "autogroup:internet"],
"ip": ["*"],
"via": ["tag:router"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
result := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
// Include should have only the subnet route.
// autogroup:internet does not produce via route effects.
require.Contains(t, result.Include, mp("10.0.0.0/24"))
require.Len(t, result.Include, 1)
require.Empty(t, result.Exclude)
})
t.Run("autogroup_internet_exit_routes", func(t *testing.T) {
t.Parallel()
nodes := types.Nodes{
{
ID: 1,
Hostname: "viewer",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "exit-node",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:exit"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
mp("0.0.0.0/0"),
mp("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
mp("0.0.0.0/0"),
mp("::/0"),
},
},
{
ID: 3,
Hostname: "non-exit",
IPv4: ap("100.64.0.3"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:other"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
mp("0.0.0.0/0"),
mp("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
mp("0.0.0.0/0"),
mp("::/0"),
},
},
}
pol := `{
"tagOwners": {
"tag:exit": ["user1@"],
"tag:other": ["user1@"]
},
"grants": [{
"src": ["user1@"],
"dst": ["autogroup:internet"],
"ip": ["*"],
"via": ["tag:exit"]
}]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
// autogroup:internet via grants do NOT affect AllowedIPs or
// route steering. Tailscale SaaS handles exit traffic through
// the client's exit node mechanism, not ViaRoutesForPeer.
// Verified by golden captures GRANT-V14 through GRANT-V36.
resultExit := pm.ViaRoutesForPeer(nodes[0].View(), nodes[1].View())
require.Empty(t, resultExit.Include)
require.Empty(t, resultExit.Exclude)
resultOther := pm.ViaRoutesForPeer(nodes[0].View(), nodes[2].View())
require.Empty(t, resultOther.Include)
require.Empty(t, resultOther.Exclude)
})
t.Run("via_routes_survive_reduce_routes", func(t *testing.T) {
t.Parallel()
// This test validates that via-included routes are not
// filtered out by ReduceRoutes. The viewer's matchers
// allow tag-to-tag IP connectivity but don't explicitly
// cover the subnet prefix, so ReduceRoutes alone would
// drop it. The fix in state.RoutesForPeer applies
// ReduceRoutes first, then appends via-included routes.
nodes := types.Nodes{
{
ID: 1,
Hostname: "client",
IPv4: ap("100.64.0.1"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:group-a"},
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2,
Hostname: "router",
IPv4: ap("100.64.0.2"),
User: new(users[0]),
UserID: new(users[0].ID),
Tags: []string{"tag:router-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{mp("10.0.0.0/24")},
},
ApprovedRoutes: []netip.Prefix{mp("10.0.0.0/24")},
},
}
pol := `{
"tagOwners": {
"tag:router-a": ["user1@"],
"tag:group-a": ["user1@"]
},
"grants": [
{
"src": ["tag:group-a", "tag:router-a"],
"dst": ["tag:group-a", "tag:router-a"],
"ip": ["*"]
},
{
"src": ["tag:group-a"],
"dst": ["10.0.0.0/24"],
"ip": ["*"],
"via": ["tag:router-a"]
}
]
}`
pm, err := NewPolicyManager([]byte(pol), users, nodes.ViewSlice())
require.NoError(t, err)
client := nodes[0].View()
router := nodes[1].View()
// ViaRoutesForPeer says router should include 10.0.0.0/24.
viaResult := pm.ViaRoutesForPeer(client, router)
require.Equal(t, []netip.Prefix{mp("10.0.0.0/24")}, viaResult.Include)
require.Empty(t, viaResult.Exclude)
// Matchers for the client cover tag-to-tag connectivity
// but do NOT cover the 10.0.0.0/24 subnet prefix.
matchers, err := pm.MatchersForNode(client)
require.NoError(t, err)
require.NotEmpty(t, matchers)
// CanAccessRoute with the client's matchers returns false for
// 10.0.0.0/24 because the matchers only cover tag-to-tag IPs.
// This means ReduceRoutes would filter it out, which is why
// state.RoutesForPeer must add via routes AFTER ReduceRoutes.
canAccess := client.CanAccessRoute(matchers, mp("10.0.0.0/24"))
require.False(t, canAccess,
"client should NOT be able to access 10.0.0.0/24 via matchers alone; "+
"state.RoutesForPeer adds via routes after ReduceRoutes to fix this")
})
}
@@ -0,0 +1,552 @@
// This file implements a data-driven test runner for ACL compatibility tests.
// It loads HuJSON golden files from testdata/acl_results/acl-*.hujson and
// compares headscale's ACL engine output against the expected packet filter
// rules captured from Tailscale SaaS by the tscap tool.
//
// Each file is a testcapture.Capture containing:
// - The full policy that was POSTed to the Tailscale SaaS API
// - The 8-node topology used for the capture run
// - Expected packet_filter_rules per node (or error metadata for
// scenarios that the SaaS rejected)
//
// Test data source: testdata/acl_results/acl-*.hujson
// Source format: github.com/juanfont/headscale/hscontrol/types/testcapture
package v2
import (
"encoding/json"
"net/netip"
"path/filepath"
"strings"
"testing"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/juanfont/headscale/hscontrol/policy/policyutil"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/testcapture"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"tailscale.com/tailcfg"
)
// ptrAddr is a helper to create a pointer to a netip.Addr.
func ptrAddr(s string) *netip.Addr {
addr := netip.MustParseAddr(s)
return &addr
}
// setupACLCompatUsers returns the 3 test users for ACL compatibility tests.
// Names and emails match the anonymized identifiers tscap writes into the
// capture files (see github.com/kradalby/tscap/anonymize): users get
// norse-god names and nodes get original-151 pokémon names.
func setupACLCompatUsers() types.Users {
return types.Users{
{Model: gorm.Model{ID: 1}, Name: "odin", Email: "odin@example.com"},
{Model: gorm.Model{ID: 2}, Name: "thor", Email: "thor@example.com"},
{Model: gorm.Model{ID: 3}, Name: "freya", Email: "freya@example.com"},
}
}
// setupACLCompatNodes returns the 8 test nodes for ACL compatibility tests.
// Node GivenNames match tscap's anonymized pokémon naming.
func setupACLCompatNodes(users types.Users) types.Nodes {
return types.Nodes{
{
ID: 1, GivenName: "bulbasaur",
User: &users[0], UserID: &users[0].ID,
IPv4: ptrAddr("100.90.199.68"), IPv6: ptrAddr("fd7a:115c:a1e0::2d01:c747"),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 2, GivenName: "ivysaur",
User: &users[1], UserID: &users[1].ID,
IPv4: ptrAddr("100.110.121.96"), IPv6: ptrAddr("fd7a:115c:a1e0::1737:7960"),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 3, GivenName: "venusaur",
User: &users[2], UserID: &users[2].ID,
IPv4: ptrAddr("100.103.90.82"), IPv6: ptrAddr("fd7a:115c:a1e0::9e37:5a52"),
Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 4, GivenName: "beedrill",
IPv4: ptrAddr("100.108.74.26"), IPv6: ptrAddr("fd7a:115c:a1e0::b901:4a87"),
Tags: []string{"tag:server"}, Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 5, GivenName: "kakuna",
IPv4: ptrAddr("100.103.8.15"), IPv6: ptrAddr("fd7a:115c:a1e0::5b37:80f"),
Tags: []string{"tag:prod"}, Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 6, GivenName: "weedle",
IPv4: ptrAddr("100.83.200.69"), IPv6: ptrAddr("fd7a:115c:a1e0::c537:c845"),
Tags: []string{"tag:client"}, Hostinfo: &tailcfg.Hostinfo{},
},
{
ID: 7, GivenName: "squirtle",
IPv4: ptrAddr("100.92.142.61"), IPv6: ptrAddr("fd7a:115c:a1e0::3e37:8e3d"),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
},
{
ID: 8, GivenName: "charmander",
IPv4: ptrAddr("100.85.66.106"), IPv6: ptrAddr("fd7a:115c:a1e0::7c37:426a"),
Tags: []string{"tag:exit"}, Hostinfo: &tailcfg.Hostinfo{},
},
}
}
// findNodeByGivenName finds a node by its GivenName field.
func findNodeByGivenName(nodes types.Nodes, name string) *types.Node {
for _, n := range nodes {
if n.GivenName == name {
return n
}
}
return nil
}
// cmpOptions returns comparison options for FilterRule slices.
// It sorts SrcIPs and DstPorts to handle ordering differences.
func cmpOptions() []cmp.Option {
return []cmp.Option{
cmpopts.EquateComparable(netip.Prefix{}, netip.Addr{}),
cmpopts.SortSlices(func(a, b string) bool { return a < b }),
cmpopts.SortSlices(func(a, b tailcfg.NetPortRange) bool {
if a.IP != b.IP {
return a.IP < b.IP
}
if a.Ports.First != b.Ports.First {
return a.Ports.First < b.Ports.First
}
return a.Ports.Last < b.Ports.Last
}),
cmpopts.SortSlices(func(a, b int) bool { return a < b }),
cmpopts.SortSlices(func(a, b netip.Prefix) bool {
if a.Addr() != b.Addr() {
return a.Addr().Less(b.Addr())
}
return a.Bits() < b.Bits()
}),
// Compare json.RawMessage semantically rather than by exact
// bytes to handle indentation differences between the policy
// source and the golden capture data.
cmp.Comparer(func(a, b json.RawMessage) bool {
var va, vb any
err := json.Unmarshal(a, &va)
if err != nil {
return string(a) == string(b)
}
err = json.Unmarshal(b, &vb)
if err != nil {
return string(a) == string(b)
}
ja, _ := json.Marshal(va)
jb, _ := json.Marshal(vb)
return string(ja) == string(jb)
}),
// Compare tailcfg.RawMessage semantically (it's a string type
// containing JSON) to handle indentation differences.
cmp.Comparer(func(a, b tailcfg.RawMessage) bool {
var va, vb any
err := json.Unmarshal([]byte(a), &va)
if err != nil {
return a == b
}
err = json.Unmarshal([]byte(b), &vb)
if err != nil {
return a == b
}
ja, _ := json.Marshal(va)
jb, _ := json.Marshal(vb)
return string(ja) == string(jb)
}),
}
}
// buildACLUsersAndNodes constructs users and nodes from an ACL
// golden file's topology. This ensures the test creates the same
// nodes that were present during the Tailscale SaaS capture.
func buildACLUsersAndNodes(
t *testing.T,
tf *testcapture.Capture,
) (types.Users, types.Nodes) {
t.Helper()
users := setupACLCompatUsers()
nodes := make(types.Nodes, 0, len(tf.Topology.Nodes))
autoID := 1
for name, nodeDef := range tf.Topology.Nodes {
node := &types.Node{
ID: types.NodeID(autoID), //nolint:gosec
GivenName: name,
IPv4: ptrAddr(nodeDef.IPv4),
IPv6: ptrAddr(nodeDef.IPv6),
Tags: nodeDef.Tags,
}
autoID++
hostinfo := &tailcfg.Hostinfo{}
if len(nodeDef.RoutableIPs) > 0 {
routableIPs := make(
[]netip.Prefix, 0, len(nodeDef.RoutableIPs),
)
for _, r := range nodeDef.RoutableIPs {
routableIPs = append(
routableIPs, netip.MustParsePrefix(r),
)
}
hostinfo.RoutableIPs = routableIPs
}
node.Hostinfo = hostinfo
if len(nodeDef.ApprovedRoutes) > 0 {
approved := make(
[]netip.Prefix, 0, len(nodeDef.ApprovedRoutes),
)
for _, r := range nodeDef.ApprovedRoutes {
approved = append(
approved, netip.MustParsePrefix(r),
)
}
node.ApprovedRoutes = approved
} else {
node.ApprovedRoutes = []netip.Prefix{}
}
// Assign user — untagged nodes get user1
if len(nodeDef.Tags) == 0 {
if nodeDef.User != "" {
for i := range users {
if users[i].Name == nodeDef.User {
node.User = &users[i]
node.UserID = &users[i].ID
break
}
}
} else {
node.User = &users[0]
node.UserID = &users[0].ID
}
}
nodes = append(nodes, node)
}
return users, nodes
}
// loadACLTestFile loads and parses a single ACL capture HuJSON file.
func loadACLTestFile(t *testing.T, path string) *testcapture.Capture {
t.Helper()
c, err := testcapture.Read(path)
require.NoError(t, err, "failed to read test file %s", path)
return c
}
// aclSkipReasons documents WHY tests are expected to fail and WHAT needs to be
// implemented to fix them. Tests are grouped by root cause.
//
// Impact summary:
//
// SRCIPS_FORMAT - tests: SrcIPs use adapted format (100.64.0.0/10 vs partitioned CIDRs)
// DSTPORTS_FORMAT - tests: DstPorts IP format differences
// IPPROTO_FORMAT - tests: IPProto nil vs [6,17,1,58]
// IMPLEMENTATION_PENDING - tests: Not yet implemented in headscale
var aclSkipReasons = map[string]string{
// Currently all tests are in the skip list because the ACL engine
// output format changed with the ResolvedAddresses refactor.
// Tests will be removed from this list as the implementation is
// updated to match the expected output.
}
// TestACLCompat is a data-driven test that loads all ACL-*.json test files
// and compares headscale's ACL engine output against the expected behavior.
//
// Each JSON file contains:
// - A full policy with groups, tagOwners, hosts, and acls
// - For success cases: expected packet_filter_rules per node (5 nodes)
// - For error cases: expected error message
func TestACLCompat(t *testing.T) {
t.Parallel()
files, err := filepath.Glob(
filepath.Join("testdata", "acl_results", "acl-*.hujson"),
)
require.NoError(t, err, "failed to glob test files")
require.NotEmpty(
t,
files,
"no acl-*.hujson test files found in testdata/acl_results/",
)
t.Logf("Loaded %d ACL test files", len(files))
for _, file := range files {
tf := loadACLTestFile(t, file)
t.Run(tf.TestID, func(t *testing.T) {
t.Parallel()
// Check skip list
if reason, ok := aclSkipReasons[tf.TestID]; ok {
t.Skipf(
"TODO: %s — see aclSkipReasons for details",
reason,
)
return
}
if tf.Error {
testACLError(t, tf)
return
}
// Build nodes per-scenario from this file's topology.
// tscap uses clean-slate mode, so each scenario has
// different node IPs; using a shared topology would
// cause IP mismatches in filter rule comparisons.
users, nodes := buildACLUsersAndNodes(t, tf)
require.NotEmpty(t, nodes, "%s: topology is empty", tf.TestID)
testACLSuccess(t, tf, users, nodes)
})
}
}
// aclErrorMessageMap maps Tailscale SaaS error substrings to headscale
// equivalents. Populated as mismatches are discovered.
var aclErrorMessageMap = map[string]string{
// Add known wording differences here as they are discovered.
// Example: "tag not found" -> "undefined tag",
}
// testACLError verifies that an invalid policy produces the expected error.
func testACLError(t *testing.T, tf *testcapture.Capture) {
t.Helper()
policyJSON := convertPolicyUserEmails(tf.Input.FullPolicy)
pol, err := unmarshalPolicy(policyJSON)
if err != nil {
// Parse-time error.
if tf.Input.APIResponseBody != nil {
wantMsg := tf.Input.APIResponseBody.Message
if wantMsg != "" {
assertACLErrorContains(
t, err, wantMsg, tf.TestID,
)
}
}
return
}
err = pol.validate()
if err != nil {
if tf.Input.APIResponseBody != nil {
wantMsg := tf.Input.APIResponseBody.Message
if wantMsg != "" {
assertACLErrorContains(
t, err, wantMsg, tf.TestID,
)
}
}
return
}
t.Errorf(
"%s: expected error but policy parsed and validated successfully",
tf.TestID,
)
}
// assertACLErrorContains checks that an error message matches the
// expected Tailscale SaaS message, using progressive fallbacks:
// 1. Direct substring match
// 2. Mapped equivalent from aclErrorMessageMap
// 3. Key-part extraction (tags, autogroups, port, undefined)
// 4. t.Errorf on no match (strict)
func assertACLErrorContains(
t *testing.T,
err error,
wantMsg string,
testID string,
) {
t.Helper()
errStr := err.Error()
// 1. Direct substring match.
if strings.Contains(errStr, wantMsg) {
return
}
// 2. Mapped equivalent.
for tsKey, hsKey := range aclErrorMessageMap {
if strings.Contains(wantMsg, tsKey) &&
strings.Contains(errStr, hsKey) {
return
}
}
// 3. Key-part extraction.
for _, part := range []string{
"autogroup:self",
"not valid on the src",
"port range",
"tag not found",
"tag:",
"undefined",
"capability",
} {
if strings.Contains(wantMsg, part) &&
strings.Contains(errStr, part) {
return
}
}
// 4. No match — strict failure.
t.Errorf(
"%s: error message mismatch\n"+
" want (tailscale): %q\n"+
" got (headscale): %q",
testID,
wantMsg,
errStr,
)
}
// testACLSuccess verifies that a valid policy produces the expected
// packet filter rules for each node.
func testACLSuccess(
t *testing.T,
tf *testcapture.Capture,
users types.Users,
nodes types.Nodes,
) {
t.Helper()
// Convert Tailscale SaaS user emails to headscale @example.com format.
policyJSON := convertPolicyUserEmails(tf.Input.FullPolicy)
pol, err := unmarshalPolicy(policyJSON)
require.NoError(
t,
err,
"%s: policy should parse successfully",
tf.TestID,
)
err = pol.validate()
require.NoError(
t,
err,
"%s: policy should validate successfully",
tf.TestID,
)
for nodeName, capture := range tf.Captures {
t.Run(nodeName, func(t *testing.T) {
captureIsNull := len(capture.PacketFilterRules) == 0 ||
string(capture.PacketFilterRules) == "null" //nolint:goconst
node := findNodeByGivenName(nodes, nodeName)
if node == nil {
t.Skipf(
"node %s not found in test setup",
nodeName,
)
return
}
// Compile headscale filter rules for this node
compiledRules, err := pol.compileFilterRulesForNode(
users,
node.View(),
nodes.ViewSlice(),
)
require.NoError(
t,
err,
"%s/%s: failed to compile filter rules",
tf.TestID,
nodeName,
)
gotRules := policyutil.ReduceFilterRules(
node.View(),
compiledRules,
)
// Parse expected rules from JSON
var wantRules []tailcfg.FilterRule
if !captureIsNull {
err = json.Unmarshal(
capture.PacketFilterRules,
&wantRules,
)
require.NoError(
t,
err,
"%s/%s: failed to unmarshal expected rules",
tf.TestID,
nodeName,
)
}
// Compare
opts := append(
cmpOptions(),
cmpopts.EquateEmpty(),
)
if diff := cmp.Diff(
wantRules,
gotRules,
opts...,
); diff != "" {
t.Errorf(
"%s/%s: filter rules mismatch (-want +got):\n%s",
tf.TestID,
nodeName,
diff,
)
}
})
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,636 @@
// This file implements a data-driven test runner for grant compatibility
// tests. It loads HuJSON golden files from testdata/grant_results/grant-*.hujson
// and via-grant-*.hujson, captured from Tailscale SaaS by tscap, and compares
// headscale's grants engine output against the captured packet filter rules.
//
// Each file is a testcapture.Capture containing:
// - A full policy with grants (and optionally ACLs)
// - The expected packet_filter_rules for each of 8-15 test nodes
// - Or an error response for invalid policies
//
// Tests known to fail due to unimplemented features or known differences are
// skipped with a TODO comment explaining the root cause. As headscale's grants
// implementation improves, tests should be removed from the skip list.
//
// Test data source: testdata/grant_results/{grant,via-grant}-*.hujson
// Source format: github.com/juanfont/headscale/hscontrol/types/testcapture
package v2
import (
"encoding/json"
"net/netip"
"path/filepath"
"strings"
"testing"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/juanfont/headscale/hscontrol/policy/policyutil"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/testcapture"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"tailscale.com/tailcfg"
)
// setupGrantsCompatUsers returns the 3 test users for grants compatibility tests.
// Users get norse-god names; nodes get original-151 pokémon names — matching
// the anonymized identifiers tscap writes into the capture files
// (see github.com/kradalby/tscap/anonymize).
func setupGrantsCompatUsers() types.Users {
return types.Users{
{Model: gorm.Model{ID: 1}, Name: "odin", Email: "odin@example.com"},
{Model: gorm.Model{ID: 2}, Name: "thor", Email: "thor@example.com"},
{Model: gorm.Model{ID: 3}, Name: "freya", Email: "freya@example.com"},
}
}
// setupGrantsCompatNodes returns the 15 test nodes for grants compatibility tests.
// The node configuration matches the Tailscale test environment:
// - 3 user-owned nodes (bulbasaur, ivysaur, venusaur)
// - 12 tagged nodes (beedrill, kakuna, weedle, squirtle, charmander,
// pidgey, pidgeotto, rattata, raticate, spearow, fearow, blastoise)
func setupGrantsCompatNodes(users types.Users) types.Nodes {
nodeBulbasaur := &types.Node{
ID: 1,
GivenName: "bulbasaur",
User: &users[0],
UserID: &users[0].ID,
IPv4: ptrAddr("100.90.199.68"),
IPv6: ptrAddr("fd7a:115c:a1e0::2d01:c747"),
Hostinfo: &tailcfg.Hostinfo{},
}
nodeIvysaur := &types.Node{
ID: 2,
GivenName: "ivysaur",
User: &users[1],
UserID: &users[1].ID,
IPv4: ptrAddr("100.110.121.96"),
IPv6: ptrAddr("fd7a:115c:a1e0::1737:7960"),
Hostinfo: &tailcfg.Hostinfo{},
}
nodeVenusaur := &types.Node{
ID: 3,
GivenName: "venusaur",
User: &users[2],
UserID: &users[2].ID,
IPv4: ptrAddr("100.103.90.82"),
IPv6: ptrAddr("fd7a:115c:a1e0::9e37:5a52"),
Hostinfo: &tailcfg.Hostinfo{},
}
nodeBeedrill := &types.Node{
ID: 4,
GivenName: "beedrill",
IPv4: ptrAddr("100.108.74.26"),
IPv6: ptrAddr("fd7a:115c:a1e0::b901:4a87"),
Tags: []string{"tag:server"},
Hostinfo: &tailcfg.Hostinfo{},
}
nodeKakuna := &types.Node{
ID: 5,
GivenName: "kakuna",
IPv4: ptrAddr("100.103.8.15"),
IPv6: ptrAddr("fd7a:115c:a1e0::5b37:80f"),
Tags: []string{"tag:prod"},
Hostinfo: &tailcfg.Hostinfo{},
}
nodeWeedle := &types.Node{
ID: 6,
GivenName: "weedle",
IPv4: ptrAddr("100.83.200.69"),
IPv6: ptrAddr("fd7a:115c:a1e0::c537:c845"),
Tags: []string{"tag:client"},
Hostinfo: &tailcfg.Hostinfo{},
}
nodeSquirtle := &types.Node{
ID: 7,
GivenName: "squirtle",
IPv4: ptrAddr("100.92.142.61"),
IPv6: ptrAddr("fd7a:115c:a1e0::3e37:8e3d"),
Tags: []string{"tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.33.0.0/16")},
}
nodeCharmander := &types.Node{
ID: 8,
GivenName: "charmander",
IPv4: ptrAddr("100.85.66.106"),
IPv6: ptrAddr("fd7a:115c:a1e0::7c37:426a"),
Tags: []string{"tag:exit"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
}
// --- New nodes for expanded via grant topology ---
nodePidgey := &types.Node{
ID: 9,
GivenName: "pidgey",
IPv4: ptrAddr("100.124.195.93"),
IPv6: ptrAddr("fd7a:115c:a1e0::7837:c35d"),
Tags: []string{"tag:exit-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
}
nodePidgeotto := &types.Node{
ID: 10,
GivenName: "pidgeotto",
IPv4: ptrAddr("100.116.18.24"),
IPv6: ptrAddr("fd7a:115c:a1e0::ff37:1218"),
Tags: []string{"tag:exit-b"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
}
nodeRattata := &types.Node{
ID: 11,
GivenName: "rattata",
IPv4: ptrAddr("100.107.162.14"),
IPv6: ptrAddr("fd7a:115c:a1e0::a237:a20e"),
Tags: []string{"tag:group-a"},
Hostinfo: &tailcfg.Hostinfo{},
}
nodeRaticate := &types.Node{
ID: 12,
GivenName: "raticate",
IPv4: ptrAddr("100.77.135.18"),
IPv6: ptrAddr("fd7a:115c:a1e0::4b37:8712"),
Tags: []string{"tag:group-b"},
Hostinfo: &tailcfg.Hostinfo{},
}
nodeSpearow := &types.Node{
ID: 13,
GivenName: "spearow",
IPv4: ptrAddr("100.109.43.124"),
IPv6: ptrAddr("fd7a:115c:a1e0::a537:2b7c"),
Tags: []string{"tag:router-a"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.44.0.0/16")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.44.0.0/16")},
}
nodeFearow := &types.Node{
ID: 14,
GivenName: "fearow",
IPv4: ptrAddr("100.65.172.123"),
IPv6: ptrAddr("fd7a:115c:a1e0::5a37:ac7c"),
Tags: []string{"tag:router-b"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{netip.MustParsePrefix("10.55.0.0/16")},
},
ApprovedRoutes: []netip.Prefix{netip.MustParsePrefix("10.55.0.0/16")},
}
nodeBlastoise := &types.Node{
ID: 15,
GivenName: "blastoise",
IPv4: ptrAddr("100.105.127.107"),
IPv6: ptrAddr("fd7a:115c:a1e0::9537:7f6b"),
Tags: []string{"tag:exit", "tag:router"},
Hostinfo: &tailcfg.Hostinfo{
RoutableIPs: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
},
ApprovedRoutes: []netip.Prefix{
netip.MustParsePrefix("10.33.0.0/16"),
netip.MustParsePrefix("0.0.0.0/0"),
netip.MustParsePrefix("::/0"),
},
}
return types.Nodes{
nodeBulbasaur,
nodeIvysaur,
nodeVenusaur,
nodeBeedrill,
nodeKakuna,
nodeWeedle,
nodeSquirtle,
nodeCharmander,
nodePidgey,
nodePidgeotto,
nodeRattata,
nodeRaticate,
nodeSpearow,
nodeFearow,
nodeBlastoise,
}
}
// findGrantsNode finds a node by its GivenName in the grants test environment.
func findGrantsNode(nodes types.Nodes, name string) *types.Node {
for _, n := range nodes {
if n.GivenName == name {
return n
}
}
return nil
}
// buildGrantsNodesFromCapture constructs types.Nodes from a capture's
// topology section. Each scenario in tscap uses clean-slate mode, so
// node IPs differ between scenarios; this builds the node set with
// the IPs that were actually present during that capture.
func buildGrantsNodesFromCapture(
users types.Users,
tf *testcapture.Capture,
) types.Nodes {
nodes := make(types.Nodes, 0, len(tf.Topology.Nodes))
autoID := 1
for _, nodeDef := range tf.Topology.Nodes {
node := &types.Node{
ID: types.NodeID(autoID), //nolint:gosec
GivenName: nodeDef.Hostname,
IPv4: ptrAddr(nodeDef.IPv4),
IPv6: ptrAddr(nodeDef.IPv6),
Tags: nodeDef.Tags,
}
autoID++
hostinfo := &tailcfg.Hostinfo{}
if len(nodeDef.RoutableIPs) > 0 {
routableIPs := make([]netip.Prefix, 0, len(nodeDef.RoutableIPs))
for _, r := range nodeDef.RoutableIPs {
routableIPs = append(routableIPs, netip.MustParsePrefix(r))
}
hostinfo.RoutableIPs = routableIPs
}
node.Hostinfo = hostinfo
if len(nodeDef.ApprovedRoutes) > 0 {
approved := make([]netip.Prefix, 0, len(nodeDef.ApprovedRoutes))
for _, r := range nodeDef.ApprovedRoutes {
approved = append(approved, netip.MustParsePrefix(r))
}
node.ApprovedRoutes = approved
} else {
node.ApprovedRoutes = []netip.Prefix{}
}
// Assign user — untagged nodes look up by User field.
if len(nodeDef.Tags) == 0 && nodeDef.User != "" {
for i := range users {
if users[i].Name == nodeDef.User {
node.User = &users[i]
node.UserID = &users[i].ID
break
}
}
}
nodes = append(nodes, node)
}
return nodes
}
// convertPolicyUserEmails used to map SaaS-side emails to @example.com.
// tscap now anonymizes the policy JSON at write time (kratail2tid -> odin,
// kristoffer -> thor, monitorpasskeykradalby -> freya), so the captured
// FullPolicy is already in its final form and this is a passthrough.
func convertPolicyUserEmails(policyJSON []byte) []byte {
return policyJSON
}
// loadGrantTestFile loads and parses a single grant capture HuJSON file.
func loadGrantTestFile(t *testing.T, path string) *testcapture.Capture {
t.Helper()
c, err := testcapture.Read(path)
require.NoError(t, err, "failed to read test file %s", path)
return c
}
// Skip categories document WHY tests are expected to differ from Tailscale SaaS.
// Tests are grouped by root cause.
//
// USER_PASSKEY_WILDCARD - 2 tests: user:*@passkey wildcard pattern not supported
//
// Total: 2 tests skipped, ~246 tests expected to pass.
var grantSkipReasons = map[string]string{
// USER_PASSKEY_WILDCARD (2 tests)
//
// Tailscale SaaS policies can use user:*@passkey as a wildcard matching
// all passkey-authenticated users. headscale does not support passkey
// authentication and has no equivalent for this wildcard pattern.
"grant-k20": "USER_PASSKEY_WILDCARD: src=user:*@passkey not supported in headscale",
"grant-k21": "USER_PASSKEY_WILDCARD: dst=user:*@passkey not supported in headscale",
}
// TestGrantsCompat is a data-driven test that loads all GRANT-*.json
// test files captured from Tailscale SaaS and compares headscale's grants
// engine output against the real Tailscale behavior.
//
// Each JSON file contains:
// - A full policy (groups, tagOwners, hosts, autoApprovers, grants, optionally acls)
// - For success cases: expected packet_filter_rules per node
// - For error cases: expected error message
//
// The test converts Tailscale user email formats (@passkey, @dalby.cc) to
// headscale format (@example.com) and runs the policy through unmarshalPolicy,
// validate, compileFilterRulesForNode, and ReduceFilterRules.
//
// 2 tests are skipped for user:*@passkey wildcard (not supported in headscale).
func TestGrantsCompat(t *testing.T) {
t.Parallel()
files, err := filepath.Glob(filepath.Join("testdata", "grant_results", "*-*.hujson"))
require.NoError(t, err, "failed to glob test files")
require.NotEmpty(t, files, "no grant test files found in testdata/grant_results/")
t.Logf("Loaded %d grant test files", len(files))
users := setupGrantsCompatUsers()
for _, file := range files {
tf := loadGrantTestFile(t, file)
t.Run(tf.TestID, func(t *testing.T) {
t.Parallel()
// Check if this test is in the skip list
if reason, ok := grantSkipReasons[tf.TestID]; ok {
t.Skipf("TODO: %s — see grantSkipReasons comments for details", reason)
return
}
// Build nodes per-scenario from this file's topology.
// tscap uses clean-slate mode, so each scenario has
// different node IPs.
nodes := buildGrantsNodesFromCapture(users, tf)
// Use the captured full policy verbatim (anonymization
// in tscap already rewrote SaaS emails).
policyJSON := convertPolicyUserEmails(tf.Input.FullPolicy)
if tf.Input.APIResponseCode == 400 || tf.Error {
testGrantError(t, policyJSON, tf)
return
}
testGrantSuccess(t, policyJSON, tf, users, nodes)
})
}
}
// testGrantError verifies that an invalid policy produces the expected error.
func testGrantError(t *testing.T, policyJSON []byte, tf *testcapture.Capture) {
t.Helper()
wantMsg := ""
if tf.Input.APIResponseBody != nil {
wantMsg = tf.Input.APIResponseBody.Message
}
pol, err := unmarshalPolicy(policyJSON)
if err != nil {
// Parse-time error
if wantMsg != "" {
assertGrantErrorContains(t, err, wantMsg, tf.TestID)
}
return
}
err = pol.validate()
if err != nil {
// Validation error
if wantMsg != "" {
assertGrantErrorContains(t, err, wantMsg, tf.TestID)
}
return
}
t.Errorf("%s: expected error (api_response_code=400) but policy parsed and validated successfully; want message: %q",
tf.TestID, wantMsg)
}
// grantErrorMessageMap maps Tailscale error messages to their headscale equivalents
// where the wording differs but the meaning is the same.
var grantErrorMessageMap = map[string]string{
// Tailscale says "ip and app can not both be empty",
// headscale says "grants must specify either 'ip' or 'app' field"
"ip and app can not both be empty": "grants must specify either",
// Tailscale says "via can only be a tag",
// headscale rejects at unmarshal time via Tag.UnmarshalJSON: "tag must start with 'tag:'"
"via can only be a tag": "tag must start with",
}
// assertGrantErrorContains checks that an error message contains the expected
// Tailscale error message (or its headscale equivalent).
func assertGrantErrorContains(t *testing.T, err error, wantMsg string, testID string) {
t.Helper()
errStr := err.Error()
// First try direct substring match
if strings.Contains(errStr, wantMsg) {
return
}
// Try mapped equivalent
if mapped, ok := grantErrorMessageMap[wantMsg]; ok {
if strings.Contains(errStr, mapped) {
return
}
}
// Try matching key parts of the error message
// Extract the most distinctive part of the Tailscale message
keyParts := extractErrorKeyParts(wantMsg)
for _, part := range keyParts {
if strings.Contains(errStr, part) {
return
}
}
t.Errorf("%s: error message mismatch\n tailscale wants: %q\n headscale got: %q",
testID, wantMsg, errStr)
}
// extractErrorKeyParts extracts distinctive substrings from an error message
// that should appear in any equivalent error message.
func extractErrorKeyParts(msg string) []string {
var parts []string
// Common patterns to extract
if strings.Contains(msg, "tag:") {
// Extract tag references like tag:nonexistent
for word := range strings.FieldsSeq(msg) {
word = strings.Trim(word, `"'`)
if strings.HasPrefix(word, "tag:") {
parts = append(parts, word)
}
}
}
if strings.Contains(msg, "autogroup:") {
for word := range strings.FieldsSeq(msg) {
word = strings.Trim(word, `"'`)
if strings.HasPrefix(word, "autogroup:") {
parts = append(parts, word)
}
}
}
if strings.Contains(msg, "capability name") {
parts = append(parts, "capability")
}
if strings.Contains(msg, "port range") {
parts = append(parts, "port")
}
return parts
}
// testGrantSuccess verifies that a valid policy produces the expected
// packet filter rules for each node.
func testGrantSuccess(
t *testing.T,
policyJSON []byte,
tf *testcapture.Capture,
users types.Users,
nodes types.Nodes,
) {
t.Helper()
pol, err := unmarshalPolicy(policyJSON)
require.NoError(t, err, "%s: policy should parse successfully", tf.TestID)
err = pol.validate()
require.NoError(t, err, "%s: policy should validate successfully", tf.TestID)
for nodeName, capture := range tf.Captures {
t.Run(nodeName, func(t *testing.T) {
// Check if this node was offline during capture.
// kakuna (tag:prod) was frequently offline (132 of 188 success tests).
// When offline, packet_filter_rules is null and topology shows
// hostname="unknown" with empty tags.
captureIsNull := len(capture.PacketFilterRules) == 0 ||
string(capture.PacketFilterRules) == "null"
if captureIsNull {
topoNode, exists := tf.Topology.Nodes[nodeName]
if exists && (topoNode.Hostname == "unknown" || topoNode.Hostname == "") {
t.Skipf(
"node %s was offline during Tailscale capture (hostname=%q)",
nodeName,
topoNode.Hostname,
)
return
}
// Node was online but has null/empty rules — means Tailscale
// produced no rules. headscale should also produce no rules.
}
node := findGrantsNode(nodes, nodeName)
if node == nil {
t.Skipf(
"node %s not found in test setup (may be a test-specific node)",
nodeName,
)
return
}
// Compile headscale filter rules for this node
gotRules, err := pol.compileFilterRulesForNode(
users,
node.View(),
nodes.ViewSlice(),
)
require.NoError(
t,
err,
"%s/%s: failed to compile filter rules",
tf.TestID,
nodeName,
)
gotRules = policyutil.ReduceFilterRules(node.View(), gotRules)
// Unmarshal Tailscale expected rules from JSON capture
var wantRules []tailcfg.FilterRule
if !captureIsNull {
err = json.Unmarshal(
[]byte(capture.PacketFilterRules),
&wantRules,
)
require.NoError(
t,
err,
"%s/%s: failed to unmarshal expected rules from JSON",
tf.TestID,
nodeName,
)
}
// Compare headscale output against Tailscale expected output.
// The diff labels show (-tailscale +headscale) to make clear
// which side produced which output.
// EquateEmpty treats nil and empty slices as equal since
// Tailscale's JSON null -> nil, headscale may return empty slice.
opts := append(cmpOptions(), cmpopts.EquateEmpty())
if diff := cmp.Diff(wantRules, gotRules, opts...); diff != "" {
t.Errorf(
"%s/%s: filter rules mismatch (-tailscale +headscale):\n%s",
tf.TestID,
nodeName,
diff,
)
}
})
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -1,89 +1,73 @@
// This file is "generated" by Claude.
// It contains a data-driven test that reads SSH-*.json test files captured
// from Tailscale SaaS. Each file contains:
// - The SSH section of the policy
// - The expected SSHPolicy rules for each of 5 test nodes
// This file implements a data-driven test runner for SSH compatibility tests.
// It loads HuJSON golden files from testdata/ssh_results/ssh-*.hujson, captured
// from Tailscale SaaS by tscap, and compares headscale's SSH policy compilation
// against the captured SSH rules.
//
// The test loads each JSON file, constructs a full policy from the SSH section,
// applies it through headscale's SSH policy compilation, and compares the output
// against Tailscale's actual behavior.
// Each file is a testcapture.Capture containing:
// - The full policy that was POSTed to Tailscale SaaS (we use tf.Input.FullPolicy
// directly instead of reconstructing it from a sub-section)
// - The expected SSH rules for each of the 8 test nodes (in tf.Captures[name].SSHRules)
//
// Tests that are known to fail due to unimplemented features or known
// differences are skipped with a TODO comment explaining the root cause.
// As headscale's SSH implementation improves, tests should be removed
// from the skip list.
// Tests known to fail due to unimplemented features or known differences are
// skipped with a TODO comment explaining the root cause.
//
// Test data source: testdata/ssh_results/SSH-*.json
// Captured from: Tailscale SaaS API + tailscale debug localapi
// Test data source: testdata/ssh_results/ssh-*.hujson
// Source format: github.com/juanfont/headscale/hscontrol/types/testcapture
package v2
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/juanfont/headscale/hscontrol/types"
"github.com/juanfont/headscale/hscontrol/types/testcapture"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"tailscale.com/tailcfg"
)
// sshTestFile represents the JSON structure of a captured SSH test file.
type sshTestFile struct {
TestID string `json:"test_id"`
PolicyFile string `json:"policy_file"`
SSHSection json.RawMessage `json:"ssh_section"`
Nodes map[string]sshNodeCapture `json:"nodes"`
}
// sshNodeCapture represents the expected SSH rules for a single node.
type sshNodeCapture struct {
Rules json.RawMessage `json:"rules"`
}
// setupSSHDataCompatUsers returns the 3 test users for SSH data-driven
// compatibility tests. The user configuration matches the Tailscale test
// environment with email domains preserved for localpart matching:
// - kratail2tid@example.com (converted from @passkey)
// - kristoffer@dalby.cc (kept as-is — different domain for localpart exclusion)
// - monitorpasskeykradalby@example.com (converted from @passkey)
// compatibility tests. Users get norse-god names; nodes get original-151
// pokémon names — matching the anonymized identifiers tscap writes into
// the capture files (see github.com/kradalby/tscap/anonymize).
func setupSSHDataCompatUsers() types.Users {
return types.Users{
{
Model: gorm.Model{ID: 1},
Name: "kratail2tid",
Email: "kratail2tid@example.com",
Name: "odin",
Email: "odin@example.com",
},
{
Model: gorm.Model{ID: 2},
Name: "kristoffer",
Email: "kristoffer@dalby.cc",
Name: "thor",
Email: "thor@example.com",
},
{
Model: gorm.Model{ID: 3},
Name: "monitorpasskeykradalby",
Email: "monitorpasskeykradalby@example.com",
Name: "freya",
Email: "freya@example.com",
},
}
}
// setupSSHDataCompatNodes returns the 5 test nodes for SSH data-driven
// compatibility tests. Node GivenNames match the keys in the JSON files:
// - user1 (owned by kratail2tid)
// - user-kris (owned by kristoffer)
// - user-mon (owned by monitorpasskeykradalby)
// - tagged-server (tag:server)
// - tagged-prod (tag:prod)
// setupSSHDataCompatNodes returns the test nodes for SSH data-driven
// compatibility tests. Node GivenNames match the anonymized pokémon names:
// - bulbasaur (owned by odin)
// - ivysaur (owned by thor)
// - venusaur (owned by freya)
// - beedrill (tag:server)
// - kakuna (tag:prod)
func setupSSHDataCompatNodes(users types.Users) types.Nodes {
return types.Nodes{
&types.Node{
ID: 1,
GivenName: "user1",
GivenName: "bulbasaur",
User: &users[0],
UserID: &users[0].ID,
IPv4: ptrAddr("100.90.199.68"),
@@ -92,7 +76,7 @@ func setupSSHDataCompatNodes(users types.Users) types.Nodes {
},
&types.Node{
ID: 2,
GivenName: "user-kris",
GivenName: "ivysaur",
User: &users[1],
UserID: &users[1].ID,
IPv4: ptrAddr("100.110.121.96"),
@@ -101,7 +85,7 @@ func setupSSHDataCompatNodes(users types.Users) types.Nodes {
},
&types.Node{
ID: 3,
GivenName: "user-mon",
GivenName: "venusaur",
User: &users[2],
UserID: &users[2].ID,
IPv4: ptrAddr("100.103.90.82"),
@@ -110,7 +94,7 @@ func setupSSHDataCompatNodes(users types.Users) types.Nodes {
},
&types.Node{
ID: 4,
GivenName: "tagged-server",
GivenName: "beedrill",
IPv4: ptrAddr("100.108.74.26"),
IPv6: ptrAddr("fd7a:115c:a1e0::b901:4a87"),
Tags: []string{"tag:server"},
@@ -118,7 +102,7 @@ func setupSSHDataCompatNodes(users types.Users) types.Nodes {
},
&types.Node{
ID: 5,
GivenName: "tagged-prod",
GivenName: "kakuna",
IPv4: ptrAddr("100.103.8.15"),
IPv6: ptrAddr("fd7a:115c:a1e0::5b37:80f"),
Tags: []string{"tag:prod"},
@@ -127,119 +111,52 @@ func setupSSHDataCompatNodes(users types.Users) types.Nodes {
}
}
// convertSSHPolicyEmails converts Tailscale SaaS email domains to
// headscale-compatible format in the raw policy JSON.
//
// Tailscale uses provider-specific email formats:
// - kratail2tid@passkey (passkey auth)
// - kristoffer@dalby.cc (email auth — kept as-is)
// - monitorpasskeykradalby@passkey (passkey auth)
//
// The @passkey domain is converted to @example.com. The @dalby.cc domain
// is kept as-is to preserve localpart matching semantics (kristoffer should
// NOT match localpart:*@example.com, just as it doesn't match
// localpart:*@passkey in Tailscale SaaS).
func convertSSHPolicyEmails(s string) string {
s = strings.ReplaceAll(s, "@passkey", "@example.com")
return s
}
// constructSSHFullPolicy builds a complete headscale policy from the
// ssh_section captured from Tailscale SaaS.
//
// The base policy includes:
// - groups matching the Tailscale test environment
// - tagOwners for tag:server and tag:prod
// - A permissive ACL allowing all traffic (matches the grants wildcard
// in the original Tailscale policy)
// - The SSH section from the test file
func constructSSHFullPolicy(sshSection json.RawMessage) string {
// Base policy template with groups, tagOwners, and ACLs
// User references match the converted email addresses.
const basePolicyPrefix = `{
"groups": {
"group:admins": ["kratail2tid@example.com"],
"group:developers": ["kristoffer@dalby.cc", "kratail2tid@example.com"],
"group:empty": []
},
"tagOwners": {
"tag:server": ["kratail2tid@example.com"],
"tag:prod": ["kratail2tid@example.com"]
},
"acls": [{"action": "accept", "src": ["*"], "dst": ["*:*"]}]`
// Handle null or empty SSH section
if len(sshSection) == 0 || string(sshSection) == "null" {
// No SSH section at all (like SSH-E4)
return basePolicyPrefix + "\n}"
}
sshStr := string(sshSection)
// Convert Tailscale email domains
sshStr = convertSSHPolicyEmails(sshStr)
return basePolicyPrefix + `,
"ssh": ` + sshStr + "\n}"
}
// loadSSHTestFile loads and parses a single SSH test JSON file.
func loadSSHTestFile(t *testing.T, path string) sshTestFile {
// loadSSHTestFile loads and parses a single SSH capture HuJSON file.
func loadSSHTestFile(t *testing.T, path string) *testcapture.Capture {
t.Helper()
content, err := os.ReadFile(path)
c, err := testcapture.Read(path)
require.NoError(t, err, "failed to read test file %s", path)
var tf sshTestFile
err = json.Unmarshal(content, &tf)
require.NoError(t, err, "failed to parse test file %s", path)
return tf
return c
}
// sshSkipReasons documents why each skipped test fails and what needs to be
// fixed. Tests are grouped by root cause to identify high-impact changes.
//
// 37 of 39 tests are expected to pass.
var sshSkipReasons = map[string]string{
// user:*@domain source alias not yet implemented.
// These tests use "src": ["user:*@passkey"] which requires UserWildcard
// alias type support. Will be added in a follow-up PR that implements
// user:*@domain across all contexts (ACLs, grants, tagOwners, autoApprovers).
"SSH-B5": "user:*@domain source alias not yet implemented",
"SSH-D10": "user:*@domain source alias not yet implemented",
// user:*@passkey wildcard pattern not supported in headscale.
// headscale does not support passkey authentication and has no
// equivalent for this wildcard pattern.
"ssh-b5": "user:*@passkey wildcard not supported in headscale",
"ssh-d10": "user:*@passkey wildcard not supported in headscale",
}
// TestSSHDataCompat is a data-driven test that loads all SSH-*.json test files
// captured from Tailscale SaaS and compares headscale's SSH policy compilation
// against the real Tailscale behavior.
// TestSSHDataCompat is a data-driven test that loads all ssh-*.hujson test
// files captured from Tailscale SaaS and compares headscale's SSH policy
// compilation against the real Tailscale behavior.
//
// Each JSON file contains:
// - The SSH section of the policy
// - Expected SSH rules per node (5 nodes)
// Each capture file contains:
// - The full policy that was POSTed to the SaaS API (Input.FullPolicy)
// - Expected SSH rules per node (Captures[name].SSHRules)
//
// The test constructs a full headscale policy from the SSH section, converts
// Tailscale user email formats to headscale format, and runs the policy
// through unmarshalPolicy and compileSSHPolicy.
// The test converts Tailscale user email formats to headscale format and runs
// the captured policy through unmarshalPolicy and compileSSHPolicy.
func TestSSHDataCompat(t *testing.T) {
t.Parallel()
files, err := filepath.Glob(
filepath.Join("testdata", "ssh_results", "SSH-*.json"),
filepath.Join("testdata", "ssh_results", "ssh-*.hujson"),
)
require.NoError(t, err, "failed to glob test files")
require.NotEmpty(
t,
files,
"no SSH-*.json test files found in testdata/ssh_results/",
"no ssh-*.hujson test files found in testdata/ssh_results/",
)
t.Logf("Loaded %d SSH test files", len(files))
users := setupSSHDataCompatUsers()
nodes := setupSSHDataCompatNodes(users)
for _, file := range files {
tf := loadSSHTestFile(t, file)
@@ -257,10 +174,22 @@ func TestSSHDataCompat(t *testing.T) {
return
}
// Construct full policy from SSH section
policyJSON := constructSSHFullPolicy(tf.SSHSection)
// Skip captures the SaaS rejected — no expected SSH rules to compare against.
if tf.Error {
t.Skipf("%s: SaaS rejected the policy (api_response_code=%d); no expected SSH rules captured", tf.TestID, tf.Input.APIResponseCode)
return
}
pol, err := unmarshalPolicy([]byte(policyJSON))
// Build nodes per-scenario from this file's topology.
// tscap uses clean-slate mode, so each scenario has
// different node IPs.
nodes := buildGrantsNodesFromCapture(users, tf)
// Use the captured full policy verbatim. Anonymization in
// tscap already rewrites SaaS emails to @example.com.
policyJSON := tf.Input.FullPolicy
pol, err := unmarshalPolicy(policyJSON)
require.NoError(
t,
err,
@@ -269,15 +198,13 @@ func TestSSHDataCompat(t *testing.T) {
policyJSON,
)
for nodeName, capture := range tf.Nodes {
for nodeName, capture := range tf.Captures {
t.Run(nodeName, func(t *testing.T) {
node := findNodeByGivenName(nodes, nodeName)
require.NotNilf(
t,
node,
"node %s not found in test setup",
nodeName,
)
if node == nil {
t.Skipf("node %s not in this test's node set", nodeName)
return
}
// Compile headscale SSH policy for this node
gotSSH, err := pol.compileSSHPolicy(
@@ -296,9 +223,9 @@ func TestSSHDataCompat(t *testing.T) {
// Parse expected rules from JSON capture
var wantRules []*tailcfg.SSHRule
if len(capture.Rules) > 0 &&
string(capture.Rules) != "null" {
err = json.Unmarshal(capture.Rules, &wantRules)
if len(capture.SSHRules) > 0 &&
string(capture.SSHRules) != "null" {
err = json.Unmarshal(capture.SSHRules, &wantRules)
require.NoError(
t,
err,
@@ -323,11 +250,41 @@ func TestSSHDataCompat(t *testing.T) {
// EquateEmpty treats nil and empty slices as equal.
// Sort principals within rules (order doesn't matter).
// Do NOT sort rules — order matters (first-match-wins).
//
// Ignore SSHAction fields that are known to differ
// between headscale and Tailscale SaaS for "check"
// actions:
//
// - SessionDuration: headscale defaults to a
// 12-hour check window; SaaS emits 0s when the
// scenario policy does not specify a checkPeriod.
//
// - AllowAgentForwarding / AllowLocalPortForwarding
// / AllowRemotePortForwarding: headscale hardcodes
// these to true for both accept and check actions
// (filter.go sshAccept / sshCheck); SaaS emits
// false for check actions.
//
// - HoldAndDelegate: headscale uses a URL template
// containing "/from/…?ssh_user=$SSH_USER&local_user=…"
// so its own SSH check handler can identify the
// requested SSH user; SaaS uses "…?local_user=…"
// without the ssh_user query parameter. Comparing
// the literal template would flag every check
// action — we still assert presence via a
// separate check below.
opts := cmp.Options{
cmpopts.SortSlices(func(a, b *tailcfg.SSHPrincipal) bool {
return a.NodeIP < b.NodeIP
}),
cmpopts.EquateEmpty(),
cmpopts.IgnoreFields(tailcfg.SSHAction{},
"SessionDuration",
"AllowAgentForwarding",
"AllowLocalPortForwarding",
"AllowRemotePortForwarding",
"HoldAndDelegate",
),
}
if diff := cmp.Diff(wantSSH, gotSSH, opts...); diff != "" {
t.Errorf(
@@ -337,6 +294,32 @@ func TestSSHDataCompat(t *testing.T) {
diff,
)
}
// Separate presence check: the fields ignored by
// the diff above must still be populated on matching
// rules. This catches regressions where headscale
// would silently drop the HoldAndDelegate URL or
// flip Accept to false while we are not looking.
if wantSSH != nil && gotSSH != nil {
for i, wantRule := range wantSSH.Rules {
if i >= len(gotSSH.Rules) {
break
}
gotRule := gotSSH.Rules[i]
if wantRule.Action == nil || gotRule.Action == nil {
continue
}
wantIsCheck := wantRule.Action.HoldAndDelegate != ""
gotIsCheck := gotRule.Action.HoldAndDelegate != ""
assert.Equalf(t, wantIsCheck, gotIsCheck,
"%s/%s rule %d: HoldAndDelegate presence mismatch",
tf.TestID, nodeName, i,
)
}
}
})
}
})
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More