From 6777a82ee6c16cbeff08a5b0c8aa72e481e80b0e Mon Sep 17 00:00:00 2001 From: Florian Preinstorfer Date: Mon, 1 Jun 2026 10:22:22 +0200 Subject: [PATCH] Rewrite reverse proxy documentation - Restructure and cleanup reverse proxy docs - Update examples for Apache, Caddy, Nginx - Provide some basic usage examples --- CHANGELOG.md | 1 + docs/ref/integration/reverse-proxy.md | 291 ++++++++++++++++++-------- 2 files changed, 200 insertions(+), 92 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 89f7337c..4702bc00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -308,6 +308,7 @@ connected" routers that maintain their control session but cannot route packets. - Fix user-owned re-registration with zero client expiry and no default storing `0001-01-01 00:00:00` in the database instead of `NULL` [#3199](https://github.com/juanfont/headscale/pull/3199) - Fix `tailscaled` restart on a node with no expiry resetting `NULL` to `0001-01-01 00:00:00` in the database, affecting both tagged and untagged nodes [#3197](https://github.com/juanfont/headscale/pull/3197) - Backfill `nodes.expiry` rows persisted by older versions as `0001-01-01 00:00:00` to `NULL`, so nodes upgraded from <0.28 stop reporting as expired [#3284](https://github.com/juanfont/headscale/issues/3284) +- Update reverse proxy documentation for `trusted_proxies` configuration option [#3292](https://github.com/juanfont/headscale/pull/3292) ## 0.28.0 (2026-02-04) diff --git a/docs/ref/integration/reverse-proxy.md b/docs/ref/integration/reverse-proxy.md index 792328eb..723ef634 100644 --- a/docs/ref/integration/reverse-proxy.md +++ b/docs/ref/integration/reverse-proxy.md @@ -1,42 +1,55 @@ -# Running headscale behind a reverse proxy +# Running Headscale behind a reverse proxy !!! warning "Community documentation" - This page is not actively maintained by the headscale authors and is - written by community members. It is _not_ verified by headscale developers. + This page is not actively maintained by the Headscale authors and is + written by community members. It is _not_ verified by Headscale developers. **It might be outdated and it might miss necessary steps**. -Running headscale behind a reverse proxy is useful when running multiple applications on the same server, and you want to reuse the same external IP and port - usually tcp/443 for HTTPS. +Running Headscale behind a reverse proxy is useful when running multiple applications on the same server, and you want +to reuse the same external IP and port - usually tcp/443 for HTTPS. -### WebSockets +Please see [limitations](#limitations) for known issues and limitations. -The reverse proxy MUST be configured to support WebSockets to communicate with Tailscale clients. +## Configuration -WebSockets support is also required when using the Headscale [embedded DERP server](../derp.md). In this case, you will also need to expose the UDP port used for STUN (by default, udp/3478). Please check our [config-example.yaml](https://github.com/juanfont/headscale/blob/main/config-example.yaml). +The configuration depends on the set of Headscale features you intend to use. Please have a look at the +[requirements](../../setup/requirements.md) and especially the [ports in use](../../setup/requirements.md#ports-in-use) +section to learn what a Tailscale clients expects. -### Cloudflare +The configuration examples in this documentation are basic and cover only HTTP and HTTPS traffic. Other features such as +STUN for Headscale's [embedded DERP server](../derp.md) are expected to be exposed directly or to be only available on +localhost. -Running headscale behind a cloudflare proxy or cloudflare tunnel is not supported and will not work as Cloudflare does not support WebSocket POSTs as required by the Tailscale protocol. See [this issue](https://github.com/juanfont/headscale/issues/1468) +### WebSocket + +Tailscale clients are using a custom protocol (Tailscale Control Protocol) to communicate with a control server such as +Headscale. The reverse proxy **must** be configured to support WebSockets in order to communicate with Tailscale clients +and it needs to handle two peculiarities of the Tailscale Control Protocol: + +- The POST method is used to upgrade the WebSocket connection. +- The value for the `Upgrade` header is `tailscale-control-protocol`. ### TLS -Headscale can be configured not to use TLS, leaving it to the reverse proxy to handle. Add the following configuration values to your headscale config file. +Headscale can be configured not to use TLS, leaving it to the reverse proxy to handle. Add the following configuration +values to your Headscale [configuration file](../configuration.md): -```yaml title="config.yaml" -server_url: https:// # This should be the FQDN at which headscale will be served -listen_addr: 0.0.0.0:8080 -metrics_listen_addr: 0.0.0.0:9090 +```yaml title="config.yaml" hl_lines="1" +server_url: https:// tls_cert_path: "" tls_key_path: "" ``` +Headscale logs `WRN listening without TLS but ServerURL does not start with http://` during startup. This is expected +and indicates that the reverse proxy is in charge of terminating TLS. + ### Trusted proxies -Headscale ignores `True-Client-IP`, `X-Real-IP` and `X-Forwarded-For` -unless the request's TCP peer matches `trusted_proxies`. Set this to -the CIDR(s) your reverse proxy connects from so the real client IP -appears in access logs: +Headscale ignores `True-Client-IP`, `X-Real-IP` and `X-Forwarded-For` headers unless the request's TCP peer matches the +`trusted_proxies` configuration option. Set this to the CIDR(s) your reverse proxy connects from so the real client IP +appears in access logs. ```yaml title="config.yaml" trusted_proxies: @@ -44,71 +57,130 @@ trusted_proxies: - ::1/128 ``` -The reverse proxy must also strip any client-supplied -`True-Client-IP` / `X-Real-IP` / `X-Forwarded-For` on inbound requests -and set its own values. nginx's `$proxy_add_x_forwarded_for` only -appends to whatever the client sent — pair it with -`proxy_set_header X-Real-IP $remote_addr;` and clear the inbound XFF -yourself if your nginx version does not do so. +The reverse proxy is responsible to replace any client-supplied `True-Client-IP`, `X-Real-IP`, `X-Forwarded-For` headers +on inbound requests with sanitized values. Headscale picks the first valid IP address supplied by headers in this order: -Leaving `trusted_proxies` empty when there is no proxy in front is -safe: the headers are dropped from every request and the access log -shows the directly-connecting TCP peer. +- `True-Client-IP` +- `X-Real-IP` +- `X-Forwarded-For` -## nginx +## Limitations -The following example configuration can be used in your nginx setup, substituting values as necessary. `` should be the IP address and port where headscale is running. In most cases, this will be `http://localhost:8080`. +- A reverse proxy adds another layer of complexity that needs to be able to handle the [Tailscale Control + Protocol](#websocket) properly. Be sure to test your setup without a reverse proxy before raising an issue. +- STUN (used along with the [embedded DERP server](../derp.md)) requires udp/3478 to be served publicly. +- [gRPC](../api.md#grpc) (used to remote control Headscale) may not be proxied. -```nginx title="nginx.conf" -map $http_upgrade $connection_upgrade { - default upgrade; - '' close; +## Reverse proxy specific configuration + +!!! warning "Third-party software and services" + + This section of the documentation is specific for third-party software and services. We recommend users read the + third-party documentation for a secure configuration. + +This following Headscale configuration may be used as base for the various reverse proxy examples below. The following +is [assumed](../../setup/requirements.md): + +- Service for Tailscale clients is served via HTTPS on port 443. +- The reverse proxy redirects HTTP to HTTPS and is terminating TLS. +- Both Headscale and the reverse proxy are running on the same host. +- [Metrics](../debug.md#metrics-and-debug-endpoint) and [gRPC](../api.md#grpc) are not proxied, those are available via + localhost. + +```yaml title="config.yaml" hl_lines="1" +server_url: https:// +listen_addr: 127.0.0.1:8080 +metrics_listen_addr: 127.0.0.1:9090 +grpc_listen_addr: 127.0.0.1:50443 +trusted_proxies: + - 127.0.0.1/32 + - ::1/128 +tls_cert_path: "" +tls_key_path: "" +``` + +### Apache + +The following basic Apache configuration works with the Headscale configuration [as shown +above](#reverse-proxy-specific-configuration). Substitute placeholders and adjust the configuration as needed: + +- ``: The server name for your instance, e.g. `headscale.example.com` +- ``: Absolute path to your TLS certificate +- ``: Absolute path to your TLS private key + +```apache title="apache.conf" hl_lines="2 7 11 14-15" + + ServerName + + # Tailscale captive portal detection + RedirectMatch 204 ^/generate_204$ + + RedirectMatch permanent "^/(.*)$" "https:///$1" + + + + ServerName + + SSLEngine On + SSLCertificateFile + SSLCertificateKeyFile + + RequestHeader set True-Client-IP "%{REMOTE_ADDR}s" + RequestHeader set X-Real-IP "%{REMOTE_ADDR}s" + + ProxyPreserveHost On + ProxyPass / http://127.0.0.1:8080/ upgrade=any + +``` + +Note that `upgrade=any` is required as a parameter for `ProxyPass` so that WebSocket traffic whose `Upgrade` header +value is not equal to `WebSocket` (i. e. Tailscale Control Protocol) is forwarded correctly. See the [Apache +docs](https://httpd.apache.org/docs/current/mod/mod_proxy.html#upgrade) for more information on this. + +### Caddy + +The following basic Caddyfile works with the Headscale configuration [as shown +above](#reverse-proxy-specific-configuration). Substitute placeholders and adjust the configuration as needed: + +- ``: The server name for your instance, e.g. `headscale.example.com` + +```none title="Caddyfile" hl_lines="1 12" +http:// { + # Tailscale captive portal detection + handle /generate_204 { + respond 204 + } + + handle * { + redir https://{host}{uri} + } } -server { - listen 80; - listen [::]:80; - - listen 443 ssl http2; - listen [::]:443 ssl http2; - - server_name ; - - ssl_certificate ; - ssl_certificate_key ; - ssl_protocols TLSv1.2 TLSv1.3; - - location / { - proxy_pass http://; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection $connection_upgrade; - proxy_set_header Host $server_name; - proxy_redirect http:// https://; - proxy_buffering off; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; - add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always; - } + { + reverse_proxy 127.0.0.1:8080 { + header_up True-Client-IP {remote_host} + header_up X-Real-IP {remote_host} + } } ``` -## istio/envoy +Caddy will [automatically](https://caddyserver.com/docs/automatic-https) provision a certificate for your +domain/subdomain, force HTTPS, and proxy WebSocket connections. -If you using [Istio](https://istio.io/) ingressgateway or [Envoy](https://www.envoyproxy.io/) as reverse proxy, there are some tips for you. If not set, you may see some debug log in proxy as below: +### Cloudflare -```log -Sending local reply with details upgrade_failed -``` +Running Headscale behind a Cloudflare Proxy or Cloudflare Tunnel is not supported and will not work as Cloudflare does +not support [WebSocket POSTs as required by the Tailscale protocol](#websocket). See [issue +1468](https://github.com/juanfont/headscale/issues/1468) for more information. ### Envoy -You need to add a new upgrade_type named `tailscale-control-protocol`. [see details](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-upgradeconfig) +You need to add a new upgrade_type named `tailscale-control-protocol`. [See +details](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto#extensions-filters-network-http-connection-manager-v3-httpconnectionmanager-upgradeconfig). ### Istio -Same as envoy, we can use `EnvoyFilter` to add upgrade_type. +Same as [envoy](#envoy), we can use `EnvoyFilter` to add a new upgrade_type named `tailscale-control-protocol`. ```yaml apiVersion: networking.istio.io/v1alpha3 @@ -133,33 +205,68 @@ spec: - upgrade_type: tailscale-control-protocol ``` -## Caddy +### Nginx -The following Caddyfile is all that is necessary to use Caddy as a reverse proxy for headscale, in combination with the `config.yaml` specifications above to disable headscale's built in TLS. Replace values as necessary - `` should be the FQDN at which headscale will be served, and `` should be the IP address and port where headscale is running. In most cases, this will be `localhost:8080`. +The following basic Nginx configuration works with the Headscale configuration [as shown +above](#reverse-proxy-specific-configuration). Substitute placeholders and adjust the configuration as needed: -```none title="Caddyfile" - { - reverse_proxy +- ``: The server name for your instance, e.g. `headscale.example.com` +- ``: Absolute path to your TLS certificate +- ``: Absolute path to your TLS private key + +```nginx title="nginx.conf" hl_lines="19 37 39-40" +# headscale +upstream headscale { + zone upstreams 64K; + server 127.0.0.1:8080 max_fails=1 fail_timeout=5s; + keepalive 2; +} + +# websocket +map $http_upgrade $connection_upgrade { + default keep-alive; + '' close; +} + +# http +server { + listen 80; + listen [::]:80; + + server_name ; + + # Tailscale captive portal detection + location = /generate_204 { + return 204; + } + + location / { + return 301 https://$server_name$request_uri; + } +} + +# https +server { + listen 443 ssl; + listen [::]:443 ssl; + http2 on; + + server_name ; + + ssl_certificate ; + ssl_certificate_key ; + + location / { + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + proxy_set_header Host $host; + proxy_set_header True-Client-IP $remote_addr; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_buffering off; + proxy_pass http://headscale; + } } ``` - -Caddy v2 will [automatically](https://caddyserver.com/docs/automatic-https) provision a certificate for your domain/subdomain, force HTTPS, and proxy websockets - no further configuration is necessary. - -For a slightly more complex configuration which utilizes Docker containers to manage Caddy, headscale, and Headscale-UI, [Guru Computing's guide](https://blog.gurucomputing.com.au/smart-vpns-with-headscale/) is an excellent reference. - -## Apache - -The following minimal Apache config will proxy traffic to the headscale instance on ``. Note that `upgrade=any` is required as a parameter for `ProxyPass` so that WebSockets traffic whose `Upgrade` header value is not equal to `WebSocket` (i. e. Tailscale Control Protocol) is forwarded correctly. See the [Apache docs](https://httpd.apache.org/docs/2.4/mod/mod_proxy_wstunnel.html) for more information on this. - -```apache title="apache.conf" - - ServerName - - ProxyPreserveHost On - ProxyPass / http:/// upgrade=any - - SSLEngine On - SSLCertificateFile - SSLCertificateKeyFile - -```