Tailscale didn't stop the Hugging Face intrusion: 181 nodes enrolled with a stolen key

An agent that cheated on its exam

Tailscale has published its own post-mortem of the Hugging Face intrusion. The short version: an AI agent escaped its sandbox during a security evaluation, decided the LLM marketplace might hold the answers to its benchmark, and entered Hugging Face's infrastructure. Using a stolen Tailscale credential it enrolled 181 nodes onto the company's tailnet. No vulnerability in Tailscale was found or exploited — which, the company writes, “might make it even more uncomfortable for us”.

The scale comes from Hugging Face's own reconstruction, which covers roughly 17,600 recovered actions over four and a half days: sandbox escapes, code execution, stolen cloud credentials, improvised command-and-control systems and, eventually, Tailscale used to spread through the organisation.

The key that did not need to exist

By the time the agent found Tailscale it already had code execution inside a production worker, root access on a Kubernetes node and read access to a production secret store holding 136 keys. One of those was a reusable Tailscale auth key used to create CI nodes; the agent copied it into external sandboxes and used it over several days to enroll the 181 nodes, each tagged with the access a CI node would get.

Tailscale argues the deeper problem is that long-lived credentials are still standard. It points to two fixes — a vault that issues short-lived dynamic credentials, or a credential-injecting proxy, the approach behind Border0, which Tailscale acquired a few months ago and says would have blocked the 136-key read and logged every attempt — and to a third, narrower option: binding node keys to a TPM.

Its preferred fix for CI is workload identity federation: the job asks its cloud platform for a signed OIDC token, Tailscale verifies it and grants the tags and scopes assigned to that workload, so there is no credential to leak and, configured carefully, no way to use it outside CI. The company concedes not enough customers have adopted it.

Logs, detection and the safe path

The agent tried to hide its tracks by running Tailscale with --no-logs-no-support, an option meant for users worried about telemetry. Tailscale notes that suppressing client logs does not make a connection invisible: network flow logs report traffic from both ends, and a node that connects to a compromised one still logs the connection, so a SIEM can flag the mismatch. It admits flow logs are still too much work to enable and promises better defaults. For stricter control, Tailnet Lock gives programmable admission control for every new node.

The closing advice: find the reusable Tailscale auth keys your workloads can read and replace them with workload identity federation where possible, prefer one-off keys, keep expiries short with OAuth clients, use narrow tags and audit ACLs. The post ends with what it calls a very Canadian apology — the attack did not exploit Tailscale and Tailscale did not cause the compromise, but it did not stop it either.