GitHub post-mortem: a misconfigured Istio policy and a retry storm behind the 7h 47m outage

GitHub has published a report on the incident that degraded its platform on August 17, 2026. The disruption began at 13:28 UTC, lasted 7 hours and 47 minutes, and was fully resolved at 21:15 UTC.

What happened

The first public update came at 13:40 UTC, when the company said it was investigating reports of impacted performance across several services. By 13:45 it reported an approximate 20% error rate affecting Pull Requests, Issues and other experiences, and further updates flagged degraded API requests, Actions, Webhooks and Pages. At peak, error rates on web and API traffic reached about 20%, while archive downloads and raw repository content downloads hit roughly 50%. SAML and OIDC authentication, SCIM and Team Sync were affected as well, together with Actions workflows in GitHub Enterprise Cloud with Data Residency that depend on public workflow step definitions hosted on GitHub.com.

Most services recovered by 16:36 UTC, when the Central US datacenter recovered. Actions remained degraded until approximately 18:03 UTC, and the Copilot Token Service fully recovered by 21:02 UTC.

Root cause: load-balancer saturation and a retry storm

According to GitHub, the immediate cause was network saturation on load balancers in its Central US datacenter due to a new peak in traffic. An Istio sidecar pod reached its concurrency limits and failed to autoscale correctly, because a policy was misconfigured to watch host service limits but not sidecar limits. The failure cascaded until four HAProxy nodes exhausted their flow limits, degrading the gateway authentication path and causing widespread authentication latency and failures. Optimistic retry logic made things worse by overloading internal load balancers; pausing HAProxy on the affected nodes produced immediate, broad recovery.

A second problem compounded the first: delayed replies from a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic roughly tenfold. Requests to the Copilot Token Service rose from a normal 7–9K per second to 70–100K per second. GitHub fixed the retry storm by temporarily reducing gateway retry logic and blocking inbound Copilot token requests at the load balancers with a 403 response, then gradually ramping traffic back up site by site. Scraping attacks on codeload endpoints further complicated recovery.

What GitHub plans to change

The report lists five follow-up actions: correcting autoscaling policies so they account for service-mesh sidecar concurrency and capacity; auditing Istio request, concurrency and scaling limits across affected services; reviewing retry limits and backoff behaviour in gateways and clients; addressing the VS Code retry behaviour that amplified Copilot token traffic; and improving load-balancer capacity monitoring together with regional failover safeguards.