Cloudflare Hyperdrive MySQL Support Now Generally Available After 20 Bug Fixes

Cloudflare has announced general availability for MySQL support in Hyperdrive, its distributed database connection pooler for Workers. The service originally supported only Postgres, and the beta for MySQL protocol support was announced last year. According to the company, the difference between beta and GA came down to 20 bug fixes.

Encoding bugs fixed during the beta

One of the first problems reported in the beta was that emojis written to the database were not returned correctly. The issue was traced to character set encoding. Hyperdrive negotiates the encoding during the connection startup handshake and was using utf8, which is a deprecated alias for utf8mb3. That encoding supports only 1 to 3 bytes per Unicode code point, while full UTF-8 allows up to 4 bytes per code point. The team changed the handshake to negotiate utf8mb4, which fixed the emoji issue.

A second encoding issue appeared after Hyperdrive added connection cleanup via COM_RESET_CONNECTION, a lightweight MySQL command that clears session state so that no session-level state leaks between checkouts. The problem is that COM_RESET_CONNECTION resets a connection to the server default encoding rather than the one negotiated at startup. On a MySQL server defaulting to latin1, the connection switched to latin1 after a reset. The fix was to always run a command setting the encoding to utf8mb4 after a connection reset.

A rare caching bug

During the beta, Hyperdrive's highest-volume MySQL customer reported a small but consistent rate of empty query responses where a non-empty response was always expected. The error occurred about 0.001% of the time, making it difficult to capture relevant end-to-end execution traces. One clue stood out: the bug only happened on Hyperdrive instances with caching enabled.

Each Hyperdrive instance uses a split architecture with a Client node and an Endpoint node. The Client runs on the same machine as the user's Worker and forwards queries if needed, while the Endpoint holds the connection pool in the data center closest to the origin database. Both nodes use the Cloudflare CDN cache local to their data center, so if they run in the same data center they share a cache. Correct operation requires that the same bytes be cached for the same response, regardless of which node stored them.

The investigation eventually pointed at capability negotiation in the MySQL protocol. Hyperdrive acts as a client to the origin database and as a server to the Worker, and the two sides can support different capabilities. For example, the popular Node.js library mysql2 does not support CLIENT_DEPRECATE_EOF, which changes how result sets are terminated. Hyperdrive therefore rewrites result sets, adding EOF packets and translating OK packets to EOFs as needed to match what the Worker driver expects.

The root cause was an assumption carried over from Postgres caching: that the exact bytes written back to the Worker can be cached. For MySQL this does not hold, because the Client writes bytes translated to the Worker's capabilities while the Endpoint always works with Hyperdrive's own capabilities. When a Client and an Endpoint share a cache, one could read what the other stored. In one scenario, the Endpoint cached response bytes and a Client served them to a Worker without translating. In the other, the Client cached translated bytes, the Endpoint read them and passed them to a different Client, which tried to translate them again and sent an early EOF or hung.

The fix was to always write back a canonical version of the query result bytes in the cache, encoded using Hyperdrive's capabilities, no matter which node stored them. After the change, the empty row problem disappeared and the success rate for Client connections reached the 99.99% availability target.

Sources: Cloudflare