Fedora Account System
Red Hat Associate
Red Hat Customer
# Product and environment: Product: Red Hat Edge Manager 1.3.0 (upstream: github.com/flightctl/flightctl) Deployment: standalone RHEL VM with Podman quadlets (not OpenShift) Fleet: about 165 RHEL 9 bootc kiosk devices connected over 4G links through a VPN Affected components: flightctl-api (agent endpoint, port 7443) and the remote access server Affected versions: the code in question is in current main and, as far as I can tell, has been there since 2024, so all current releases are likely affected. # Description: internal/api_server/middleware/tls.go, function TLSClientCertConnContext, calls tls.Conn.HandshakeContext(ctx) from inside http.Server.ConnContext. Go's net/http runs ConnContext synchronously in the server's accept loop, before the per-connection goroutine starts. The context passed in has no deadline, and the handshake has no timeout (unlike the handshake that net/http does itself, which is bounded by tlsHandshakeTimeout()). As a result, a client that opens a TCP connection, starts the TLS handshake and then stops responding keeps the accept loop blocked indefinitely. While this lasts, the server accepts no new connections on that listener at all. Existing connections keep working, so the service looks healthy from the outside. The same ConnContext function is used by the agent server ( internal/api_server/agentserver/server.go, via NewHTTPServerWithTLSContext) and by the remote access server ( internal/remote_access_server/server.go). # Evidence from production: Remote console from the RHEM UI failed for most devices, and new agent connections hung. The Recv-Q on the API server's listening socket stayed above zero (connections waiting, not being accepted). A goroutine dump of flightctl-api (pprof) showed the accept loop stuck in ConnContext → HandshakeContext → readClientCertificate. Every time, the blocking connection was a single device whose connection had gone silent mid-handshake for many minutes. # Suggested fix Do not perform the handshake in ConnContext. Read the peer certificate lazily from the request ( r.TLS.PeerCertificates) or in middleware, after net/http has completed the handshake with its own timeout. At minimum, run the handshake with a bounded context, for example context.WithTimeout derived from the server's configured handshake or read timeout. The first option seems cleaner, since net/http already does the handshake per connection, outside the accept loop.