The package openssh-clients ships with two systemd files, namely /usr/lib/systemd/user/ssh-agent.service and /usr/lib/systemd/user/ssh-agent.socket

The service file has the following code:
ExecStartPre=/usr/bin/rm -f $SSH_AUTH_SOCK
ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK

As one can see it removes any existing $SSH_AUTH_SOCK and recreates it's own. The socket unit creates the socket and expects the target to support socket activation (which openssh does not support) and on the first access the service unit is started, deletes the socket and then start ssh-agent.

At this point the program that initially connected to the socket is broken/hanging (in my case this is keepassxc) because nothing will answer on that fd.

The relevant docs say (https://www.freedesktop.org/software/systemd/man/systemd.socket.html):

Note that the daemon software configured for socket activation with socket units needs to be able to accept sockets from systemd, either via systemd's native socket passing interface (see sd_listen_fds(3) for details about the precise protocol used and the order in which the file descriptors are passed) or via traditional inetd(8)-style socket passing (i.e. sockets passed in via standard input and output, using StandardInput=socket in the service file).

If ssh-agent doesn't support either of those options I think it shouldn't have a socket unit.