Description of problem:

I have a PKCS11 key setup in my ssh_config, and it requires a PIN on each use.  If I use normal ssh with this key, I see:

[lis@fedora ~]$ ssh allison@humber
Enter PIN for '/CN=SSH-key/': 
Boot Status is GREEN - Health Check SUCCESS
Last login: Tue Apr 13 21:59:20 2021 from 192.168.178.24
[allison@humber ~]$ 

but when I try to use git, scp, sftp or any other tool which takes over the stdin of ssh, it fails and falls back to password authentication, like so:

[lis@fedora ~]$ scp allison@humber:file .
ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory
login failed
pkcs11_get_key failed
sign_and_send_pubkey: signing failed for RSA "pkcs11:?module-path=/lib64/opensc-pkcs11.so": error in libcrypto
allison@humber's password: 

Enabling verbose output, the direct error is:

[lis@fedora ~]$ ssh -v allison@humber < /dev/null
OpenSSH_8.5p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021
... snip ...
debug1: read_passphrase: stdin is not a tty
ssh_askpass: exec(/usr/libexec/openssh/ssh-askpass): No such file or directory

On my Debian toolbox, this is working fine:

toolbox:~$ ssh allison@humber < /dev/null
Pseudo-terminal will not be allocated because stdin is not a terminal.
Enter PIN for '/CN=SSH-key/': 
Boot Status is GREEN - Health Check SUCCESS
toolbox:~$ 

The cause for this is a little bit complicated: the patch openssh-8.0p1-pkcs11-uri.patch in the Fedora 34 packaging introduces the RP_ALLOW_STDIN flag when calling read_passphrase() for the PIN:

@@ -262,12 +402,12 @@ pkcs11_login_slot(struct pkcs11_provider *provider, struct pkcs11_slotinfo *si,
        else {
                snprintf(prompt, sizeof(prompt), "Enter PIN for '%s': ",
                    si->token.label);
-               if ((pin = read_passphrase(prompt, RP_ALLOW_EOF)) == NULL) {
+               if ((pin = read_passphrase(prompt, RP_ALLOW_EOF|RP_ALLOW_STDIN)) == NULL) {
                        debug_f("no pin specified");
                        return (-1);    /* bail out */
                }

which seems nice and helpful, except that the RP_ALLOW_STDIN flag isn't implemented in the way that the name suggests.  From read_passphrase():

/*
 * Reads a passphrase from /dev/tty with echo turned off/on.  Returns the
 * passphrase (allocated with xmalloc).  Exits if EOF is encountered. If
 * RP_ALLOW_STDIN is set, the passphrase will be read from stdin if no
 * tty is available
 */
... okay, so far so good.  But later, in the implementation ...
        else if (flags & RP_ALLOW_STDIN) {
                if (!isatty(STDIN_FILENO)) {
                        debug("read_passphrase: stdin is not a tty");
                        use_askpass = 1;
                }

oof.  So, setting this flag, which should like it will only increase our possibilities, will actually *force* stdin to be a tty, or else force the use of ssh-askpass, causing the error.

Due to heuristics further up in the same function it's possible to work around this issue by either unsetting DISPLAY, setting is to the empty string, or by setting SSH_ASKPASS_REQUIRE to "never".

[lis@fedora ~]$ SSH_ASKPASS_REQUIRE=never ssh allison@humber < /dev/null
[lis@fedora ~]$ DISPLAY= ssh allison@humber < /dev/null
[lis@fedora ~]$ env -u DISPLAY ssh allison@humber < /dev/null

...all produce a good result.

So this is basically an upstream bug in read_passphrase(), which needs to be fixed, but the result is that ssh is broken in Fedora in a way that it's not broken upstream, because of our patch.  We need to address that ourselves.