It turned out that the new OpenSSL is sending more error codes than OpenSSH is expecting. The parsing of them is kind of fragile too. When I try to print the errors, there are new errors from the decoder layer and from passphrase, which are not recognized by openssh:

[root@rhel-9-0-0-20210629-1 openssh-portable]# ./ssh-keygen -y -f ../test.key 
4087F1A8C07F0000:error:07880109:common libcrypto routines:do_ui_passphrase:interrupted or cancelled:crypto/passphrase.c:175:
4087F1A8C07F0000:error:04800068:PEM routines:PEM_do_header:bad password read:crypto/pem/pem_lib.c:439:
4087F1A8C07F0000:error:1E08010C:DECODER routines:OSSL_DECODER_from_bio:unsupported:crypto/encode_decode/decoder_lib.c:83:Not supported for the data to decode.

OpenSSH is using ERR_peek_error() and ERR_peek_last_error() to look for errors, but it does not know either of these (it expects the middle one).

There are couple of options what we could do:
 * look into all of the errors in order until we find some familiar error suggesting the passphrase is wrong (probably best)
 * add new error codes that we will consider as a wrong passphrase
 * the cipher/passphrase.c in openssl suggests it can prompt for passphrase itself through some UI, but I do not think this is a good idea.

I tried the following patch, which allowed me to use the key I previously generated on my Fedora and that was failing to be accepted previously in RHEL9, but it will require some more love:

diff --git a/sshkey.c b/sshkey.c
index 51482929..7317549a 100644
--- a/sshkey.c
+++ b/sshkey.c
@@ -4519,6 +4519,18 @@ translate_libcrypto_error(unsigned long pem_err)
 		default:
 			return SSH_ERR_LIBCRYPTO_ERROR;
 		}
+#ifdef ERR_LIB_OSSL_DECODER
+	case ERR_LIB_OSSL_DECODER:
+		switch (pem_reason) {
+#ifdef ERR_R_UNSUPPORTED
+		case ERR_R_UNSUPPORTED:
+			return SSH_ERR_KEY_WRONG_PASSPHRASE;
+#endif
+		default:
+			return SSH_ERR_LIBCRYPTO_ERROR;
+		}
+
+#endif
 	case ERR_LIB_ASN1:
 		return SSH_ERR_INVALID_FORMAT;
 	}

RHEL 8.5 in this case requests the passphrase which I think is the desired behaviour

https://github.com/openssl/openssl/pull/14834 seems to be the relevant OpenSSL changeset

Created attachment 1804145 [details]
An openssl 3.0-compatibility patch

Works for OpenSSL 3.0-beta1 without any patches.