Hi, I was working on RHED 6.8 and upgraded to Fedora 28 recently. SSH failed to load my ECDSA key with error Error loading key "id_ecdsa": invalid format let me say, that I restored private key several times to be sure, it is correct, also rights are fine. So I created virtual server, installed CentOS 6.9 and tested key there - and all was fine. I was digging around for last weeks and find some suggestions about missing EC definitions. So I tested those, but both openssl knows correct EC. Here is comparison of versions OLD = CentOS 6.9 NEW = Fedora 28 OLD rpms: openssh-5.3p1-123.el6_9.x86_64 openssl-1.0.1e-57.el6.x86_64 NEW rpms: openssh-7.7p1-5.fc28.x86_64 openssl-libs-1.1.0h-3.fc28.i686 OLD openssl ecparam -list_curves secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field # <--- NEW openssl ecparam -list_curves secp224r1 : NIST/SECG curve over a 224 bit prime field secp256k1 : SECG curve over a 256 bit prime field secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field # <--- prime256v1: X9.62/SECG curve over a 256 bit prime field key was created using secp521r1 - so both knows this EC ! also new openssl can validate key: NEW: openssl ec -in id_ecdsa -check read EC key EC Key valid. # <-------- check option is not present in previous version (but that should be OK too, when it works there) So this seems to be issue with openssh , it can not read ecdsa keys created in previous version openssl/h. NEW: ssh-add id_ecdsa Error loading key "id_ecdsa": invalid format
That is interesting. I assume you still use the key so you will not want to share it. Can you share a debug log from the new openssh trying to use the key, such as ssh -vvv -i id_ecdsa example.com
Ahoj Jakube, I do not know, if log will be helpful, I see only relevant info in following lines debug1: Trying private key: id_ecdsa Load key "id_ecdsa": invalid format but anyway, full log is here $ ssh -p 2222 -vvv -i id_ecdsa forpsi OpenSSH_7.7p1, OpenSSL 1.1.0h-fips 27 Mar 2018 debug1: Reading configuration data /home/prema/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug3: /etc/ssh/ssh_config line 52: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0 debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-] debug3: kex names ok: [curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1] debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for * debug2: resolving "forpsi" port 2222 debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to forpsi [80.211.194.91] port 2222. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file id_ecdsa-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.7 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3 debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000 debug2: fd 4 setting O_NONBLOCK debug1: Authenticating to forpsi:2222 as 'prema' debug3: put_host_port: [forpsi]:2222 debug3: hostkeys_foreach: reading file "/home/prema/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/prema/.ssh/known_hosts:20 debug3: load_hostkeys: loaded 1 keys from [forpsi]:2222 debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,ssh-ed25519-cert-v01,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519 debug2: ciphers ctos: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: ciphers stoc: aes256-gcm,chacha20-poly1305,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc debug2: MACs ctos: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: MACs stoc: hmac-sha2-256-etm,hmac-sha1-etm,umac-128-etm,hmac-sha2-512-etm,hmac-sha2-256,hmac-sha1,umac-128,hmac-sha2-512 debug2: compression ctos: none,zlib,zlib debug2: compression stoc: none,zlib,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: host key algorithms: ssh-rsa,ssh-dss debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc.se debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160,hmac-sha1-96,hmac-md5-96 debug2: compression ctos: none,zlib debug2: compression stoc: none,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-256 compression: none debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-256 compression: none debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32 debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32 debug3: send packet: type 34 debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent debug3: receive packet: type 31 debug1: got SSH2_MSG_KEX_DH_GEX_GROUP debug2: bits set: 4066/8192 debug3: send packet: type 32 debug1: SSH2_MSG_KEX_DH_GEX_INIT sent debug3: receive packet: type 33 debug1: got SSH2_MSG_KEX_DH_GEX_REPLY debug1: Server host key: ssh-rsa SHA256:+oZe9VtO47fD4CGJgIBCxDxGj4PAiuZnGN4hnS1ln24 debug3: put_host_port: [80.211.194.91]:2222 debug3: put_host_port: [forpsi]:2222 debug3: hostkeys_foreach: reading file "/home/prema/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/prema/.ssh/known_hosts:20 debug3: load_hostkeys: loaded 1 keys from [forpsi]:2222 debug3: hostkeys_foreach: reading file "/home/prema/.ssh/known_hosts" debug3: record_hostkey: found key type RSA in file /home/prema/.ssh/known_hosts:20 debug3: load_hostkeys: loaded 1 keys from [80.211.194.91]:2222 debug1: Host '[forpsi]:2222' is known and matches the RSA host key. debug1: Found key in /home/prema/.ssh/known_hosts:20 debug2: bits set: 4183/8192 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 4294967296 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 4294967296 blocks debug2: key: id_ecdsa ((nil)), explicit debug3: send packet: type 5 debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup gssapi-keyex debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_is_enabled gssapi-keyex debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug2: we did not send a packet, disable method debug3: authmethod_lookup gssapi-with-mic debug3: remaining preferred: publickey,keyboard-interactive,password debug3: authmethod_is_enabled gssapi-with-mic debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available (default cache: KEYRING:persistent:1000) debug2: we did not send a packet, disable method debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Trying private key: id_ecdsa Load key "id_ecdsa": invalid format debug2: we did not send a packet, disable method debug3: authmethod_lookup password debug3: remaining preferred: ,password debug3: authmethod_is_enabled password debug1: Next authentication method: password
The ecdsa support in RHEL6 was backported from newer versions so there might some some issues with writing the file or generating the key correctly. I see I can reproduce the same with my RHEL6 virtual machine and my Fedora 28 workstation: $ ssh-keygen -vvv -lf ecdsa debug1: load public "ecdsa": No such file or directory debug1: load private "ecdsa": invalid format ecdsa is not a key file. Inspecting the generated key, using openssl shows that there are no named curves OID, but raw parameters that reflect the P-521 NIST curve: $ openssl ec -in ecdsa -text read EC key Private-Key: (521 bit) priv: [...] pub: [...] Field Type: prime-field Prime: 01:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff A: 01:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:fc B: 51:95:3e:b9:61:8e:1c:9a:1f:92:9a:21:a0:b6:85: 40:ee:a2:da:72:5b:99:b3:15:f3:b8:b4:89:91:8e: f1:09:e1:56:19:39:51:ec:7e:93:7b:16:52:c0:bd: 3b:b1:bf:07:35:73:df:88:3d:2c:34:f1:ef:45:1f: d4:6b:50:3f:00 Generator (uncompressed): 04:00:c6:85:8e:06:b7:04:04:e9:cd:9e:3e:cb:66: 23:95:b4:42:9c:64:81:39:05:3f:b5:21:f8:28:af: 60:6b:4d:3d:ba:a1:4b:5e:77:ef:e7:59:28:fe:1d: c1:27:a2:ff:a8:de:33:48:b3:c1:85:6a:42:9b:f9: 7e:7e:31:c2:e5:bd:66:01:18:39:29:6a:78:9a:3b: c0:04:5c:8a:5f:b4:2c:7d:1b:d9:98:f5:44:49:57: 9b:44:68:17:af:bd:17:27:3e:66:2c:97:ee:72:99: 5e:f4:26:40:c5:50:b9:01:3f:ad:07:61:35:3c:70: 86:a2:72:c2:40:88:be:94:76:9f:d1:66:50 Order: 01:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff:ff: ff:ff:ff:fa:51:86:87:83:bf:2f:96:6b:7f:cc:01: 48:f7:09:a5:d0:3b:b5:c9:b8:89:9c:47:ae:bb:6f: b7:1e:91:38:64:09 Cofactor: 1 (0x1) Seed: d0:9e:88:00:29:1c:b8:53:96:cc:67:17:39:32:84: aa:a0:da:64:ba OpenSSL should be able to convert among these two representations, but it does not seem to work here: $ openssl ec -in ecdsa -text -param_enc named_curve 140115271563072:error:10095020:elliptic curve routines:ECPKParameters_print:BIO lib:crypto/ec/eck_prn.c:228: 140115271563072:error:100DD010:elliptic curve routines:do_EC_KEY_print:EC lib:crypto/ec/ec_ameth.c:398: In OpenSSH, the issue exhibits itself in the sshkey_ecdsa_key_to_nid(), where we are trying to compare the groups from the file with the named groups and for some reason, they do not match (even though the text-compared groups match if I hand-compared them). I do not see any obvious error in there so I assume something changed with openssl that prevents this group comparison from working. I will try to investigate it later.
Hi Jakub, jsut FYI there is also similar issue reported by RedHat https://access.redhat.com/solutions/3175661 just resolution is kind of invalid. P.
I suppose you could recreate working key by generating a new key with the nist521p curve and replace public and private key values with the values from the original key. Not user friendly though. :)
One more RH nonsense recommendation: https://access.redhat.com/solutions/3382491 and I will try your solution ... I'm just not sure, if changing header/footer will make it acceptable .. but I will try. thanks P.
Tomas, as I expected, replacing just string did not help Error loading key "id_ecdsa2": invalid format P.
No, that's not what I suggested. You have to convert the key to DER format and use hexedit (or some similar binary editor) to find the placement of the public/private key bits in the new key and replace them with the bits from the old key. Then you can convert the key back to PEM format. I am sorry but I have no better/easier way. I suppose someone could write a tool to do proper conversion via loading the old key with openssl library, detecting which named curve is used and create a new key with the named curve and old key data, then saving the new key. It would be quite trivial but I unfortunately do not have time to do that.
Tomas, as I wrote in the original comment, the openssl tools _should_ support this type of conversion, but for some reason, it did not work for me with the key I generated. OpenSSH should again support both versions of keys, but again, for some reason the group comparison does not work with the key I generated (while it indeed worked before) so I assume something changed somewhere (OpenSSL regression?), but I did not have time and motivation to debug openssl structures further.
This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.