Bug 1632902

Unfortunately I cannot reproduce the issue. I have an ssh RSA key with Proc-Type: 4,ENCRYPTED; DEK-Info: AES-128-CBC that works with openssl-1.1.0i just fine. Can you please reproduce the issue with some testing key that you do not use so you could attach it here?

(In reply to Tomas Mraz from comment #1)
> Unfortunately I cannot reproduce the issue. I have an ssh RSA key with
> Proc-Type: 4,ENCRYPTED; DEK-Info: AES-128-CBC that works with openssl-1.1.0i
> just fine. Can you please reproduce the issue with some testing key that you
> do not use so you could attach it here?

.. strangely enough, now I cannot reproduce this either. It seems after downgrading and then upgrading to: 

openssl-1.1.0h-3                                                                       
openssl-devel-1.1.0h-3
openssl-libs-1.1.0h-3

no errors about the key's formatting were given. ... I'll make sure to reinstall packages when troubleshooting next time.

I'm closing the bug then. Please reopen if you're able to reproduce ideally with a testing key you can attach to bz.

I have same problem with my private key.

Attempt to add key into ssh agent with ssh-add ~/.ssh/id_rsa fails with error message
Error loading key "/home/alapshin/.ssh/id_rsa": invalid format

Current versions
openssl 1:1.1.0i-1.fc28
openssl-libs 1:1.1.0i-1.fc28
openssh 7.8p1-3.fc28

Downgrade to
openssh 7.8p1-2.fc28
openssh-clients 7.8p1-2.fc28
openssl 1:1.1.0h-3.fc28
openssl-libs 1:1.1.0h-3.fc28
openssl-libs 1:1.1.0h-3.fc28

Solves this problem.
Sadly I can't attach this key since it used in some places.
But key have same header as key in original bug report
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC

Additional information:
This key was created 4 years ago at minimum. File creation date indicates that it was created on 2014-12-05 but really it could be copied from other machine and created even earlier.

As a workaround

1. Downgrade openssl to 1:1.1.0h-3
2. Change key's password using ssh-keygen -p -f ~/.ssh/id_rsa (old password could be reused)
3. Upgrade openssl to 1:1.1.0i-1

After this steps I was able to successfully add key to ssh-agent

Does/did the password contain any special (non ASCII) characters?

No, password contains only ASCII characters.

*** Bug 1635044 has been marked as a duplicate of this bug. ***

I am reopening since the issue is real and probably not something obscure, however to be able to fix it I still need a key that is broken so I can debug the issue.

Created attachment 1489314 [details]
Key causing invalid format error

This is a key that is causing ssh-agent to fail with `invalid format` error.

Feel free to use as necessary (since it no longer used anywhere) however I still can't provide steps to create key from scratch to reproduce this case. I tried to create new key using Fedora 20 docker image but no luck.

My broken key also starts with:
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-128-CBC,

Thank you for the broken key example. This is really helpful.

I forgot, since the key is encrypted, I also need a correct passphrase for the key to be able to debug further. Hopefully it is something you can spare.

Tomas, I sent passphrase to your email at redhat.com. Is it ok or should I send it somewhere else?

It's OK.

This is really strange bug, because using openssl rsa or openssl pkey the key can be read fine. Only the ssh-keygen fails.

So after investigation I've found this is really openssh bug that is triggered by change of openssl behavior that is present also in openssl-1.1.1.

The problem is that openssh in sshkey_parse_private_pem_fileblob() calls PEM_read_bio_PrivateKey() without supplying passphrase callback. Thus the default one is used which provides 0 length passphrase. Previously in openssl-1.1.0h the 0 length passphrase was regarded as error but now it is not and in some cases the decryption passes correctly (i.e. the padding is accidentally decrypted OK). In such case the decrypted garbage is now parsed as private key structure which is of course corrupted and so the parser reports different error.

The openssh should call PEM_read_bio_PrivateKey() in this case with non-default callback which would always return -1. So the error returned in case of encrypted PEM files will be consistent and the further processing of the file can continue.

This also explains why openssl utilites work fine with the "invalid" key file.

You can for example use command:

openssl pkey -in invalid_key -aes128 -traditional -out new_key

to re-encrypt the file which will most probably make the padding different and so the issue is not triggered with it anymore.

It looks like my 'workaround' for downgrading then updating the openssl package or simply just reinstalling it didn't last.

It's possible that it didn't survive a reboot somehow. I'm getting the same invalid format error message. I will play around some more and provide more information later.

@jorp - A workaround that worked for me was to regenerate the key after downgrading openssl, using "ssh-keygen -p" with the same password as before. The updated key is likely to work also with the newer openssl. To test the key after upgrading openssl you have to unset SSH_ASKPASS and SSH_AUTH_SOCK in the environment if you are running Gnome, otherwise ssh-agent will keep the cached key in memory. I guess that is why your old key failed first after you rebooted the machine.

You can use the instructions from comment #17 as a workaround although you can also change the format of the private key file to something more modern with:

openssl pkey -in invalid_key -aes128 -out new_key

This will change the format to PKCS#8 encrypted private key file.

This does not require downgrade of the openssl and the PKCS#8 format includes HMAC of the key so the openssh issue cannot manifest with it.

This was fixed in upstream OpenSSH and will be in the next release.

openssh-7.9p1-1.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5be740cab4

openssh-7.9p1-1.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5be740cab4

openssh-7.9p1-1.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.

*** Bug 1636706 has been marked as a duplicate of this bug. ***