Bug 1687230

Summary: When using ssh-agent with ecdsa-sha2-nistp384 key on a security key (Yubikey 5 NC in this case) ssh-agent fails to sign and then seg faults.
Product: [Fedora] Fedora Reporter: Nick P <npope>
Component: opensshAssignee: Jakub Jelen <jjelen>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: dwalsh, jfch, jjelen, lkundrak, mattias.ellert, npope, plautrba, tmraz, victor.andreasson
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openssh-8.0p1-1.fc30 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-04 00:16:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Nick P 2019-03-11 01:46:15 UTC
Description of Problem:
When using ssh-agent with ecdsa-sha2-nistp384 key on a security key (Yubikey 5 NC in this case) ssh-agent fails to sign and then seg faults.  FWIW RSA keys on a security key work with ssh-agent.

How reproducible:
Consistently

Steps to reproduce:
# Create ECDSA key
$ yubico-piv-tool -s 9a -AECCP384 -a generate -o public.pem --key
Enter management key: 
Successfully generated a new private key.
$ yubico-piv-tool -a verify-pin -a selfsign-certificate -s 9a -S "/CN=SSH key/" -i public.pem -o cert.pem
Enter PIN: 
Successfully verified PIN.
Successfully generated a new self signed certificate.
$ yubico-piv-tool -a import-certificate -s 9a -i cert.pem --key
Enter management key: 
Successfully imported a new certificate.
$ ssh-keygen -D /usr/lib64/opensc-pkcs11.so -e
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIYBDDS9N+FlOyRviNzcg6RNGFU37MdSfgWFOWOvS/ge9BPl976rU27gvo/Vd+sIp7VHZYrVL8vidQ1pjb8DMr0lbSRnSFk6YPw4MPqEdPl0QLwzCBsR6niM2i4D2cm5EA==
# Copy the public key to localhost ~/.ssh/authorized_keys file
$ echo 'ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIYBDDS9N+FlOyRviNzcg6RNGFU37MdSfgWFOWOvS/ge9BPl976rU27gvo/Vd+sIp7VHZYrVL8vidQ1pjb8DMr0lbSRnSFk6YPw4MPqEdPl0QLwzCBsR6niM2i4D2cm5EA==' > ~/.ssh/authorized_keys
# Cleanup any currently running ssh-agent and start a ssh-agent in debug mode
$ pkill -9 ssh-agent; rm -rf /tmp/ssh-*; ssh-agent -D
SSH_AUTH_SOCK=/tmp/ssh-gbmJQM6y7tq8/agent.9141; export SSH_AUTH_SOCK;
echo Agent pid 9141;
# In another terminal add the SSH_AUTH_SOCK environment variable
$ SSH_AUTH_SOCK=/tmp/ssh-gbmJQM6y7tq8/agent.9141; export SSH_AUTH_SOCK;
$ ssh-add -s /usr/lib64/opensc-pkcs11.so
Enter passphrase for PKCS#11: 
Card added: /usr/lib64/opensc-pkcs11.so
$ ssh-add -L
ecdsa-sha2-nistp384 AAAAE2VjZHNhLXNoYTItbmlzdHAzODQAAAAIbmlzdHAzODQAAABhBIYBDDS9N+FlOyRviNzcg6RNGFU37MdSfgWFOWOvS/ge9BPl976rU27gvo/Vd+sIp7VHZYrVL8vidQ1pjb8DMr0lbSRnSFk6YPw4MPqEdPl0QLwzCBsR6niM2i4D2cm5EA== /usr/lib64/opensc-pkcs11.so
$ ssh localhost
$ ssh localhost

Actual results:
$ ssh localhost
sign_and_send_pubkey: signing failed: agent refused operation
user@localhost: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
$ ssh localhost

Now notice in the ssh-agent debug output it failed to sign, and when we ran ssh localhost again ssh-agent seg faulted
echo Agent pid 9141;
process_sign_request2: sshkey_sign: error in libcrypto
Segmentation fault

Expected results:
Successfully authenticated against ssh server

Additional info:
$ cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)
$ rpm -qa | grep -E 'ssh|opensc|yubico-piv'
NetworkManager-ssh-1.2.7-5.fc29.x86_64
opensc-0.19.0-3.fc29.x86_64
libssh2-1.8.0-8.fc29.x86_64
libssh-0.8.7-1.fc29.x86_64
qemu-block-ssh-3.0.0-3.fc29.x86_64
openssh-7.9p1-4.fc29.x86_64
openssh-server-7.9p1-4.fc29.x86_64
sshpass-1.06-6.fc29.x86_64
yubico-piv-tool-1.6.2-1.fc29.x86_64
NetworkManager-ssh-gnome-1.2.7-5.fc29.x86_64
openssh-clients-7.9p1-4.fc29.x86_64

Comment 1 Jakub Jelen 2019-03-11 09:57:22 UTC
It looks like I can reproduce the issue with my yubikey 4. The ssh-agent support is still a bit clunky (bug #1609055) and the upstream rewrote [1] the ECDSA support from scratch recently so it should land in the next release in coming weeks or few months. Given these constraints, I do not think it makes sense to fix it in the current code, but I will make sure it will work with the next upstream release. In the meantime, you should be able to use either RSA keys or ECDSA keys not through the agent.

[1] https://bugzilla.mindrot.org/show_bug.cgi?id=2474

Comment 2 Nick P 2019-03-11 12:47:13 UTC
Thanks for the quick confirmation.  I'll keep my eye on this bugzilla and retest when there is a relevant update in the Fedora repos.

Comment 3 Jakub Jelen 2019-04-29 12:38:01 UTC
This should be addressed by the rebase that I will be pushing to testing soon. See the bug #1701072

Comment 4 Fedora Update System 2019-04-29 13:08:21 UTC
openssh-8.0p1-1.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-0f4190cdb0

Comment 5 Fedora Update System 2019-04-30 02:49:07 UTC
openssh-8.0p1-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-0f4190cdb0

Comment 6 Fedora Update System 2019-05-04 00:16:33 UTC
openssh-8.0p1-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.

Comment 7 Nick P 2019-05-14 11:32:33 UTC
I've retested this on Fedora 30 and can confirm it works.

$ cat /etc/fedora-release 
Fedora release 30 (Thirty)
$ rpm -qa | grep -E 'ssh|opensc|yubico|piv'
sshpass-1.06-7.fc30.x86_64
NetworkManager-ssh-1.2.9-1.fc30.x86_64
opensc-0.19.0-6.fc30.x86_64
openssh-8.0p1-1.fc30.x86_64
libssh2-1.8.2-1.fc30.x86_64
libssh-0.8.7-1.fc30.i686
yubico-piv-tool-1.7.0-1.fc30.x86_64
openssh-clients-8.0p1-1.fc30.x86_64
libssh-0.8.7-1.fc30.x86_64
fuse-sshfs-3.5.1-1.fc30.x86_64
openssh-server-8.0p1-1.fc30.x86_64
qemu-block-ssh-3.1.0-7.fc30.x86_64
NetworkManager-ssh-gnome-1.2.9-1.fc30.x86_64

Thank you.