Description of problem: we have an inhouse pam module, like pam-ssh-agent-auth except that it works with ssh certs. this works fine with stock openssh-7.4p1, but with redhat's "7.4p1 release 1.fc25", it fails. if you run ssh-agent -d -a /tmp/ssh.sock and use that auth sock, the agent prints the following. debug1: type 11 debug1: type 13 process_sign_request2: RSA-CERT key not found debug1: XXX shrink: 5 < 6 type 11 is the request identities and 13 is a signing request. Version-Release number of selected component (if applicable): 7.4p1 release 1.fc25 How reproducible: I've got some go code that will give a 'minimal' reproducer but I need to get approval from the muckimucks here b/c it's pulled straight from this internal pam module. the basic concept is that, on a remote machine with a forwarded ssh-agent, the pam module connects to SSH_AUTH_SOCK, lists the keys on the agent looking for one of our ssh certificates and when it finds one, it asks the agent to sign some random data with the associated private key and then verifies the signature. that signing operation is what's failing. I'm going to try and get permission to post this repro, but hopefully there's enough info there to point at the errant fedora patch that's causing this. as I said, this works fine in stock 7.4p1 Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1244891 [details] go repro build and run like so: $ go get $ go build -o sign sign.go $ ./sign you should see $ ./sign error signing data: agent: failed to sign challenge false if you're running the ssh-agent in the foreground and you've set your SSH_AUTH_SOCK to use that agent, you'll see the "process_sign_request2: RSA-CERT key not found" message in that window.
Hello. Thank you for the report. It will be most probably again related to the openssl 1.1.0 patch, which is not upstream. I hope I will have a look into it during this week.
The problem is again ssh-agent protocol [1], which does not contain N and E parameters of public key, but new OpenSSL requires it. I already added code to compute N, but computing E is not so simple. I don't have definite fix, except for rebuilding with older openssl. [1] https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.agent#L245
Jakub, have you tried to set zero bignum for E? It's a hack but it should work if the values are not used in the computation. (It should not be used unless RSA_FLAG_EXT_PKEY is set.) Unfortunately the N value is used to obtain the size of the signature, so you have to compute it. I wonder what the code did with old openssl because the N value is clearly used in the source for the private key signature operation - I suppose that is what openssh later does with the private key. Or what are the operations done with the private key?
how does the agent sign requests that come from a remote sshd? in our environment, these certs are used to auth to sshd as well as to pam. and sshd definitely asks the agent to sign some random data with the private key associated with this cert. eg. this is me using a cert to auth to one of our prod hosts. debug3: fd 4 is O_NONBLOCK debug1: type 11 # request identities debug1: XXX shrink: 3 < 4 debug3: fd 4 is O_NONBLOCK debug1: type 11 # request identities debug1: type 13 # sign request debug3: fd 5 is O_NONBLOCK debug1: type 11 # request identities debug1: type 13 # sign request debug3: fd 6 is O_NONBLOCK debug1: XXX shrink: 5 < 6 # now I'm authenticated. also, if this is just about computing N and E (sorry, my openssl library knowledge pretty much begins and ends with how it's spelled), does that mean that it shouldn't affect other key types, like ed25519 or ecdsa keys? I haven't tried those yet, but I can if it would help.
The other key types should work fine. They do not have so weird interface in the agent (I hope). The keys are now set to zero, but from there comes this bug during the attempt to use the certificate for the signature. With ssh-signature request, the agent gets keyblob and fails when it searches for matching key already stored in the agent (comparing directly N and E parts, which obviously fails if one of them is zero). Removing the whole patch in Fedora 25 solves the problem. I didn't find the minimal subset which is causing the problem yet. In F25 we don't have OpenSSL 1.1.0 so the change is most probably not related to openssl 1.1.0 itself, but only to the new interface or some broken logic in the patch. I will have a further look tomorrow.
How could it match N and E before when it was not set at all?
After further inspection, the N and E are already set in the rsa structure from the certificate blob (which I somehow missed during the original implementation and which was already missed by the original author of the patch). Skipping the assignment of these values makes it working again. I will send an update soon. So far testing scratch build for F25: https://koji.fedoraproject.org/koji/taskinfo?taskID=17563323 Thanks for the good reproducer and patience :)
openssh-7.4p1-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-40057be2c5
Thanks, Jackub. is there a package that my fedora user(s) can test?
Yes, there is the scratch build in the comment #8 or the update for Fedora 25 in comment #9 (should go into the testing soon). So far you can pick that up from koji: https://koji.fedoraproject.org/koji/buildinfo?buildID=838740
openssh-7.4p1-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-40057be2c5
I am the original user affected by this bug (I work with Peter, and run Fedora). I can confirm that the packages you linked to in #11 fix this bug for me. Thanks so much for your help -- you already fixed bug 1402029 for us, and I appreciate the hard work to fix this issue as well.
openssh-7.4p1-2.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.