Bug 1879183

Summary: ssh (the client) fails to use signed public key passed via `ssh -i`
Product: Red Hat Enterprise Linux 8 Reporter: Christopher J. Ruwe <cjr>
Component: opensshAssignee: Jakub Jelen <jjelen>
Status: CLOSED NOTABUG QA Contact: BaseOS QE Security Team <qe-baseos-security>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.0CC: cjr, tmraz
Target Milestone: rc   
Target Release: 8.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-16 08:42:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Christopher J. Ruwe 2020-09-15 15:53:22 UTC
Description of problem:

The openssh client does not accept a public key when the public key is an (arbitrary) signed certificate.

Instead, the client complains about "invalid format".


Version-Release number of selected component (if applicable):

8.0p1-4.el8_1

How reproducible:

The original context was that an external instance (Hashicorp Vault) should sign arbitrary public ssh keys via http POST. Authorization is deferred to an OIDC IdP (keycloak). 

I reproduced the issue without.

Steps to Reproduce:

1) Generate an arbitrary SSH key pair, for instance calling `ssh-keygen -t rsa  -b 4096 -f somekey`. 
2) Generate a CA to sing that key `ssh-keygen -f server_ca`.
3) Sign the key `ssh-keygen -s server_ca -I 9000 -V +1d -n hal9000 /home/hal9000/.ssh/somekey.pub`.
4) Move `somekey-cert.pub` to for instance `$PWD/somekey` and delete `home/hal9000/.ssh/somekey*`.


Calling  `ssh -vi somekey localhost` (where to does not matter) fails

Actual results:

```
OpenSSH_8.0p1, OpenSSL 1.1.1c FIPS  28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug1: identity file somekey type 4
debug1: identity file somekey-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to localhost:22 as 'hal9000'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:dxo/wyn6LAgnaNjX77Qb5BN41Su7yQdHg3M6F6mhtME
debug1: Host 'localhost' is known and matches the ECDSA host key.
debug1: Found key in /home/hal9000/.ssh/known_hosts:1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: somekey RSA-CERT SHA256:/5VW1ftZJUQrgX6xL71e0XfuKgGFB2ua5LPpYJqwbJo explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: KCM:)


debug1: Next authentication method: publickey
debug1: Offering public key: somekey RSA-CERT SHA256:/5VW1ftZJUQrgX6xL71e0XfuKgGFB2ua5LPpYJqwbJo explicit
debug1: Server accepts key: somekey RSA-CERT SHA256:/5VW1ftZJUQrgX6xL71e0XfuKgGFB2ua5LPpYJqwbJo explicit
debug1: sign_and_send_pubkey: no separate private key for certificate "somekey"
Load key "somekey": invalid format
debug1: Next authentication method: password

```

I got it working only when all key file were present and named $HOME/.ssh/id_rsa, $HOME/.ssh/id_rsa.pub $HOME/.ssh/id_rsa-cert.pub. I did not try other permutations.


Expected results:

It should just work. The ssh-client should accept 1) any signed public ssh key passwd with `ssh -i`, 2) it should not care about any naming of files and 3) it should neither demand the corresponding private nor the unsigned public key to be present.

I did not examine the case when more than one SSH key pair is present on a given system (for different security realms for instance). I would expect that to be an issue as well.

Thanks for your consideration.

Comment 1 Jakub Jelen 2020-09-16 08:42:59 UTC
> 3) it should neither demand the corresponding private nor the unsigned public key to be present.

How the client should then prove to the server it has the private key when it does not have it? The certificate authentication does not replace public key authentication. The certificate file is just public key with some metadata (the same as X509). It is "certification" addition to the public key send on the wire. The private key is always needed also for certificate authentication, otherwise there would be no security at all. Please, check the specification:

https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.certkeys

Comment 2 Tomas Mraz 2020-09-16 11:45:58 UTC
To make it clear - the unsigned public key should not be needed. But the private key is absolutely required. Anybody holding the public key would be able to impersonate you otherwise.

Comment 3 Christopher J. Ruwe 2020-09-16 13:31:39 UTC
Thank you for both your answers.

What you are telling me then is that _both_ private and signed public keys are _required_? What you are also telling me is that signed keys is not a mechanism different from the "normal" public/private key scheme, but an _extension_ building on the former?

That I can actually verify when calling `ssh` with both keys passed as in `ssh -i <private-key> -i <public-and-signed-key> ...`. When only the public-and-signed-key is passed, the client looks for a
corresponding private key in the standard locations and fails when none or no matching can be found there, correct?

If I have understood correctly so far, it might actually be nice to drop a line in the docs. ssh-keygen(1) elaborates on signed keys in that

> Clients or servers may then trust _only_ the CA key and verify its
> signature on a certificate rather _than trusting many user/host
> keys_. (Emphasis mine)

That could - as I have - be misunderstood, which I will explain later.

> To make it clear - the unsigned public key should not be needed. But
> the private key is absolutely required. Anybody holding the public
> key would be able to impersonate you otherwise.

Thank you for clarifying that reasoning, I only now understand my error. With the trust relationship worded as in the the ssh-keygen(1) man page ("only"), I imagined "sealed box signing
mechanisms", where _noone_ has access to the private key of the CA certificate and thus only the sealed box could impersonate, which it won't.

Giving _actual persons_ access to the CA keys completely escaped my notice, even as it is, when I now think about it, probably the default case. 

Could you please consider dropping a line in the documentation elaborating that 1) "key signing is not separate, but only extends", because 2) otherwise anyone with access to the CA keys, which per
default perspective is a human operator, could impersonate? Perhaps it would be helpful to describe how to pass private and signed public keys with `ssh -i ... -i ...` when having keys with or non-default names or locations?

In any case, the last explanation has been extremely helpful, thank you very much.

Comment 4 Jakub Jelen 2020-09-16 15:59:47 UTC
(In reply to Christopher J. Ruwe from comment #3)
> Thank you for both your answers.
> 
> What you are telling me then is that _both_ private and signed public keys
> are _required_?

Certainly the private key is needed. Public key should not be needed as it can
be trivially extracted from the certificate and from some types of private keys
(though it might not be implemented in openssh-8.0p1). But it would need to be
tested.

> What you are also telling me is that signed keys is not a
> mechanism different from the "normal" public/private key scheme, but an
> _extension_ building on the former?

Right, as described in the previously mentioned PROTOCOL.certkeys.

> That I can actually verify when calling `ssh` with both keys passed as in
> `ssh -i <private-key> -i <public-and-signed-key> ...`. When only the
> public-and-signed-key is passed, the client looks for a
> corresponding private key in the standard locations and fails when none or
> no matching can be found there, correct?

The certificate, if in non-standard location or non-standard naming from the private
key specified by -i, needs to be specified with -oCertificateFile= from manual pages.
It might work also with -i, but might go into weird issues too. The manual page for
ssh describes how the certificates for keys are searched (not the other way round),
so if it has correct suffix, it might work to pass only the private key. But I did
not try that.

> If I have understood correctly so far, it might actually be nice to drop a
> line in the docs. ssh-keygen(1) elaborates on signed keys in that
> 
> > Clients or servers may then trust _only_ the CA key and verify its
> > signature on a certificate rather _than trusting many user/host
> > keys_. (Emphasis mine)

This is still correct. You trust the CA by specifying it in TrustedUserCAKeys or
in authorized_keys as @cert-authority. This CA signs the certificates, which contain
the public keys of the owners, but they still need to use the private key to provide
the signature, which you verify with public key from that signed certificate.
If we would not have the step of signing, it authentication would be replayable and not
secure at all.

The trust is what you write in your configuration files, transitively you trust all
the keys that were signed by the CA (indeed, can be limited by principals).

> That could - as I have - be misunderstood, which I will explain later.
> 
> > To make it clear - the unsigned public key should not be needed. But
> > the private key is absolutely required. Anybody holding the public
> > key would be able to impersonate you otherwise.
> 
> Thank you for clarifying that reasoning, I only now understand my error.
> With the trust relationship worded as in the the ssh-keygen(1) man page
> ("only"), I imagined "sealed box signing
> mechanisms", where _noone_ has access to the private key of the CA
> certificate and thus only the sealed box could impersonate, which it won't.

I think this does not contradict. It is up to the implementation how the CA is
protected (either by sealing it into offline box, HSM, or just keeping it as online
service behind some other authentication method).

> Giving _actual persons_ access to the CA keys completely escaped my notice,
> even as it is, when I now think about it, probably the default case. 
> 
> Could you please consider dropping a line in the documentation elaborating
> that 1) "key signing is not separate, but only extends", because 2)
> otherwise anyone with access to the CA keys, which per
> default perspective is a human operator, could impersonate?

If anyone gets access to any CA keys to use them, you are screwed. That is no
different for any other CA keys in any other cryptosystem. From my POV, this
is already in the definition of CA. But feel free to propose better wording
if the current is confusing for you. Ideally in the openssh bugzilla:

https://bugzilla.mindrot.org/

> Perhaps it would
> be helpful to describe how to pass private and signed public keys with `ssh
> -i ... -i ...` when having keys with or non-default names or locations?

I think this is specified in manual page for ssh, in the description of the -i switch.

> In any case, the last explanation has been extremely helpful, thank you very
> much.

You are welcome.