It looks like a consequence of disabling SHA1 in openssl by default. SHA1 is insecure nowadays and we are working on eliminating it. To temporary enable SHA1, you may use LEGACY crypto policy. If it is a self-test, it should be rewritten to use SHA256.

Created attachment 1864597 [details]
patch to test theory

I believe the same problem currently causes libtpms to fail to build from source with a test failure.

I suspect this is because calling EVP_PKEY_CTX_set_rsa_padding(ctx, RSA_PKCS1_PSS_PADDING) sets a default signature digest algorithm of SHA1 inside of OpenSSL, which fails due to bug 2031742. This does not happen if a non-SHA1 digest algorithm was set before using EVP_PKEY_CTX_set_signature_md.

I am attaching a patch that changes the order of calls to EVP_PKEY_CTX_set_rsa_padding() and EVP_PKEY_CTX_set_signature_md() in libtmps, which fixes the build test failure for me. Can you confirm this also fixes the runtime issue?

I believe this is something we should attempt to fix in OpenSSL. I'll look into where and how this could be done.

scratch-build with the patch,
https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=43639934

QA, can you check the avocado test passes with it? thanks

Here's a c9s scratch build with a change in OpenSSL that should also fix that, I'd appreciate if you could also test that with an unmodified libtpms: https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=1030176

(In reply to Dmitry Belyavskiy from comment #1)
> It looks like a consequence of disabling SHA1 in openssl by default. SHA1 is
> insecure nowadays and we are working on eliminating it. To temporary enable
> SHA1, you may use LEGACY crypto policy. If it is a self-test, it should be
> rewritten to use SHA256.

sha256 is used by default now by swtpm_setup:

(1)/etc/swtpm_setup.conf 
# Program invoked for creating certificates
create_certs_tool= /usr/bin/swtpm_localca
create_certs_tool_config = /etc/swtpm-localca.conf
create_certs_tool_options = /etc/swtpm-localca.options
# Comma-separated list (no spaces) of PCR banks to activate by default
active_pcr_banks = sha256

(2)libvirtd log:
2022-03-07 08:24:29.403+0000: 19252: debug : virCommandRunAsync:2630 : About to run /usr/bin/swtpm_setup --tpm2 --tpm-state /var/lib/libvirt/swtpm/0f9ea348-d4a1-400e-a641-75d8247f4ee3/tpm2 --vmid avocado-vt-vm1:0f9ea348-d4a1-400e-a641-75d8247f4ee3 --logfile /var/log/swtpm/libvirt/qemu/avocado-vt-vm1-swtpm.log --createek --create-ek-cert --create-platform-cert --lock-nvram --not-overwrite

(3)swtpm log
...Successfully created EK certificate locally.
Successfully created NVRAM area 0x1c00016 for ECC EK certificate.
Successfully activated PCR banks sha256 among sha1,sha256,sha384,sha512.
Successfully authored TPM state.
Ending vTPM manufacturing @ Mon 07 Mar 2022 03:24:29 AM EST
libtpms/tpm2: Entering failure mode; code: 6, location: TestRsaSignAndVerify line 534
libtpms/tpm2: TPM2_Process: Entered failure mode through command:
80 01 00 00 00 0b 00 00 01 43 00

(In reply to Clemens Lang from comment #8)
> Here's a c9s scratch build with a change in OpenSSL that should also fix
> that, I'd appreciate if you could also test that with an unmodified libtpms:
> https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=1030176

Yes, it's passed with this scratch build.

(In reply to Marc-Andre Lureau from comment #7)
> scratch-build with the patch,
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=43639934
> 
> QA, can you check the avocado test passes with it? thanks

Sorry, I didn't test this scratch build because it was buid with RHEL 8.6.0 but not RHEL 9.0. In RHEL 8.6.0 we don't have this issue now. But the test result is passed with the openssl scratch build in comment 8.

Sorry for deleting all needinfo by mistake, and here restore it.

(In reply to Meina Li from comment #13)
> Sorry, I didn't test this scratch build because it was buid with RHEL 8.6.0
> but not RHEL 9.0. In RHEL 8.6.0 we don't have this issue now. But the test
> result is passed with the openssl scratch build in comment 8.
 
(my bad), since the regression came from openssl, and there is a fix for openssl, should we move the bug back to it? Clemens?

(In reply to Marc-Andre Lureau from comment #15)

> since the regression came from openssl, and there is a fix for openssl, should we move the bug back to it? Clemens?

Yes. I would appreciate if you could help fill the blocker information from comment 11, though, since I believe you have a stronger rationale than "we're fixing a regression in OpenSSL that might affect some packages".

(In reply to Clemens Lang from comment #16)
> (In reply to Marc-Andre Lureau from comment #15)
> 
> > since the regression came from openssl, and there is a fix for openssl, should we move the bug back to it? Clemens?
> 
> Yes. I would appreciate if you could help fill the blocker information from
> comment 11, though, since I believe you have a stronger rationale than
> "we're fixing a regression in OpenSSL that might affect some packages".

Actually, I don't :) But I managed to get a backtrace of the failing path:

#0  rsa_setup_md (ctx=ctx@entry=0x4aa6e0, mdname=0x7ffff7d78df5 "SHA1", mdprops=<optimized out>) at providers/implementations/signature/rsa_sig.c:310
#1  0x00007ffff7cbd24c in rsa_set_ctx_params (vprsactx=0x4aa6e0, params=<optimized out>) at providers/implementations/signature/rsa_sig.c:1364
#2  0x00007ffff7bafc60 in evp_pkey_ctx_ctrl_to_param (pctx=0x4a5550, keytype=<optimized out>, optype=<optimized out>, cmd=4097, p1=6, p2=<optimized out>)
    at crypto/evp/ctrl_params_translate.c:2567
#3  0x00007ffff7f3c5fd in CryptRsaSign (sigOut=0x7fffffffcf10, key=0x7fffffffd0a0, hIn=0x7fffffffcec0, rand=0x0) at tpm2/crypto/openssl/CryptRsa.c:1538

#0  rsa_setup_md (ctx=ctx@entry=0x4aa6e0, mdname=0x7ffff7d78df5 "SHA1", mdprops=<optimized out>) at providers/implementations/signature/rsa_sig.c:310
310	                ERR_raise_data(ERR_LIB_PROV, PROV_R_DIGEST_NOT_ALLOWED,

Since EVP_PKEY_CTX_set_rsa_padding(ctx, padding) is called before EVP_PKEY_CTX_set_signature_md(ctx, md=sha512_md), it bails out as the context is still the default sha1.

Imho, openssl shouldn't fail at this initialization time, but rather when sha1 is actually used.

Thanks. I'm aware of the path that causes this to fail, but "a particular order of calls into OpenSSL cause an error that wasn't here before" doesn't really provide an answer what the impact on RHEL 9.0 would be if it isn't fixed.
Am I correct in assuming that anything that uses libtpms would fail? So basically every component that uses libtpms would no longer work?

Do you have a list of such components in RHEL?

(In reply to Clemens Lang from comment #18)
> Thanks. I'm aware of the path that causes this to fail, but "a particular
> order of calls into OpenSSL cause an error that wasn't here before" doesn't
> really provide an answer what the impact on RHEL 9.0 would be if it isn't
> fixed.
> Am I correct in assuming that anything that uses libtpms would fail? So
> basically every component that uses libtpms would no longer work?
> 
> Do you have a list of such components in RHEL?

A VM (QEMU/libvirt) with a vTPM would fail to run a self-test, which is a fairly common operation done during TPM initialization afaik. Practically, I think a VM with a vTPM would be unusable. And there might be other vTPM usages and code path impacted.

(In reply to Dmitry Belyavskiy from comment #1)
> It looks like a consequence of disabling SHA1 in openssl by default. SHA1 is
> insecure nowadays and we are working on eliminating it. To temporary enable
> SHA1, you may use LEGACY crypto policy. If it is a self-test, it should be
> rewritten to use SHA256.

What is 'LEGACY' now in OpenSSL 3.0 and how do I enable legacy algorithms? Or is sha1 not even compiled? The page https://www.openssl.org/docs/OpenSSL300Design.html  mentions the following algorithms as being legacy:


"The methods for the following algorithms will become dispatch tables in the Legacy provider module:

    Blowfish
    CAST
    DES (but not 3DES)
    DSA
    IDEA
    MD2
    MD4
    MDC2
    RC2
    RC4
    RC5
    RIPEMD160
    SEED
    Whirlpool"

SHA1 being quite prominent is not mentioned.

libtpms CANNOT easily get rid of sha1 without loosing the upgrade path.

Also reflected in restoring from a file or migration:
1.# virsh managedsave vm-yqz 
Domain 'vm-yqz' state saved by libvirt

# virsh start vm-yqz 
error: Failed to start domain 'vm-yqz'
error: internal error: qemu unexpectedly closed the monitor: 2022-03-10T04:11:52.887572Z qemu-kvm: tpm-emulator: TPM result for CMD_INIT: 0x101 operation failed
2022-03-10T04:11:52.887591Z qemu-kvm: error while loading state for instance 0x0 of device 'tpm-emulator'
2022-03-10T04:11:52.887967Z qemu-kvm: load of migration failed: Input/output error

2.# virsh -c 'qemu:///system' migrate --live --p2p --verbose --domain vm-yqz --desturi qemu+ssh://hostb/system
Migration: [100 %]error: operation failed: domain is no longer running
#cat qemu.log:
2022-03-10T03:43:48.536114Z qemu-kvm: tpm-emulator: TPM result for CMD_INIT: 0x101 operation failed
2022-03-10T03:43:48.536138Z qemu-kvm: error while loading state for instance 0x0 of device 'tpm-emulator'
2022-03-10T03:43:48.537479Z qemu-kvm: load of migration failed: Input/output error
2022-03-10 03:43:48.741+0000: shutting down, reason=crashed

After downgrade openssl on both src and target hosts, issue disappeared.

(In reply to Stefan Berger from comment #21)
> (In reply to Dmitry Belyavskiy from comment #1)
> > It looks like a consequence of disabling SHA1 in openssl by default. SHA1 is
> > insecure nowadays and we are working on eliminating it. To temporary enable
> > SHA1, you may use LEGACY crypto policy. If it is a self-test, it should be
> > rewritten to use SHA256.
> 
> What is 'LEGACY' now in OpenSSL 3.0 and how do I enable legacy algorithms?
> Or is sha1 not even compiled?

There are 2 different things:

1. Red Hat specific crypto policies I talk about represent the system-wide crypto profile. 
2. OpenSSL 3.0 put the algorithms they provide in different providers. Legacy provider provides definitely old algorithms. SHA1 is provided in the default (and even FIPS) provider but we don't want to use it for signatures.

(In reply to Dmitry Belyavskiy from comment #23)
> > What is 'LEGACY' now in OpenSSL 3.0 and how do I enable legacy algorithms?
> > Or is sha1 not even compiled?
> 
> There are 2 different things:
> 
> 1. Red Hat specific crypto policies I talk about represent the system-wide
> crypto profile. 
> 2. OpenSSL 3.0 put the algorithms they provide in different providers.
> Legacy provider provides definitely old algorithms. SHA1 is provided in the
> default (and even FIPS) provider but we don't want to use it for signatures.

- how is a program policy set? Is swtpm using the legacy policy? (and thus can still use sha1 for signing)

- unless the code explicitly set the provider, the default is being used, right?

(In reply to Marc-Andre Lureau from comment #26)
> - how is a program policy set? Is swtpm using the legacy policy? (and thus
> can still use sha1 for signing)

There is no program-specific policy. The crypto-policies package provides system-wide defaults that apply to all programs on startup. Binaries may choose to modify some of the settings locally, e.g. by adding TLS cipher suites to the default policy.

As a consequence, whether or not swtpm can use SHA-1 depends on whether the system crypto policy is "DEFAULT" or "LEGACY" (update-crypto-policies --set LEGACY). This is a setting to be modified by the system administrator, and swtpm should respect it when deciding whether to support SHA-1 signatures (as it will by default if it does nothing, because OpenSSL will read the default crypto policy setting and respect it).

> - unless the code explicitly set the provider, the default is being used,
> right?

Correct.

 
> As a consequence, whether or not swtpm can use SHA-1 depends on whether the
> system crypto policy is "DEFAULT" or "LEGACY" (update-crypto-policies --set
> LEGACY). This is a setting to be modified by the system administrator, and
> swtpm should respect it when deciding whether to support SHA-1 signatures
> (as it will by default if it does nothing, because OpenSSL will read the
> default crypto policy setting and respect it).
> 

For better or worse libtpms has SHA1 enable via compile time options and needs to carry it forward to provide an upgrade path for those VMs that have been using SHA1 and may be verifying/creating signatures with SHA1. I do not know whether there are any or how many there are.

Is openssl preventing signature creation with SHA1 or also signature verification involving SHA1?

With openssl-3.0.1-14.el9.x86_64, we also can't ssh the RHEL6 guest because of the error in libcrypto. But it can success with openssl-3.0.1-5.el9.x86_64. So I think it may also be caused by this bug.

# virsh edit guest
...
    <disk type='file' device='disk' model='virtio-transitional'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
...
    <interface type='network'>
      <mac address='52:54:00:c4:3b:14'/>
      <source network='default'/>
      <model type='virtio-transitional'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
...
# virsh start guest
Domain 'guest' started
# ssh -vvv 192.168.122.72
OpenSSH_8.7p1, OpenSSL 3.0.1 14 Dec 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 55: Including file /etc/ssh/ssh_config.d/50-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/50-redhat.conf
debug2: checking match for 'final all' host 192.168.122.72 originally 192.168.122.72
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 3: not matched 'final'
debug2: match not found
debug3: /etc/ssh/ssh_config.d/50-redhat.conf line 5: Including file /etc/crypto-policies/back-ends/openssh.config depth 1 (parse only)
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-curve25519-sha256-,gss-nistp256-sha256-,gss-group14-sha256-,gss-group16-sha512-]
debug3: kex names ok: [curve25519-sha256,curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512]
debug1: configuration requests final Match pass
debug2: resolve_canonicalize: hostname 192.168.122.72 is address
debug1: re-parsing configuration
......
debug3: load_hostkeys_file: loaded 1 keys from 192.168.122.72
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.168.122.72' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: bits set: 4119/8192
ssh_dispatch_run_fatal: Connection to 192.168.122.72 port 22: error in libcrypto

Dear Meina,

It is caused by the same change, but I think it's not a bug at all. SHA1 is insecure and you need LEGACY crypto-policy to use it.
If you provide a ecdsa key inside the RHEL6 machine, you should still be able to login there because ecdsa keys use SHA2 hashes that are secure.

(In reply to Stefan Berger from comment #28)
> For better or worse libtpms has SHA1 enable via compile time options and
> needs to carry it forward to provide an upgrade path for those VMs that have
> been using SHA1 and may be verifying/creating signatures with SHA1. I do not
> know whether there are any or how many there are.

And that is still supported if the crypto policy is set to LEGACY.


> Is openssl preventing signature creation with SHA1 or also signature
> verification involving SHA1?

Both signing and verification are disabled, unless the system was set to the legacy crypto policy. If you need SHA1 signatures, that is what you should do until you have had the chance to transition off SHA1.

(In reply to Clemens Lang from comment #31)
> (In reply to Stefan Berger from comment #28)
> > For better or worse libtpms has SHA1 enable via compile time options and
> > needs to carry it forward to provide an upgrade path for those VMs that have
> > been using SHA1 and may be verifying/creating signatures with SHA1. I do not
> > know whether there are any or how many there are.
> 
> And that is still supported if the crypto policy is set to LEGACY.
> 
> 
> > Is openssl preventing signature creation with SHA1 or also signature
> > verification involving SHA1?
> 
> Both signing and verification are disabled, unless the system was set to the
> legacy crypto policy. If you need SHA1 signatures, that is what you should
> do until you have had the chance to transition off SHA1.

Can't swtpm/libtpms use OPENSSL_INIT_set_config_appname() & OPENSSL_init_crypto() to "rh-allow-sha1-signatures = yes" in the meantime? Disabling sha1-in-sig only in vTPM is not possible. We would have to disable sha1 altogether, and only on fresh vTPMs, where capabilities are set during "manufacturing". Also we need to check the compability with various OS, in particular Windows.

No, swtpm should not hack around the SHA-1 disablement, that's something that the administrator should be aware of and agree to before it's performed.

As such, the expected behaviour is:
1. you need sha-1: stay on RHEL-8 or switch RHEL-9 to LEGACY policy
2. swtpm is executed in DEFAULT policy on RHEL-9: no SHA-1 in signatures are enabled or used.

You mentioned that you need sha-1 signatures because of a self-test... What decides what's part of the self-test, what operations are performed?

> we need to check the compability with various OS, in particular Windows.

please do

(In reply to Hubert Kario from comment #35)
> No, swtpm should not hack around the SHA-1 disablement, that's something
> that the administrator should be aware of and agree to before it's performed.
> 
> As such, the expected behaviour is:
> 1. you need sha-1: stay on RHEL-8 or switch RHEL-9 to LEGACY policy
> 2. swtpm is executed in DEFAULT policy on RHEL-9: no SHA-1 in signatures are
> enabled or used.
> 

We will have to communicate and document properly that any VM with vTPM from rhel8 will need the legacy system-wide policy to be migrated.


> You mentioned that you need sha-1 signatures because of a self-test... What
> decides what's part of the self-test, what operations are performed?

As said, the self-test is using sha512 already.

> > we need to check the compability with various OS, in particular Windows.
> 
> please do

easy to say..

If this is something that can be done on a per-VM basis I think it *might* be reasonable to add a config option in libvirt to enable sha1 for a specific guest, and implement it via custom settings as mentioned in #34, so that an administrator can allow specific VMs to use it for backwards compatibility.
However if this is a per-host setting then the only option should be for the admin to either set the crypto-policy to LEGACY or via a custom policy that re-enables just SHA1.

I have added an issue entry into the libtpms project that describes the problem with disabling SHA-1 for TPM 2 users: https://github.com/stefanberger/libtpms/issues/303

In short, we cannot just generally disable SHA-1 in TPM 2 because it will break people's upgrade path. Needless to say, this isn't a problem only for RHEL but for any other distro. So just disabling SHA-1 is a non-starter for the TPM 2 code.

Please advise on how to deal with the possible consequences of disabling SHA-1 because I have no answers.

(In reply to Marc-Andre Lureau from comment #36)
> (In reply to Hubert Kario from comment #35)
> > No, swtpm should not hack around the SHA-1 disablement, that's something
> > that the administrator should be aware of and agree to before it's performed.
> > 
> > As such, the expected behaviour is:
> > 1. you need sha-1: stay on RHEL-8 or switch RHEL-9 to LEGACY policy
> > 2. swtpm is executed in DEFAULT policy on RHEL-9: no SHA-1 in signatures are
> > enabled or used.
> > 
> 
> We will have to communicate and document properly that any VM with vTPM from
> rhel8 will need the legacy system-wide policy to be migrated.

Yes, we already have Release Notes doc text to notify users of that: see bug 2031742

> > You mentioned that you need sha-1 signatures because of a self-test... What
> > decides what's part of the self-test, what operations are performed?
> 
> As said, the self-test is using sha512 already.
> 
> > > we need to check the compability with various OS, in particular Windows.
> > 
> > please do
> 
> easy to say..

Don't we support running the host in FIPS mode? SHA-1 signatures in FIPS mode are already disabled in RHEL-8, so if there was no issue there, we shouldn't have one now.

(In reply to Stefan Berger from comment #38)
> I have added an issue entry into the libtpms project that describes the
> problem with disabling SHA-1 for TPM 2 users:
> https://github.com/stefanberger/libtpms/issues/303
> 
> In short, we cannot just generally disable SHA-1 in TPM 2 because it will
> break people's upgrade path. Needless to say, this isn't a problem only for
> RHEL but for any other distro. So just disabling SHA-1 is a non-starter for
> the TPM 2 code.
> 
> Please advise on how to deal with the possible consequences of disabling
> SHA-1 because I have no answers.

Sorry, but I don't see how the upgrade path is blocked...

Let's consider the situation:
- the user has a VM that uses the TPM, the VM regularly uses the TPM for RSA signatures with SHA-1
- they use RHEL-8, so it's working
- they upgrade to RHEL-9
- the OS in the VM stops working because the TPM refuses the RSA signatures with SHA-1
- the user switches the system to LEGACY policy with `update-crypto-policies --set LEGACY`
- the OS in the VM starts working again as now RSA signatures with SHA-1 are allowed
- the user updates the OS in the VM to not use SHA-1 any more
- they return the system to the default policy with `update-crypto-policies --set DEFAULT`

So they user can migrate to a stronger hash both on RHEL-8 and RHEL-9.

> - the OS in the VM stops working because the TPM refuses the RSA signatures with SHA-1
> - the user switches the system to LEGACY policy with `update-crypto-policies --set LEGACY`
> - the OS in the VM starts working again as now RSA signatures with SHA-1 are allowed

Sounds like a plan...

How would this work on layered products where the qemu/libvirt set is in a container?
(I think that's true for CNV and I'm not sure about RHOSP).

Where would they flip the policy? Who would do it?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: openssl), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:3900