Bug 2064115

Summary: Start encrypted tpm guest failed
Product: Red Hat Enterprise Linux 9 Reporter: Meina Li <meili>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
libvirt sub component: General QA Contact: Yanqiu Zhang <yanqzhan>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: jdenemar, jsuchane, lmen, marcandre.lureau, mpitt, mprivozn, qcheng, virt-maint, weizhan, xuwei, xuzhang, yanqzhan
Version: 9.1Keywords: Automation, Regression, Triaged, Upstream
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-8.2.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 10:03:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meina Li 2022-03-15 05:44:53 UTC
Description of problem:
Start encrypted tpm guest failed

Version-Release number of selected component (if applicable):
libvirt-8.1.0-1.el9.x86_64
qemu-kvm-6.2.0-11.el9.x86_64
swtpm-0.7.0-1.20211109gitb79fd91.el9.x86_64
libtpms-0.9.1-0.20211126git1ff6fe1f43.el9.x86_64
edk2-ovmf-20220126gitbb1bba3d77-3.el9.noarch
kernel-5.14.0-70.1.1.el9.x86_64
openssl-3.0.1-17.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a tpm secret.
# cat secret.xml 
<secret ephemeral="no" private="yes">
<description>sample vTPM secret</description>
<usage type="vtpm">
<name>VTPM_example</name>
</usage>
</secret>
# virsh secret-define secret.xml 
Secret fe7c5949-528a-404b-9a63-2be66113796b created
# MYSECRET=`printf %s "open sesame" | base64`
# virsh secret-set-value --secret fe7c5949-528a-404b-9a63-2be66113796b $MYSECRET
error: Passing secret value as command-line argument is insecure!
Secret value set
# virsh secret-list
 UUID                                   Usage
-----------------------------------------------------------
 fe7c5949-528a-404b-9a63-2be66113796b   vtpm VTPM_example

2. Define a tpm guest with tpm device.
# vim lmn.xml
......
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'>
        <encryption secret='fe7c5949-528a-404b-9a63-2be66113796b'/>
      </backend>
    </tpm>
......
# virsh define lmn.xml 
Domain 'lmn' defined from lmn.xml

3.Start the guest.
# ls /var/lib/libvirt/swtpm/  --no tpm file
# virsh start lmn
error: Failed to start domain 'lmn'
error: internal error: qemu unexpectedly closed the monitor: 2022-03-15T05:28:05.862730Z qemu-kvm: tpm-emulator: TPM result for CMD_INIT: 0x101 operation failed

Actual results:
Start the guest failed

Expected results:
Can start guest

Additional info:
1) Swtpm.log:
Successfully created EK certificate locally.
Successfully created NVRAM area 0x1c00016 for ECC EK certificate.
Successfully activated PCR banks sha256 among sha1,sha256,sha384,sha512.
Successfully authored TPM state.
Ending vTPM manufacturing @ Tue 15 Mar 2022 06:28:05 AM CET
Verification of HMAC failed. Data integrity is compromised
SWTPM_NVRAM_LoadData: Error from SWTPM_NVRAM_GetDecryptedData rc = 33
Verification of HMAC failed. Data integrity is compromised
SWTPM_NVRAM_LoadData: Error from SWTPM_NVRAM_GetDecryptedData rc = 33
libtpms/tpm2: Entering failure mode; code: 8, location: NvPowerOn line 126
Error: Could not initialize libtpms.
Error: Could not initialize the TPM
Data client disconnected
2) After step 3, if we remove the tpm file and start again, the guest will start.
# rm -rf /var/lib/libvirt/swtpm/2108a219-bdf5-46b3-9ff2-acef47b48d23/tpm2/tpm2-00.permall 
# virsh start lmn
Domain 'lmn' started

Comment 6 Jaroslav Suchanek 2022-03-21 11:01:12 UTC
Michal, pls, have a look. Thanks.

Comment 9 Michal Privoznik 2022-03-21 15:03:05 UTC
Patch proposed on the list:

https://listman.redhat.com/archives/libvir-list/2022-March/229433.html

Comment 11 Yanqiu Zhang 2022-04-02 08:08:57 UTC
Issue is not reproduced on rhel9.1 with:
qemu-kvm-6.2.0-12.el9.x86_64
libvirt-8.2.0-1.el9.x86_64
swtpm-0.7.0-1.20211109gitb79fd91.el9.x86_64
libtpms-0.9.1-0.20211126git1ff6fe1f43.el9.x86_64
edk2-ovmf-20220126gitbb1bba3d77-4.el9.noarch
openssl-3.0.1-21.el9.x86_64

Steps:
# virsh start vm-ovmf 
Domain 'vm-ovmf' started

# virsh dumpxml vm-ovmf |grep /tpm -B5
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'>
        <encryption secret='40f4e01e-02d9-48e3-8b1d-b5985238d1e2'/>
      </backend>
      <alias name='tpm0'/>
    </tpm>

In guest os:
[root@localhost ~]# ls /dev/|grep tpm
tpm0
tpmrm0
[root@localhost ~]# tpm2_getrandom  --hex 16
d6837351b53a77315daee10a1414c784

Comment 13 Michal Privoznik 2022-04-04 06:56:01 UTC
Merged upstream as:

commit 4d7bb0177a33c4e90fd001edfe27bc030354d875
Author:     Michal Prívozník <mprivozn>
AuthorDate: Mon Mar 21 13:33:06 2022 +0100
Commit:     Michal Prívozník <mprivozn>
CommitDate: Mon Mar 28 10:00:18 2022 +0200

    qemu_tpm: Do async IO when starting swtpm emulator
    
    When vTPM is secured via virSecret libvirt passes the secret
    value via an FD when swtpm is started (arguments --key and
    --migration-key). The writing of the secret into the FDs is
    handled via virCommand, specifically qemu_tpm calls
    virCommandSetSendBuffer()) and then virCommandRunAsync() spawns a
    thread to handle writing into the FD via
    virCommandDoAsyncIOHelper. But the thread is not created unless
    VIR_EXEC_ASYNC_IO flag is set, which it isn't. In order to fix
    it, virCommandDoAsyncIO() must be called.
    
    The credit goes to Marc-André Lureau
    <marcandre.lureau> who has done all the debugging and
    proposed fix in the bugzilla.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2064115
    Fixes: a9c500d2b50c5c041a1bb6ae9724402cf1cec8fe
    Signed-off-by: Michal Privoznik <mprivozn>
    Reviewed-by: Jiri Denemark <jdenemar>

v8.1.0-229-g4d7bb0177a

While there are more fixes in the patchset, those are more cleanup of an internal code than bugfixes:

https://listman.redhat.com/archives/libvir-list/2022-March/229480.html

Comment 17 Yanqiu Zhang 2022-04-14 02:33:53 UTC
Auto regression test passed:
Pkgs info:
    libvirt	libvirt-8.2.0-1.el9.x86_64
    qemu-kvm	qemu-kvm-6.2.0-12.el9.x86_64
    kernel	kernel-5.14.0-75.el9.x86_64
    swtpm       0.7.0-1.20211109gitb79fd91.el9
    libtpms     0.9.1-0.20211126git1ff6fe1f43.el9
    edk2-ovmf   20220221gitb24306f15d-1.el9
Job url:
    https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-9.1%20x86_64/job/libvirt-RHEL-9.1-runtest-x86_64-function-tpm_emulator/6/testReport/
    All cases passed except 2 skipped by existing bz2025520.

Comment 19 errata-xmlrpc 2022-11-15 10:03:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Low: libvirt security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:8003