Bug 2063723

Proposed upstream patch(es):
https://listman.redhat.com/archives/libvir-list/2022-October/234926.html

v2 posted upstream: https://listman.redhat.com/archives/libvir-list/2022-December/236390.html

Pushed upstream as:
commit ad3f33a9921feebb064e6071649486ede31a3973
Author:     Ján Tomko <jtomko>
CommitDate: 2022-12-19 10:33:20 +0100

    qemu: add external backend for tpm
    
    Introduce a new backend type 'external' for connecting to a swtpm daemon
    not managed by libvirtd.
    
    Mostly in one commit, thanks to -Wswitch and the way we generate
    capabilities.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2063723
    
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Michal Privoznik <mprivozn>

git describe: v8.10.0-132-gad3f33a992

Test with:
libvirt-9.0.0-1.el9.x86_64
qemu-kvm-7.2.0-4.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

Steps:
1.# virsh domcapabilities |grep /tpm -B12
    <tpm supported='yes'>
      <enum name='model'>
        <value>tpm-tis</value>
        <value>tpm-crb</value>
      </enum>
      <enum name='backendModel'>
        <value>emulator</value>
        <value>external</value>
      </enum>
      <enum name='backendVersion'>
        <value>2.0</value>
      </enum>
    </tpm>

2.start vm
# mkdir /tmp/mytpm
#  chcon -t virtd_exec_t /usr/bin/swtpm_setup 
#  chcon -t virtd_exec_t /usr/bin/swtpm

# systemd-run  swtpm_setup --tpm2 --tpmstate /tmp/mytpm --create-ek-cert --create-platform-cert --overwrite
Running as unit: run-r995379940785468196f426010edf0da2.service

# systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-r0a37494893cc4c6e87e80d3b53a6bcac.service

#  ll -hZ /tmp/guest-swtpm.sock 
srw-------. 1 root root system_u:object_r:tmp_t:s0 0 Jan 17 22:51 /tmp/guest-swtpm.sock

# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock
#  ll -hZ /tmp/guest-swtpm.sock 
srw-------. 1 qemu qemu system_u:object_r:svirt_image_t:s0 0 Jan 17 22:51 /tmp/guest-swtpm.sock

# virsh start avocado-vt-vm1 
Domain 'avocado-vt-vm1' started
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B5
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path='/tmp/guest-swtpm.sock'/>
      </backend>
      <alias name='tpm0'/>
    </tpm>

# virsh console avocado-vt-vm1 
[root@localhost ~]# tpm2_getrandom --hex 8
144a7ac1b8557103

# ps aux|grep tpm
root      409212  0.0  0.0   9860  6616 ?        Ss   22:51   0:00 /usr/bin/swtpm socket --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
qemu      409316 14.2  1.3 3963260 903428 ?      Sl   22:52   0:29 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,...-chardev socket,id=chrtpm,path=/tmp/guest-swtpm.sock -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm -device {"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}

audit.log
type=VIRT_RESOURCE msg=audit(1674013858.507:7924): pid=408184 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=tpm-external reason=start vm="avocado-vt-vm1" uuid=467fdf69-a66b-4202-9721-4762de554a49 device="/tmp/guest-swtpm.sock" exe="/usr/sbin/virtqemud" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=VIRT_RESOURCE msg=audit(1674013971.960:7952): pid=408184 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=tpm-external reason=start vm="avocado-vt-vm1" uuid=467fdf69-a66b-4202-9721-4762de554a49 device="/tmp/guest-swtpm.sock" exe="/usr/sbin/virtqemud" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"

3. Backend source attributes configuration
(1)source type:
a.        <source  mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define no_sourcetype.xml 
error: Failed to define domain from no_sourcetype.xml
error: XML error: missing external TPM backend type

b.       <source type='' mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define none_sourcetype.xml 
error: Failed to define domain from none_sourcetype.xml
error: XML error: missing external TPM backend type

c.   <source type='unixaa' mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define invalid_sourcetype.xml 
error: Failed to define domain from invalid_sourcetype.xml
error: unsupported configuration: unknown backend type 'unixaa' for external TPM

(2)source mode:
a.  <source  type='unix' path='/tmp/guest-swtpm.sock'/>
# virsh define no_sourcemode.xml 
Domain 'avocado-vt-vm1' defined from no_sourcemode.xml
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B4
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path='/tmp/guest-swtpm.sock'/>
      </backend>
    </tpm>

b.      <source type='unix' mode='' path='/tmp/guest-swtpm.sock'/>
# virsh define none_sourcemode.xml 
error: Failed to define domain from none_sourcemode.xml
error: XML error: Invalid value for attribute 'mode' in element 'source': ''.

c.        <source type='unix' mode='bind' path='/tmp/guest-swtpm.sock'/>
# virsh define unsupported_sourcemode.xml 
error: Failed to define domain from unsupported_sourcemode.xml
error: XML error: only 'client' mode is supported for external TPM device

d.        <source type='unix' mode='client' path='/tmp/guest-swtpm.sock'/>
# virsh define client_sourcemode.xml 
error: Failed to define domain from client_sourcemode.xml
error: XML error: Invalid value for attribute 'mode' in element 'source': 'client'.

(3)source path:
a.     <source type='unix' mode='connect' />
 virsh define no_sourcepath.xml 
error: Failed to define domain from no_sourcepath.xml
error: XML error: missing socket path for external TPM device

b.        <source type='unix' mode='connect' path=''/>
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B4
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path=''/>
      </backend>
    </tpm>
# virsh start avocado-vt-vm1 
error: Failed to start domain 'avocado-vt-vm1'
error: internal error: process exited while connecting to monitor: 2023-01-18T13:24:44.896145Z qemu-kvm: -chardev socket,id=chrtpm,path=: Failed to connect to '': Connection refused


4. # virsh suspend avocado-vt-vm1 
Domain 'avocado-vt-vm1' suspended

# virsh resume avocado-vt-vm1 
Domain 'avocado-vt-vm1' resumed

# systemctl restart libvirtd

Guest os can still be accessed with vtpm function can be used.

# virsh managedsave avocado-vt-vm1 

Domain 'avocado-vt-vm1' state saved by libvirt

# ps aux|grep swtpm
root      415641  0.0  0.0 221664  2396 pts/1    S+   03:23   0:00 grep --color=auto swtpm

# virsh start avocado-vt-vm1 
error: Failed to start domain 'avocado-vt-vm1'
error: internal error: process exited while connecting to monitor: 2023-01-18T08:23:54.685201Z qemu-kvm: -chardev socket,id=chrtpm,path=/tmp/guest-swtpm.sock: Failed to connect to '/tmp/guest-swtpm.sock': No such file or directory


Hi Ján, could you please answer some questions about my testing? Thank you!
1. In 3(1), is it possible to use “source type” instead of “backend type”? since tpm backend type is 'external' in xml, somewhat confusing.
2. From 3(2)c,d, is it better to replace "only 'client' mode" with "only 'connect' mode"? since actually string 'client' is reported as invalid.
3. From 4, it doesn’t support any operations needs vm shutoff, right? Or its user's duty to reconfigure swtpm socket running on local or remote before restart,restore,migration,etc?
4. Looking forward to updates in https://libvirt.org/formatdomain.html#tpm-device, better with user guide example for external swtpm process setup. This is a minor request, will not blocking this bug's verification.

Thank you!

1. Yes, using source type would be less confusing.
2. Oops, that is a leftover from an earlier version of the patches.

Patch sent:
https://listman.redhat.com/archives/libvir-list/2023-January/237200.html

3. That is up to whoever runs the swtpm
4. This is not really meant to be used by end users, but other applications
that want to manage swtpm by themselves. Libvirt users can just use the
one managed by libvirt which should work automatically.

Pushed as:
commit 1c7476c8797b7f0d6e8d607f6a42c5bf43441677
Author:     Ján Tomko <jtomko>
CommitDate: 2023-01-23 13:44:16 +0100

    conf: clarify some external TPM error messages
    
git describe: v9.0.0-40-g1c7476c879

1. Retested with:
libvirt-9.0.0-2.el9.x86_64
qemu-kvm-7.2.0-5.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

(Also with a scratch build libvirt-9.0.0-3.el9_rc.dd8fc4c135.x86_64 with some related fixes)

Almost same steps and results with before, except some updates:
# virsh define no_sourcetype.xml 
error: Failed to define domain from no_sourcetype.xml
error: XML error: missing external TPM backend source type

# virsh define none_sourcetype.xml 
error: Failed to define domain from none_sourcetype.xml
error: XML error: missing external TPM backend source type

# virsh define invalid_sourcetype.xml 
error: Failed to define domain from invalid_sourcetype.xml
error: unsupported configuration: unknown backend source type 'unixaa' for external TPM

# virsh define unsupported_sourcemode.xml 
error: Failed to define domain from unsupported_sourcemode.xml
error: XML error: only 'connect' mode is supported for external TPM device

2. New test with libvirt-9.0.0-3.el9_rc.dd8fc4c135.x86_64
(1)# virsh managedsave avocado-vt-vm1 

Domain 'avocado-vt-vm1' state saved by libvirt

Provide swtpm.sock again before restore:
# systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-reb7165c4ff304120a474c54f8c4682f4.service
# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock

# virsh start avocado-vt-vm1 
Domain 'avocado-vt-vm1' started

# virsh console avocado-vt-vm1 
[root@localhost ~]# tpm2_getrandom --hex 8
f616909a901bb718

(2)
Before migrate, prepare same swtpm process on target.

#  virsh migrate avocado-vt-vm1 --live qemu+ssh://hostB/system --verbose
Migration: [100 %]

-rw-r-----. 1 root root system_u:object_r:user_tmp_t:s0        0 Jan 28 04:10 .lock
-rw-------. 1 root root system_u:object_r:user_tmp_t:s0     9.0K Jan 28 04:16 tpm2-00.permall

Before migrate back, provide swtpm.sock again on hostA:
#  systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-r04623740ff6449cc9166d2ad8264e145.service
# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock

[hostB]# virsh migrate avocado-vt-vm1 --live qemu+ssh://hostA/system --verbose
Migration: [100 %]

Verify on:
libvirt-9.0.0-4.el9.x86_64
qemu-kvm-7.2.0-7.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

Other steps and results are same as before, PASS.


Except one issue:
virtqemud coredumped when vm migrating back with external vtpm.
Also reproduces when migrating a transient vm out.

[hostB]# virsh migrate avocado-vt-vm1 --live --verbose qemu+ssh://hostA/system
Migration: [ 99 %]2023-02-09 13:46:50.093+0000: 111849: info : libvirt version: 9.0.0, package: 4.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-02-09-05:54:47, )
2023-02-09 13:46:50.093+0000: 111849: info : hostname: hostB
2023-02-09 13:46:50.093+0000: 111849: warning : virDomainMigrateVersion3Full:3493 : Guest avocado-vt-vm1 probably left in 'paused' state on source
error: Disconnected from qemu:///system due to end of file
Migration: [100 %]

But on hostA, the vm running well and vtpm works well in OS.

[hostB]# coredumpctl list
TIME                           PID UID GID SIG     COREFILE EXE                 SIZE
Thu 2023-02-09 08:20:20 EST 109275   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.1M
Thu 2023-02-09 08:22:22 EST 109757   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.0M
Thu 2023-02-09 08:46:50 EST 111593   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.0M

Additional info:
1.Not reproduce for normal vtpm like:
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
    </tpm>
2. coredump info pls check attachment.


Hi Jan,

May this issue be related to my any improper configuration of external swtpm process? Or should I file a new bug to fix it?

Thank you!

Created attachment 1943113 [details]
bz2063723#c12_coredumpinfo

Please file a new bug.

Mark this bug as verified.
The comment12 issue will be tracked in https://bugzilla.redhat.com/show_bug.cgi?id=2168762.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2171