2063723 – RFE: Allow TPM device to connect to an externally launched swtpm

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2063723 - RFE: Allow TPM device to connect to an externally launched swtpm

Summary: RFE: Allow TPM device to connect to an externally launched swtpm

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Ján Tomko
QA Contact:	Yanqiu Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2022-03-14 09:05 UTC by Michal Privoznik
Modified:	2023-07-26 08:35 UTC (History)
CC List:	9 users (show)
Fixed In Version:	libvirt-9.0.0-2.el9
Doc Type:	Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed:	2023-05-09 07:26:11 UTC
Type:	Feature Request
Target Upstream Version:	9.1.0
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Issue Tracker	LIBVIRTAT-14053	None	None	None	2023-04-27 06:28:18 UTC
Red Hat Issue Tracker	LIBVIRTAT-14054	None	None	None	2023-05-26 09:06:12 UTC
Red Hat Issue Tracker	LIBVIRTAT-14055	None	None	None	2023-05-26 09:06:12 UTC
Red Hat Issue Tracker	RHELPLAN-115476	None	None	None	2022-03-14 09:16:22 UTC
Red Hat Product Errata	RHBA-2023:2171	None	None	None	2023-05-09 07:26:51 UTC

Description Michal Privoznik 2022-03-14 09:05:15 UTC

Description of problem:
The goal is to allow the user to setup their own (sw)tpm daemon and have libvirt/qemu talk to it instead of automatically creating a new one.
More specifically, this would allow running swtpm in its own separate container.
Aside from the potential security benefits, removing the extra process from the libvirt environment would help with CPU/memory guarantees.

This would probably involve adding a "socket" option to the section. Providing a socket would void most other TPM options, as libvirt wouldn't have to start anything.
On migration, I guess libvirt would assume that a socket is already created on the target. The value of the socket option probably has to be mutable on migration in case the target layout is different.


Upstream gitlab issue: https://gitlab.com/libvirt/libvirt/-/issues/281

Comment 1 Ján Tomko 2022-10-17 14:28:08 UTC

Proposed upstream patch(es):
https://listman.redhat.com/archives/libvir-list/2022-October/234926.html

Comment 2 Ján Tomko 2022-12-15 13:57:38 UTC

v2 posted upstream: https://listman.redhat.com/archives/libvir-list/2022-December/236390.html

Comment 3 Ján Tomko 2022-12-19 09:47:34 UTC

Pushed upstream as:
commit ad3f33a9921feebb064e6071649486ede31a3973
Author:     Ján Tomko <jtomko>
CommitDate: 2022-12-19 10:33:20 +0100

    qemu: add external backend for tpm
    
    Introduce a new backend type 'external' for connecting to a swtpm daemon
    not managed by libvirtd.
    
    Mostly in one commit, thanks to -Wswitch and the way we generate
    capabilities.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=2063723
    
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Michal Privoznik <mprivozn>

git describe: v8.10.0-132-gad3f33a992

Comment 4 Yanqiu Zhang 2023-01-18 07:38:34 UTC

Test with:
libvirt-9.0.0-1.el9.x86_64
qemu-kvm-7.2.0-4.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

Steps:
1.# virsh domcapabilities |grep /tpm -B12
    <tpm supported='yes'>
      <enum name='model'>
        <value>tpm-tis</value>
        <value>tpm-crb</value>
      </enum>
      <enum name='backendModel'>
        <value>emulator</value>
        <value>external</value>
      </enum>
      <enum name='backendVersion'>
        <value>2.0</value>
      </enum>
    </tpm>

2.start vm
# mkdir /tmp/mytpm
#  chcon -t virtd_exec_t /usr/bin/swtpm_setup 
#  chcon -t virtd_exec_t /usr/bin/swtpm

# systemd-run  swtpm_setup --tpm2 --tpmstate /tmp/mytpm --create-ek-cert --create-platform-cert --overwrite
Running as unit: run-r995379940785468196f426010edf0da2.service

# systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-r0a37494893cc4c6e87e80d3b53a6bcac.service

#  ll -hZ /tmp/guest-swtpm.sock 
srw-------. 1 root root system_u:object_r:tmp_t:s0 0 Jan 17 22:51 /tmp/guest-swtpm.sock

# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock
#  ll -hZ /tmp/guest-swtpm.sock 
srw-------. 1 qemu qemu system_u:object_r:svirt_image_t:s0 0 Jan 17 22:51 /tmp/guest-swtpm.sock

# virsh start avocado-vt-vm1 
Domain 'avocado-vt-vm1' started
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B5
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path='/tmp/guest-swtpm.sock'/>
      </backend>
      <alias name='tpm0'/>
    </tpm>

# virsh console avocado-vt-vm1 
[root@localhost ~]# tpm2_getrandom --hex 8
144a7ac1b8557103

# ps aux|grep tpm
root      409212  0.0  0.0   9860  6616 ?        Ss   22:51   0:00 /usr/bin/swtpm socket --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
qemu      409316 14.2  1.3 3963260 903428 ?      Sl   22:52   0:29 /usr/libexec/qemu-kvm -name guest=avocado-vt-vm1,...-chardev socket,id=chrtpm,path=/tmp/guest-swtpm.sock -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm -device {"driver":"tpm-crb","tpmdev":"tpm-tpm0","id":"tpm0"}

audit.log
type=VIRT_RESOURCE msg=audit(1674013858.507:7924): pid=408184 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=tpm-external reason=start vm="avocado-vt-vm1" uuid=467fdf69-a66b-4202-9721-4762de554a49 device="/tmp/guest-swtpm.sock" exe="/usr/sbin/virtqemud" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"
type=VIRT_RESOURCE msg=audit(1674013971.960:7952): pid=408184 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:virtd_t:s0-s0:c0.c1023 msg='virt=kvm resrc=tpm-external reason=start vm="avocado-vt-vm1" uuid=467fdf69-a66b-4202-9721-4762de554a49 device="/tmp/guest-swtpm.sock" exe="/usr/sbin/virtqemud" hostname=? addr=? terminal=? res=success'UID="root" AUID="unset"

Comment 5 Yanqiu Zhang 2023-01-18 14:10:36 UTC

3. Backend source attributes configuration
(1)source type:
a.        <source  mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define no_sourcetype.xml 
error: Failed to define domain from no_sourcetype.xml
error: XML error: missing external TPM backend type

b.       <source type='' mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define none_sourcetype.xml 
error: Failed to define domain from none_sourcetype.xml
error: XML error: missing external TPM backend type

c.   <source type='unixaa' mode='connect' path='/tmp/guest-swtpm.sock'/>
# virsh define invalid_sourcetype.xml 
error: Failed to define domain from invalid_sourcetype.xml
error: unsupported configuration: unknown backend type 'unixaa' for external TPM

(2)source mode:
a.  <source  type='unix' path='/tmp/guest-swtpm.sock'/>
# virsh define no_sourcemode.xml 
Domain 'avocado-vt-vm1' defined from no_sourcemode.xml
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B4
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path='/tmp/guest-swtpm.sock'/>
      </backend>
    </tpm>

b.      <source type='unix' mode='' path='/tmp/guest-swtpm.sock'/>
# virsh define none_sourcemode.xml 
error: Failed to define domain from none_sourcemode.xml
error: XML error: Invalid value for attribute 'mode' in element 'source': ''.

c.        <source type='unix' mode='bind' path='/tmp/guest-swtpm.sock'/>
# virsh define unsupported_sourcemode.xml 
error: Failed to define domain from unsupported_sourcemode.xml
error: XML error: only 'client' mode is supported for external TPM device

d.        <source type='unix' mode='client' path='/tmp/guest-swtpm.sock'/>
# virsh define client_sourcemode.xml 
error: Failed to define domain from client_sourcemode.xml
error: XML error: Invalid value for attribute 'mode' in element 'source': 'client'.

(3)source path:
a.     <source type='unix' mode='connect' />
 virsh define no_sourcepath.xml 
error: Failed to define domain from no_sourcepath.xml
error: XML error: missing socket path for external TPM device

b.        <source type='unix' mode='connect' path=''/>
# virsh dumpxml avocado-vt-vm1 |grep /tpm -B4
    <tpm model='tpm-crb'>
      <backend type='external'>
        <source type='unix' mode='connect' path=''/>
      </backend>
    </tpm>
# virsh start avocado-vt-vm1 
error: Failed to start domain 'avocado-vt-vm1'
error: internal error: process exited while connecting to monitor: 2023-01-18T13:24:44.896145Z qemu-kvm: -chardev socket,id=chrtpm,path=: Failed to connect to '': Connection refused


4. # virsh suspend avocado-vt-vm1 
Domain 'avocado-vt-vm1' suspended

# virsh resume avocado-vt-vm1 
Domain 'avocado-vt-vm1' resumed

# systemctl restart libvirtd

Guest os can still be accessed with vtpm function can be used.

# virsh managedsave avocado-vt-vm1 

Domain 'avocado-vt-vm1' state saved by libvirt

# ps aux|grep swtpm
root      415641  0.0  0.0 221664  2396 pts/1    S+   03:23   0:00 grep --color=auto swtpm

# virsh start avocado-vt-vm1 
error: Failed to start domain 'avocado-vt-vm1'
error: internal error: process exited while connecting to monitor: 2023-01-18T08:23:54.685201Z qemu-kvm: -chardev socket,id=chrtpm,path=/tmp/guest-swtpm.sock: Failed to connect to '/tmp/guest-swtpm.sock': No such file or directory


Hi Ján, could you please answer some questions about my testing? Thank you!
1. In 3(1), is it possible to use “source type” instead of “backend type”? since tpm backend type is 'external' in xml, somewhat confusing.
2. From 3(2)c,d, is it better to replace "only 'client' mode" with "only 'connect' mode"? since actually string 'client' is reported as invalid.
3. From 4, it doesn’t support any operations needs vm shutoff, right? Or its user's duty to reconfigure swtpm socket running on local or remote before restart,restore,migration,etc?
4. Looking forward to updates in https://libvirt.org/formatdomain.html#tpm-device, better with user guide example for external swtpm process setup. This is a minor request, will not blocking this bug's verification.

Thank you!

Comment 6 Ján Tomko 2023-01-20 12:03:38 UTC

1. Yes, using source type would be less confusing.
2. Oops, that is a leftover from an earlier version of the patches.

Patch sent:
https://listman.redhat.com/archives/libvir-list/2023-January/237200.html

3. That is up to whoever runs the swtpm
4. This is not really meant to be used by end users, but other applications
that want to manage swtpm by themselves. Libvirt users can just use the
one managed by libvirt which should work automatically.

Comment 7 Ján Tomko 2023-01-23 12:49:32 UTC

Pushed as:
commit 1c7476c8797b7f0d6e8d607f6a42c5bf43441677
Author:     Ján Tomko <jtomko>
CommitDate: 2023-01-23 13:44:16 +0100

    conf: clarify some external TPM error messages
    
git describe: v9.0.0-40-g1c7476c879

Comment 9 Yanqiu Zhang 2023-01-28 09:28:57 UTC

1. Retested with:
libvirt-9.0.0-2.el9.x86_64
qemu-kvm-7.2.0-5.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

(Also with a scratch build libvirt-9.0.0-3.el9_rc.dd8fc4c135.x86_64 with some related fixes)

Almost same steps and results with before, except some updates:
# virsh define no_sourcetype.xml 
error: Failed to define domain from no_sourcetype.xml
error: XML error: missing external TPM backend source type

# virsh define none_sourcetype.xml 
error: Failed to define domain from none_sourcetype.xml
error: XML error: missing external TPM backend source type

# virsh define invalid_sourcetype.xml 
error: Failed to define domain from invalid_sourcetype.xml
error: unsupported configuration: unknown backend source type 'unixaa' for external TPM

# virsh define unsupported_sourcemode.xml 
error: Failed to define domain from unsupported_sourcemode.xml
error: XML error: only 'connect' mode is supported for external TPM device

2. New test with libvirt-9.0.0-3.el9_rc.dd8fc4c135.x86_64
(1)# virsh managedsave avocado-vt-vm1 

Domain 'avocado-vt-vm1' state saved by libvirt

Provide swtpm.sock again before restore:
# systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-reb7165c4ff304120a474c54f8c4682f4.service
# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock

# virsh start avocado-vt-vm1 
Domain 'avocado-vt-vm1' started

# virsh console avocado-vt-vm1 
[root@localhost ~]# tpm2_getrandom --hex 8
f616909a901bb718

(2)
Before migrate, prepare same swtpm process on target.

#  virsh migrate avocado-vt-vm1 --live qemu+ssh://hostB/system --verbose
Migration: [100 %]

-rw-r-----. 1 root root system_u:object_r:user_tmp_t:s0        0 Jan 28 04:10 .lock
-rw-------. 1 root root system_u:object_r:user_tmp_t:s0     9.0K Jan 28 04:16 tpm2-00.permall

Before migrate back, provide swtpm.sock again on hostA:
#  systemd-run /usr/bin/swtpm socket  --ctrl type=unixio,path=/tmp/guest-swtpm.sock,mode=0600 --tpmstate dir=/tmp/mytpm,mode=0600 --tpm2
Running as unit: run-r04623740ff6449cc9166d2ad8264e145.service
# chcon -t svirt_image_t /tmp/guest-swtpm.sock; chown qemu:qemu   /tmp/guest-swtpm.sock

[hostB]# virsh migrate avocado-vt-vm1 --live qemu+ssh://hostA/system --verbose
Migration: [100 %]

Comment 12 Yanqiu Zhang 2023-02-09 15:19:14 UTC

Verify on:
libvirt-9.0.0-4.el9.x86_64
qemu-kvm-7.2.0-7.el9.x86_64
swtpm-0.8.0-1.el9.x86_64
libtpms-0.9.1-2.20211126git1ff6fe1f43.el9.x86_64

Other steps and results are same as before, PASS.


Except one issue:
virtqemud coredumped when vm migrating back with external vtpm.
Also reproduces when migrating a transient vm out.

[hostB]# virsh migrate avocado-vt-vm1 --live --verbose qemu+ssh://hostA/system
Migration: [ 99 %]2023-02-09 13:46:50.093+0000: 111849: info : libvirt version: 9.0.0, package: 4.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-02-09-05:54:47, )
2023-02-09 13:46:50.093+0000: 111849: info : hostname: hostB
2023-02-09 13:46:50.093+0000: 111849: warning : virDomainMigrateVersion3Full:3493 : Guest avocado-vt-vm1 probably left in 'paused' state on source
error: Disconnected from qemu:///system due to end of file
Migration: [100 %]

But on hostA, the vm running well and vtpm works well in OS.

[hostB]# coredumpctl list
TIME                           PID UID GID SIG     COREFILE EXE                 SIZE
Thu 2023-02-09 08:20:20 EST 109275   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.1M
Thu 2023-02-09 08:22:22 EST 109757   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.0M
Thu 2023-02-09 08:46:50 EST 111593   0   0 SIGSEGV present  /usr/sbin/virtqemud 1.0M

Additional info:
1.Not reproduce for normal vtpm like:
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'/>
    </tpm>
2. coredump info pls check attachment.


Hi Jan,

May this issue be related to my any improper configuration of external swtpm process? Or should I file a new bug to fix it?

Thank you!

Comment 13 Yanqiu Zhang 2023-02-09 15:22:23 UTC

Created attachment 1943113 [details]
bz2063723#c12_coredumpinfo

Comment 14 Ján Tomko 2023-02-09 15:46:59 UTC

Please file a new bug.

Comment 15 Yanqiu Zhang 2023-02-10 02:58:12 UTC

Mark this bug as verified.
The comment12 issue will be tracked in https://bugzilla.redhat.com/show_bug.cgi?id=2168762.

Comment 17 errata-xmlrpc 2023-05-09 07:26:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2171

Note You need to log in before you can comment on or make changes to this bug.