Bug 1755803 - Failed to start guest with tpm passthroughed when remember_owner = 1
Summary: Failed to start guest with tpm passthroughed when remember_owner = 1
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.1
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: 8.2
Assignee: Michal Privoznik
QA Contact: Yanqiu Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-26 08:57 UTC by Yanqiu Zhang
Modified: 2020-05-05 09:51 UTC (History)
14 users (show)

Fixed In Version: libvirt-5.9.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-05 09:50:34 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 0 None None None 2020-05-05 09:51:42 UTC

Description Yanqiu Zhang 2019-09-26 08:57:45 UTC
Description of problem:
Failed to start guest with tpm passthroughed when remember_owner = 1

Version-Release number of selected component (if applicable):
libvirt-daemon-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64
qemu-kvm-4.1.0-12.module+el8.1.0+4292+1293ae0c.x86_64
tpm2-tools-3.1.4-5.el8.x86_64
tpm2-tss-2.0.0-4.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Prepare a host with a tpm2.0 chip, and enable it in bios.

2. Try to start a guest with tpm passthrough:
    <tpm model='tpm-crb'>
      <backend type='passthrough'>
        <device path='/dev/tpm0'/>
      </backend>
      <alias name='tpm0'/>
    </tpm>

# virsh start avocado-vt-vm1
error: Failed to start domain avocado-vt-vm1
error: internal error: child reported (status=125): unable to open /dev/tpm0: Device or resource busy

2019-09-26 08:41:30.842+0000: 4291: error : virProcessRunInFork:1170 : internal error: child reported (status=125): unable to open /dev/tpm0: Device or resource busy
2019-09-26 08:41:30.847+0000: 4291: error : virProcessRunInFork:1170 : internal error: child reported (status=125): unable to open /dev/tpm0: Device or resource busy


Actual results:


Expected results:


Additional info:
Guest can be started successfully when set remember_owner = 0.

Comment 1 Yanqiu Zhang 2019-09-26 09:01:52 UTC
Supplementary info for comment0:
The parameter is set in /etc/libvirt/qemu.conf:
  # Whether libvirt should remember and restore the original
  # ownership over files it is relabeling. Defaults to 1, set
  # to 0 to disable the feature.
  #remember_owner = 1

Comment 2 Michal Privoznik 2019-10-01 15:03:13 UTC
Patches posted upstream:

https://www.redhat.com/archives/libvir-list/2019-October/msg00047.html

Comment 3 Michal Privoznik 2019-10-11 15:08:28 UTC
Patches pushed upstream:

4e95cdcbb3 security: Don't remember labels for TPM
2b44cf8c32 security_dac: Allow selective remember/recall for chardevs
1a84a1ced1 security: Try to lock only paths with remember == true

v5.8.0-80-g4e95cdcbb3

Comment 5 Yanqiu Zhang 2020-01-20 13:12:02 UTC
Verify on:
libvirt-daemon-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64
qemu-kvm-4.2.0-6.module+el8.2.0+5453+31b2b136.x86_64

Steps:
Scenaro1: Keep default value for remember_owner
1.  # grep remember_owner -i /etc/libvirt/qemu.conf
#remember_owner = 1

2. prepare tpm2.0 chip on host:
#  ll -Z /dev/tpm0
crw-------. 1 root root system_u:object_r:tpm_device_t:s0 10, 224 Jan 19 06:07 /dev/tpm0
# getfattr -m trusted.libvirt.security -d  /dev/tpm0 
# tpm2_getrandom 5
0x5C 0xAB 0xD9 0x39 0x30

3. Start guest with tpm passthroughed:
# virsh start rhel8.1-ovmf
Domain rhel8.1-ovmf started
# tpm2_getrandom 5
ERROR:tcti:src/tss2-tcti/tcti-device.c:281:Tss2_Tcti_Device_Init() Failed to open device file /dev/tpm0: Device or resource busy 
ERROR: tcti init allocation routine failed for library: "device" options: "(null)"
ERROR: Could not load tcti, got: "device"

# virsh console rhel8.1-ovmf
Connected to domain rhel8.1-ovmf
Escape character is ^]

Red Hat Enterprise Linux 8.2 Beta (Ootpa)
Kernel 4.18.0-167.el8.x86_64 on an x86_64

[root@localhost ~]# tpm2_getrandom 5
0x99 0x31 0x66 0x53 0x5C

4. Shutdown the guest:
# virsh shutdown rhel8.1-ovmf
Domain rhel8.1-ovmf is being shutdown

# tpm2_getrandom 5
0x72 0x27 0x8A 0x84 0x99

During this period, the output of 'll -Z' and 'getfattr' never changed.

Scenario2: set remember_owner = 1
#  grep remember_owner -i /etc/libvirt/qemu.conf
remember_owner = 1
# systemctl restart libvirtd

Got same result.

Scenario2: set remember_owner = 0
#  grep remember_owner -i /etc/libvirt/qemu.conf
remember_owner = 0
# systemctl restart libvirtd

Got same result.

Comment 6 Yanqiu Zhang 2020-02-06 04:42:19 UTC
Chardev regression test:
Pkgs:
libvirt-daemon-6.0.0-3.module+el8.2.0+5633+b0e06c1a.x86_64
qemu-kvm-4.2.0-8.module+el8.2.0+5607+dc756904.x86_64

Steps:
S1. Serial: pty pci-serial
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10000'/>
    </serial>

1. start guest
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# virsh dumpxml avocado-vt-vm1 |grep 'serial t' -A6
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10000'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
--

# ll -Z /dev/pts/1; getfattr -m trusted.libvirt.security -d  /dev/pts/1
crw--w----. 1 qemu tty system_u:object_r:svirt_devpts_t:s0 136, 1 Feb  5 03:09 /dev/pts/1

Communication works well:
[host]# cat /dev/pts/1
[guest]# echo abc123 > /dev/ttyS1
[host]# cat /dev/pts/1
abc123

2. hotplug and unplug
# virsh attach-device avocado-vt-vm1 serial-pty.xml
Device attached successfully

# virsh dumpxml avocado-vt-vm1 |grep 'serial t' -A6
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10000'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x01' function='0x0'/>
--
…
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target type='pci-serial' port='0'>
        <model name='pci-serial'/>
      </target>
      <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10001'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x02' function='0x0'/>

# ll -Z /dev/pts/4; getfattr -m trusted.libvirt.security -d  /dev/pts/4
crw--w----. 1 qemu tty system_u:object_r:svirt_devpts_t:s0 136, 4 Feb  5 03:21 /dev/pts/4

Communication also works well:
# cat /dev/pts/4
[guest]# echo hello! > /dev/ttyS2
[host]# cat /dev/pts/4
hello!

# virsh detach-device avocado-vt-vm1 serial-pty.xml
Device detached successfully

# ll -Z /dev/pts/4; getfattr -m trusted.libvirt.security -d  /dev/pts/4
ls: cannot access '/dev/pts/4': No such file or directory
getfattr: /dev/pts/4: No such file or directory

3. restart libvirtd
# systemctl restart libvirtd
# ll -Z /dev/pts/1; getfattr -m trusted.libvirt.security -d  /dev/pts/1
crw--w----. 1 qemu tty system_u:object_r:svirt_devpts_t:s0 136, 1 Feb  5 03:13 /dev/pts/1
4.shutdown guest
# virsh shutdown avocado-vt-vm1
Domain avocado-vt-vm1 is being shutdown

# ll -Z /dev/pts/1; getfattr -m trusted.libvirt.security -d  /dev/pts/1
ls: cannot access '/dev/pts/1': No such file or directory
getfattr: /dev/pts/1: No such file or directory

Special need be mentioned:
file can only be auto generated when guest start, edit assign will dispear in xml;
no xattr for the file;
file will be auto cleaned when vm not running.

S2. Console: unix bind 
     <console type='unix'>
     <source mode='bind' path='/var/lib/libvirt/qemu/foo'/>
     <target type='virtio' port='6'/>
     <alias name='ua-c769f8ac-922a-4518-9946-2a44172d7407'/>
    </console>

1. start guest:
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# ll -Z /var/lib/libvirt/qemu/foo;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 04:42 /var/lib/libvirt/qemu/foo
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/qemu/foo
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.timestamp_dac="1580718107"

# virsh dumpxml avocado-vt-vm1 |grep 'console t' -A6
...
    <console type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/foo'/>
      <target type='virtio' port='1'/>
      <alias name='ua-c769f8ac-922a-4518-9946-2a44172d7407'/>
    </console>

[guest]# echo hello > /dev/hvc0
[host]# nc -U /var/lib/libvirt/qemu/foo
hello

2. hotplug unplug:
# virsh attach-device avocado-vt-vm1 console-unix.xml
Device attached successfully

# virsh dumpxml avocado-vt-vm1 |grep 'console t' -A6
...    <console type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/foo'/>
      <target type='virtio' port='1'/>
      <alias name='ua-c769f8ac-922a-4518-9946-2a44172d7407'/>
    </console>
    <console type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/foo2'/>
      <target type='virtio' port='6'/>
      <alias name='ua-c769f8ac-922a-4518-9946-2a44172d7408'/>
    </console>

# ll -Z /var/lib/libvirt/qemu/foo2;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo2
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 05:05 /var/lib/libvirt/qemu/foo2

# virsh detach-device avocado-vt-vm1 console-unix.xml
Device detached successfully

[guest]# echo abc123 > /dev/hvc1
[host]# nc -U /var/lib/libvirt/qemu/foo2
abc123

# ll -Z /var/lib/libvirt/qemu/foo2;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo2
ls: cannot access '/var/lib/libvirt/qemu/foo2': No such file or directory
getfattr: /var/lib/libvirt/qemu/foo2: No such file or directory

3. restart libvirtd
# systemctl restart libvirtd
# ll -Z /var/lib/libvirt/qemu/foo;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 05:14 /var/lib/libvirt/qemu/foo
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/qemu/foo
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.timestamp_dac="1580718107"

4.shutdown
# virsh shutdown avocado-vt-vm1
Domain avocado-vt-vm1 is being shutdown

# ll -Z /var/lib/libvirt/qemu/foo;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo
ls: cannot access '/var/lib/libvirt/qemu/foo': No such file or directory
getfattr: /var/lib/libvirt/qemu/foo: No such file or directory

Special  need mention:
file will disappear when vm not running.
no xattr for hotplugged one.

S3: Channel: unix bind 
    <channel type='unix'>
     <source mode='bind' path='/var/lib/libvirt/qemu/test.agent'/>
     <target type='virtio' name='unixvirtio' state='disconnected'/>
     <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10007'/>
    </channel>

1. start guest
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# ll -Z /var/lib/libvirt/qemu/test.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test.agent
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 05:32 /var/lib/libvirt/qemu/test.agent
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/qemu/test.agent
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.timestamp_dac="1580718107"

# virsh dumpxml avocado-vt-vm1 |grep 'channel t' -A6
...
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/test.agent'/>
      <target type='virtio' name='unixvirtio' state='disconnected'/>
      <alias name='ua-d830c2c4-93ac-4eb7-b714-593483f10007'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>

[guest]# echo qwer > /dev/vport1p2
[host]# nc -U /var/lib/libvirt/qemu/test.agent
qwer

2. hotplug unplug:
# virsh attach-device avocado-vt-vm1 channel-unix.xml
Device attached successfully

# ll -Z /var/lib/libvirt/qemu/test2.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test2.agent
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 06:08 /var/lib/libvirt/qemu/test2.agent

[guest]# echo qwer2 > /dev/vport1p3
[host]# nc -U /var/lib/libvirt/qemu/test2.agent
qwer2

# virsh detach-device avocado-vt-vm1 channel-unix.xml
Device detached successfully

# ll -Z /var/lib/libvirt/qemu/test2.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test2.agent
ls: cannot access '/var/lib/libvirt/qemu/test2.agent': No such file or directory
getfattr: /var/lib/libvirt/qemu/test2.agent: No such file or directory

3. restart libvirtd
# systemctl restart libvirtd
# ll -Z /var/lib/libvirt/qemu/test.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test.agent
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 05:32 /var/lib/libvirt/qemu/test.agent
getfattr: Removing leading '/' from absolute path names
# file: var/lib/libvirt/qemu/test.agent
trusted.libvirt.security.dac="+0:+0"
trusted.libvirt.security.ref_dac="1"
trusted.libvirt.security.timestamp_dac="1580718107"

4. shutdown 
# virsh shutdown avocado-vt-vm1
Domain avocado-vt-vm1 is being shutdown

# ll -Z /var/lib/libvirt/qemu/test.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test.agent
ls: cannot access '/var/lib/libvirt/qemu/test.agent': No such file or directory
getfattr: /var/lib/libvirt/qemu/test.agent: No such file or directory

Special need mention:
file will disappear when vm not running.
no xattr for hotplugged one.

S4. disable remember_owner
# grep remember_owner /etc/libvirt/qemu.conf
remember_owner = 0
# systemctl restart libvirtd
1. Serial: pty pci-serial start and hotplug functions
start guest:
# ll -Z /dev/pts/2;  getfattr -m trusted.libvirt.security -d /dev/pts/2
crw--w----. 1 qemu tty system_u:object_r:svirt_devpts_t:s0 136, 2 Feb  5 06:18 /dev/pts/2

# cat /dev/pts/2
ABC123

# virsh attach-device avocado-vt-vm1 serial-pty.xml
Device attached successfully
# cat /dev/pts/4
ABC123
^C
# ll -Z /dev/pts/4;  getfattr -m trusted.libvirt.security -d /dev/pts/4
crw--w----. 1 qemu tty system_u:object_r:svirt_devpts_t:s0 136, 4 Feb  5 06:29 /dev/pts/4

2. Console: unix bind start and hotplug functions
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

#  ll -Z /var/lib/libvirt/qemu/foo;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 06:32 /var/lib/libvirt/qemu/foo

#  nc -U /var/lib/libvirt/qemu/foo
hello

# virsh attach-device avocado-vt-vm1 console-unix.xml
Device attached successfully

#  nc -U /var/lib/libvirt/qemu/foo2
hello

#  ll -Z /var/lib/libvirt/qemu/foo2;  getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/foo2
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 06:35 /var/lib/libvirt/qemu/foo2

3. Channel: unix bind start and hotplug functions
# virsh start avocado-vt-vm1
Domain avocado-vt-vm1 started

# ll -Z /var/lib/libvirt/qemu/test.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test.agent
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 06:38 /var/lib/libvirt/qemu/test.agent

# nc -U /var/lib/libvirt/qemu/test.agent
qwer

# virsh attach-device avocado-vt-vm1 channel-unix.xml
Device attached successfully

# ll -Z /var/lib/libvirt/qemu/test2.agent; getfattr -m trusted.libvirt.security -d /var/lib/libvirt/qemu/test2.agent
srwxrwxr-x. 1 qemu qemu system_u:object_r:qemu_var_run_t:s0 0 Feb  5 06:40 /var/lib/libvirt/qemu/test2.agent
# nc -U /var/lib/libvirt/qemu/test2.agent
qwer

Special need mention:
no xattr for files;
function works well.

Comment 7 Yanqiu Zhang 2020-02-06 04:56:59 UTC
Hi Michal,

I have two questions about chardev testing per comment6. Could you help check pls?
When remember_owner is enabled in S1-S3:
1. For pty device in S1, the /dev/pts/* has no xattr result, is it as expected for it?
2. In S2-S3, the 'hotplug'ed chardev files get no xattr result, not align with the 'start with' situation. Is it an issue here or as expected?  (same behaviour as libvirt-5.6.0, not regression by this bug)

Thanks.

Comment 8 Michal Privoznik 2020-02-06 09:58:47 UTC
(In reply to yanqzhan from comment #7)
> Hi Michal,
> 
> I have two questions about chardev testing per comment6. Could you help
> check pls?
> When remember_owner is enabled in S1-S3:
> 1. For pty device in S1, the /dev/pts/* has no xattr result, is it as
> expected for it?

Yes, this is expected; /dev/pts/* files are not created by libvirt nor their seclabels are changed by libvirt.

> 2. In S2-S3, the 'hotplug'ed chardev files get no xattr result, not align
> with the 'start with' situation. Is it an issue here or as expected?  (same
> behaviour as libvirt-5.6.0, not regression by this bug)

Yes, this is expected.

Comment 9 Yanqiu Zhang 2020-02-06 12:20:56 UTC
(In reply to Michal Privoznik from comment #8)

> > 2. In S2-S3, the 'hotplug'ed chardev files get no xattr result, *not* align
> > with the 'start with' situation. Is it an issue here or as expected?  (same
> > behaviour as libvirt-5.6.0, not regression by this bug)
> 
> Yes, this is expected.

Hi Michal,

Could you help explain why?

Thank you.

Comment 10 Michal Privoznik 2020-02-06 16:14:17 UTC
(In reply to yanqzhan from comment #9)
> Could you help explain why?

So I might have misunderstood what you are really asking. I mean, in case of UNIX sockets, if they don't exist at the time libvirt tries to label them, the labelling is skipped and thus no label is remembered. The file doesn't exist so there is nothing to remember anyway. Is this what you are asking?

Comment 11 Yanqiu Zhang 2020-02-07 08:23:02 UTC
(In reply to Michal Privoznik from comment #10)
> (In reply to yanqzhan from comment #9)
> > Could you help explain why?
> 
> So I might have misunderstood what you are really asking. I mean, in case of
> UNIX sockets, if they don't exist at the time libvirt tries to label them,
> the labelling is skipped and thus no label is remembered. The file doesn't
> exist so there is nothing to remember anyway. Is this what you are asking?

Thanks Michal. Then there's still a question: why libvirt can label them when start guest? The file also doesn't exist before guest starting. What's difference here?

Comment 12 Michal Privoznik 2020-02-07 08:45:17 UTC
(In reply to yanqzhan from comment #11)
> (In reply to Michal Privoznik from comment #10)
> > (In reply to yanqzhan from comment #9)
> > > Could you help explain why?
> > 
> > So I might have misunderstood what you are really asking. I mean, in case of
> > UNIX sockets, if they don't exist at the time libvirt tries to label them,
> > the labelling is skipped and thus no label is remembered. The file doesn't
> > exist so there is nothing to remember anyway. Is this what you are asking?
> 
> Thanks Michal. Then there's still a question: why libvirt can label them
> when start guest? The file also doesn't exist before guest starting. What's
> difference here?

Because when starting a domain, libvirt will create the UNIX socket and pass FD to it on the command line. The security label setting is done after that, so the file exists. When hotplugging it, libvirt tries to do seclabel setting (if the file exists upront) and after that it instructs qemu to create the socket. That is why you see two different behaviours.

Comment 13 Yanqiu Zhang 2020-04-20 09:32:42 UTC
On latest build, the console or channel chardev file also has no xattr when remember_owner = 1. FYI.
libvirt-daemon-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64

Comment 15 errata-xmlrpc 2020-05-05 09:50:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.