RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2169244 - Guest start successfully with passt backend interface even the passt process failed to start
Summary: Guest start successfully with passt backend interface even the passt process ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: libvirt
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
: 2178553 (view as bug list)
Depends On:
Blocks: 2176921
TreeView+ depends on / blocked
 
Reported: 2023-02-13 02:34 UTC by yalzhang@redhat.com
Modified: 2023-11-20 03:16 UTC (History)
11 users (show)

Fixed In Version: libvirt-9.0.0-8.el9_2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2176921 (view as bug list)
Environment:
Last Closed: 2023-05-09 07:27:45 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker LIBVIRTAT-14283 0 None None None 2023-05-31 01:46:02 UTC
Red Hat Issue Tracker RHELPLAN-148375 0 None None None 2023-02-13 02:35:57 UTC
Red Hat Product Errata RHBA-2023:2171 0 None None None 2023-05-09 07:28:15 UTC

Description yalzhang@redhat.com 2023-02-13 02:34:45 UTC 
Description of problem:
If passt encounters a fatal error during startup(failed to start), libvirt doesn't see this and starts up the guest with passt backend interface anyway

Version-Release number of selected component (if applicable):
libvirt-9.0.0-4.el9.x86_64
qemu-kvm-7.2.0-8.el9.x86_64
passt-0^20221110.g4129764-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. There are various situations when passt process can not run successfully. 
Just show 1 scenario here when no permission to create the log file:
$ whoami
test
$ /usr/bin/passt --one-off --socket /home/test/test.socket --log-file /var/log/passt.log
Couldn't open log file /var/log/passt.log: Permission denied

2. Prepare a vm with interface set with below:
$ virsh dumpxml test --xpath //interface
<interface type="user">
  <mac address="52:54:00:bc:fa:5e"/>
  <source dev="eno1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/var/log/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</interface>

Start the vm, it start successfully without passt process:
$ virsh start test
Domain 'test' started
$ pidof passt
(nothing)

It is because the passt process start failed with cmd line like this: 
$ /usr/bin/passt --one-off --socket /home/test/.cache/libvirt/qemu/run/passt/*-test-net0.socket --mac-addr 52:54:00:bc:fa:5e --interface eno1 --log-file /var/log/passt.log

There is no error info in the guest's log.

Actual results:
Guest start successfully with passt backend interface even the pass process failed to start

Expected results:
Guest should fail to start with some error info

Additional info:
Comment 1 Laine Stump 2023-02-23 04:23:32 UTC 
The (biggest) problem is that the original libvirt code deamonized the passt process as it was exec'ing, and so it wouldn't see any of the startup errors. The following 6 upstream libvirt commits should solve this problem (and some others):

commit 598a73335d70b4ef70b84f9730d708c116f88b15
Author: Michal Prívozník <mprivozn>
Date:   Mon Feb 13 16:05:04 2023 +0100

    qemu_passt: Report error when getting passt PID failed
    
commit c0efdbdb9f66ab5a7334fd1dc75cdfdc28a7393d
Author: Michal Prívozník <mprivozn>
Date:   Thu Feb 16 12:00:58 2023 +0100

    qemu_passt: Avoid double daemonizing passt
    
commit 02355840ced2af18df1aa9ba387a6137a515eede
Author: Michal Prívozník <mprivozn>
Date:   Thu Feb 16 12:19:26 2023 +0100

    qemu_passt: Report passt's error on failed start
    
commit e5bfc661bc181a36fa70250470554b20002fb84d
Author: Michal Prívozník <mprivozn>
Date:   Thu Feb 16 12:07:42 2023 +0100

    qemu_passt: Deduplicate passt killing code
    
commit 029a892abdb2fe508f3fb77af00a14464b98b824
Author: Michal Prívozník <mprivozn>
Date:   Thu Feb 16 11:46:55 2023 +0100

    qemu_passt: Let passt write the PID file
    
commit b7a18787de7c193dc68640668cc176ef9d887bc1
Author: Stefano Brivio <sbrivio>
Date:   Tue Feb 21 20:19:07 2023 +0100

    qemu_passt: Remove passt socket file on exit
Comment 2 Laine Stump 2023-02-23 18:55:34 UTC 
The above patches are a part of this MR that was filed against the rhel-9.2.0 branch:

  https://gitlab.com/redhat/rhel/src/libvirt/-/merge_requests/98
Comment 7 yalzhang@redhat.com 2023-03-07 02:53:25 UTC 
Test on scratch build libvirt-9.0.0-8.el9_rc.a7213e6de2.x86_64, the result is as expected.

1. No permission to create the log file:
$ whoami
test
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/var/log/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': libvirt:  error : cannot execute binary /usr/bin/passt: Permission denied
$ pidof passt
(nothing)

2. No passt package installed
$ rpm -q passt
package passt is not installed
$ getenforce 
Permissive
$ virsh dumpxml vm1 --xpath //interface
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': libvirt:  error : cannot execute binary /usr/bin/passt: No such file or directory

3. The interface do not exists on the host
$ ip l show eth0
Device "eth0" does not exist.
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eth0"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Invalid interface name eth0: No such device

4. listen on inactive interface
$ ip l show ens1f1
7: ens1f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:21:ce:a2 brd ff:ff:ff:ff:ff:ff
    altname enp59s0f1
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="ens1f1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': External interface not usable
$ cat /home/test/passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59271)
0.0678: ERROR:   External interface not usable

5. Start vm with passt backend interface, selinux is enforce(this will be fixed)
$ getenforce 
Enforcing
$ virsh start vm1
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': libvirt:  error : cannot execute binary /usr/bin/passt: Permission denied
Comment 8 yalzhang@redhat.com 2023-03-07 03:50:55 UTC 
continue with more scenarios, the result is as expected, all tests test on
passt-0^20230227.gc538ee8-1.el9.x86_64
qemu-kvm-7.2.0-9.el9.x86_64
libvirt-9.0.0-8.el9_rc.a7213e6de2.x86_64

6. the bind ip do not exists on host
$ ip addr show | grep "192.168.200.3" 
(nothing)
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp" address="192.168.200.3">
    <range start="9000"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 192.168.200.3/9000', exiting
$ cat passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59502)
0.2276: ERROR:   Failed to bind any port for '-t 192.168.200.3/9000', exiting

7. When the port is occupied
# ss -tlnpen 
LISTEN 0      64           0.0.0.0:35469      0.0.0.0:*    ino:47373 sk:2 cgroup:unreachable:12d4 <->
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp">
    <range start="35469"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 35469', exiting

$ cat passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59616)
0.4578: ERROR:   Failed to bind any port for '-t 35469', exiting

8. Use port below 1024
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp">
    <range start="200"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 200', exiting

$ cat passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59683)
0.0109: ERROR:   Failed to bind any port for '-t 200', exiting

9. Port number out of range
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp">
    <range start="65536"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Invalid port specifier 65536
$ cat passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59736)
0.6895: ERROR:   Invalid port specifier 65536

10. port overlap:
$ virsh dumpxml vm1 --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp">
    <range start="8000" end="9000"/>
  </portForward>
  <portForward proto="tcp">
    <range start="8001"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/home/test/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start vm1 
error: Failed to start domain 'vm1'
error: internal error: Could not start 'passt': Overlapping port specifier 8001
$ cat passt.log 
passt 0^20230227.gc538ee8-1.el9.x86_64: /usr/bin/passt.avx2 (59776)
0.1273: ERROR:   Overlapping port specifier 8001
Comment 14 Michal Privoznik 2023-03-23 13:57:46 UTC 
*** Bug 2178553 has been marked as a duplicate of this bug. ***
Comment 15 yalzhang@redhat.com 2023-03-26 02:31:24 UTC 
Test with libvirt-9.0.0-10.el9_2.x86_64 with selinux enabled, the result is as expected.

Try to start vm with various configurations:
1. No permission to create the log file:
$ id
uid=1000(test) gid=1000(test) groups=1000(test) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
$ virsh dumpxml rhel --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/var/log/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Couldn't open log file /var/log/passt.log: Permission denied
$ pidof passt
(nothing)

2. No passt package installed
$ rpm -q passt
package passt is not installed
$ virsh dumpxml rhel --xpath //interface
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <model type="virtio"/>
  <backend type="passt"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel
error: Failed to start domain 'rhel'
error: unable to get SELinux context for '/usr/bin/passt': No such file or directory

3. The interface do not exists on the host
$ ip l show eth0
Device "eth0" does not exist.
$ virsh dumpxml rhel --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eth0"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/run/user/1000/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Invalid interface name eth0: No such device

4. listen on inactive interface
$ ip l show ens1f1
7: ens1f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether f8:f2:1e:21:ce:a2 brd ff:ff:ff:ff:ff:ff
    altname enp59s0f1
$ virsh dumpxml rhel --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="ens1f1"/>
  <model type="virtio"/>
  <backend type="passt" logFile="/run/user/1000/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': External interface not usable
$ cat /run/user/1000/passt.log 
passt 0^20230222.g4ddbcb9-2.el9_2.x86_64: /usr/bin/passt.avx2 (23357)
0.1768: ERROR:   External interface not usable

5. the bind ip do not exists on host
$ ip addr show | grep "192.168.200.3" 
(nothing)
$ virsh dumpxml rhel --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp" address="192.168.200.3">
    <range start="9000"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/run/user/1000/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 192.168.200.3/9000', exiting
$ cat /run/user/1000/passt.log 
passt 0^20230222.g4ddbcb9-2.el9_2.x86_64: /usr/bin/passt.avx2 (23445)
0.1769: ERROR:   Failed to bind any port for '-t 192.168.200.3/9000', exiting

6. When the port is occupied
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 6010', exiting
$ cat /run/user/1000/passt.log 
passt 0^20230222.g4ddbcb9-2.el9_2.x86_64: /usr/bin/passt.avx2 (23723)
0.0716: ERROR:   Failed to bind any port for '-t 6010', exiting

7. Use port below 1024
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Failed to bind any port for '-t 1000', exiting
$ cat /run/user/1000/passt.log 
passt 0^20230222.g4ddbcb9-2.el9_2.x86_64: /usr/bin/passt.avx2 (23762)
0.2828: ERROR:   Failed to bind any port for '-t 1000', exiting

8. Port number out of range
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Invalid port specifier 65536

9. port overlap:
$ virsh dumpxml rhel --xpath //interface 
<interface type="user">
  <mac address="52:54:00:4e:92:81"/>
  <source dev="eno1"/>
  <portForward proto="tcp">
    <range start="8000" end="9000"/>
  </portForward>
  <portForward proto="tcp">
    <range start="8001"/>
  </portForward>
  <model type="virtio"/>
  <backend type="passt" logFile="/run/user/1000/passt.log"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
$ virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Overlapping port specifier 8001
$ cat /run/user/1000/passt.log 
passt 0^20230222.g4ddbcb9-2.el9_2.x86_64: /usr/bin/passt.avx2 (23878)
0.2953: ERROR:   Overlapping port specifier 8001
Comment 16 yalzhang@redhat.com 2023-03-26 02:47:53 UTC 
But for hotplug, the error msg is not so detailed, which is discribed in bz2178553. The fix is inclued in libvirt-9.1.0 release according to 2178553#c1.
@Michal, do we need to backport commit 02355840ced into rhel9.2 via this bug?
Comment 17 yalzhang@redhat.com 2023-03-26 05:13:54 UTC 
And when start vm With root user, the error msg will include 2 lines of debug info, is that acceptable?
S1. No permission to create the log file:
# virsh start rhel 
error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': 2023-03-26 04:56:30.230+0000: 24398: debug : virExec:867 : Setting child security label to system_u:system_r:passt_t:s0:c34,c430
2023-03-26 04:56:30.230+0000: 24398: debug : virExecCommon:463 : Setting child uid:gid to 107:107 with caps 0
Couldn't open log file /var/log/passt.log: Permission denied
Comment 18 Michal Privoznik 2023-03-27 13:40:17 UTC 
(In reply to yalzhang from comment #16)
> But for hotplug, the error msg is not so detailed, which is discribed in
> bz2178553.

What do you mean? I haven't found any sings of that. I mean, the error during hotplug is (at least per bug 2178553):

  Couldn't open log file /home/test2/passt.log: No such file or directory

which is true. The XML was specifically crafted so.

> The fix is inclued in libvirt-9.1.0 release according to
> 2178553#c1.
> @Michal, do we need to backport commit 02355840ced into rhel9.2 via this bug?

As I noted in bug 2178553#c2 the commit is already being bacported (see link in comment 2). What else would you like to be backported?
Comment 19 yalzhang@redhat.com 2023-03-28 07:07:45 UTC 
Hi Michal, sorry i didn't make it clear. 
Current libvirt version for rhel 9.2 is libvirt-9.0.0-10.el9_2.x86_64. The fix for bz2178553 is in libvirt-9.1.0.
And I don't think we can get libvirt-9.1.0 for rhel 9.2 release. So I‘m asking if we will backport the fix for bz2178553 into rhel 9.2.
I think it's Okay not to do the backport since it's minor fix about error msg， just double confirm it.
(Please ignore comment 17, since it seems has been fixed and I can not reproduce it now with latest version.)

With libvirt-9.0.0-10.el9_2.x86_64(9.2 latest version), when hotplug a passt backend interface with invalid setting, the error msg is not so clear as start vm.
Hotplug, the error msg is not clear enough:
1. when no permission to create the log file:
# virsh attach-device rhel interface.xml 
error: Failed to attach device from interface.xml
error: internal error: Failed to start passt

Start vm, the error msg is good enough:
while start with it, the error msg is good enough:
# virsh attach-device rhel interface.xml  --config ; virsh destroy rhel; virsh start rhel 
Device attached successfully

Domain 'rhel' destroyed

error: Failed to start domain 'rhel'
error: internal error: Could not start 'passt': Couldn't open log file /var/log/passt.log: Permission denied
Comment 22 yalzhang@redhat.com 2023-03-28 12:36:34 UTC 
Test with below packages with root user, and selinux enabled, with the steps in comment 15, the result is expected.
# rpm -q libvirt passt selinux-policy
libvirt-9.0.0-10.el9_2.x86_64
passt-0^20230222.g4ddbcb9-2.el9_2.x86_64
selinux-policy-38.1.11-1.el9_2.noarch
Comment 24 errata-xmlrpc 2023-05-09 07:27:45 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (libvirt bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2171
Note You need to log in before you can comment on or make changes to this bug.