Bug 1741390 - Exclusive check for direct type passthrough mode interface will crash the libvirtd
Summary: Exclusive check for direct type passthrough mode interface will crash the lib...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.1
Assignee: Laine Stump
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1585087
TreeView+ depends on / blocked
 
Reported: 2019-08-15 02:36 UTC by yalzhang@redhat.com
Modified: 2020-11-14 05:29 UTC (History)
4 users (show)

Fixed In Version: libvirt-5.6.0-2.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-06 07:18:29 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
backtrace for libvirtd crash (63.75 KB, text/plain)
2019-08-16 02:03 UTC, yalzhang@redhat.com
no flags Details
backtrace, output of br fu (9.38 KB, text/plain)
2019-08-16 02:06 UTC, yalzhang@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:3723 0 None None None 2019-11-06 07:18:50 UTC

Description yalzhang@redhat.com 2019-08-15 02:36:13 UTC
Description of problem:
Exclusive check for direct type passthrough mode interface will crash the libvirtd 

Version-Release number of selected component (if applicable):
libvirt-5.6.0-1.module+el8.1.0+3890+4d3d259c.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a network with direct type passthrough mode with only 1 interface device:
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26'/>
  </forward>
</network>

2. hotplug interface to a vm:
# pidof libvirtd
30149
# virsh attach-interface rhel network direct-macvtap
Interface attached successfully

# virsh net-dumpxml direct-macvtap
<network connections='1'>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26' connections='1'/>
  </forward>
</network>

3. hotplug the 2nd time, it will cause libvirtd crash:
# virsh attach-interface rhel network direct-macvtap
error: Disconnected from qemu:///system due to end of file
error: Failed to attach interface
error: End of file while reading data: Input/output error

# pidof libvirtd
30312

# grep error /var/log/libvirt/libvirtd.log
2019-08-15 02:23:39.227+0000: 30529: error : networkAllocatePort:4780 : internal error: network 'direct-macvtap' requires exclusive access to interfaces, but none are available

Actual results:
step 3, libvirtd crash

Expected results:
libvirtd should not crash in step 3, it should report error when hotplug the 2nd time like:
"error: Failed to attach interface
error: internal error: network 'direct-macvtap' requires exclusive access to interfaces, but none are available"

Additional info:
After the crash, destroy the vm and start it again, check the status:
# virsh destroy rhel; virsh start rhel
Domain rhel destroyed

Domain rhel started
1.  No interface exists in the guest xml, and there is no connection in the network xml, this is expected:
# virsh dumpxml rhel | grep /interface
# (no output)
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26'/>
  </forward>
</network>
# ifconfig  -a  | grep macvtap
# (no output)

2. The connection should be released, and hotplug the first time should succeed, but it crash the libvirtd again:
# virsh attach-interface rhel network direct-macvtap
error: Disconnected from qemu:///system due to end of file
error: Failed to attach interface
error: End of file while reading data: Input/output error

Comment 1 Ján Tomko 2019-08-15 11:20:49 UTC
Please include a backtrace for crash reports:
https://libvirt.org/bugs.html#quality

Comment 2 yalzhang@redhat.com 2019-08-16 02:03:17 UTC
Created attachment 1604255 [details]
backtrace for libvirtd crash

Comment 3 yalzhang@redhat.com 2019-08-16 02:06:29 UTC
Created attachment 1604256 [details]
backtrace, output of br fu

Comment 4 Laine Stump 2019-08-16 02:42:49 UTC
Fix posted upstream:

https://www.redhat.com/archives/libvir-list/2019-August/msg00589.html

Comment 5 Laine Stump 2019-08-17 00:15:20 UTC
These two patches were pushed upstream. The first is necessary for the bugfix. The second is a small change that was made at the same time that will make any future patches to the same code easier to backport.

commit dac697e8d7d6d9a607e61caeeec06b259edf513f
Author: Laine Stump <laine>
Date:   Thu Aug 15 21:52:28 2019 -0400

    network: fix crash during cleanup from failure to allocate port
    
commit 39de732aa728ae9b8a9414ad08b8d0ee7ed02732
origin/HEAD)
Author: Laine Stump <laine>
Date:   Thu Aug 15 22:28:27 2019 -0400

    network: replace virSaveLastError() with virErrorPreserveLast()

Comment 8 yalzhang@redhat.com 2019-08-22 03:03:43 UTC
Test on libvirt-5.6.0-2.virtcov.el8.x86_64, the result is as expected except the network connection will clear after libvirtd restart, please help to check, Thank you!

1.
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

# virsh attach-interface rhel network direct-macvtap --model virtio
Interface attached successfully

# virsh net-dumpxml direct-macvtap
<network connections='1'>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2' connections='1'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

2. Restart libvirtd will clear current the connection:
# systemctl restart libvirtd

# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

# virsh attach-interface rhel network direct-macvtap --model virtio
error: Failed to attach interface
error: error creating macvtap interface macvtap1@eno27v2 (52:54:00:5e:ac:45): Invalid argument

Comment 9 Laine Stump 2019-08-26 04:24:07 UTC
Loss of the connection counts during libvirtd restart is also the result of the refactoring that added the virNetworkPort APIs, but it is a separate Bug. Please file a BZ for this (also check the connection count for other types of network to see if they are affected similarly).

Your testing does show that *this* BZ has been fixed though, so it can be moved to VERIFIED.

Comment 10 yalzhang@redhat.com 2019-08-27 02:28:36 UTC
Hi Laine, Thank you! File Bug 1745815 and set this one to be verified.

Comment 12 errata-xmlrpc 2019-11-06 07:18:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723


Note You need to log in before you can comment on or make changes to this bug.