Bug 1741390

Summary: Exclusive check for direct type passthrough mode interface will crash the libvirtd
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: yalzhang <yalzhang>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.1CC: jdenemar, jtomko, laine, xuzhang
Target Milestone: rcKeywords: Regression
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-5.6.0-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-11-06 07:18:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1585087    
Attachments:
Description Flags
backtrace for libvirtd crash
none
backtrace, output of br fu none

Description yalzhang@redhat.com 2019-08-15 02:36:13 UTC
Description of problem:
Exclusive check for direct type passthrough mode interface will crash the libvirtd 

Version-Release number of selected component (if applicable):
libvirt-5.6.0-1.module+el8.1.0+3890+4d3d259c.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a network with direct type passthrough mode with only 1 interface device:
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26'/>
  </forward>
</network>

2. hotplug interface to a vm:
# pidof libvirtd
30149
# virsh attach-interface rhel network direct-macvtap
Interface attached successfully

# virsh net-dumpxml direct-macvtap
<network connections='1'>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26' connections='1'/>
  </forward>
</network>

3. hotplug the 2nd time, it will cause libvirtd crash:
# virsh attach-interface rhel network direct-macvtap
error: Disconnected from qemu:///system due to end of file
error: Failed to attach interface
error: End of file while reading data: Input/output error

# pidof libvirtd
30312

# grep error /var/log/libvirt/libvirtd.log
2019-08-15 02:23:39.227+0000: 30529: error : networkAllocatePort:4780 : internal error: network 'direct-macvtap' requires exclusive access to interfaces, but none are available

Actual results:
step 3, libvirtd crash

Expected results:
libvirtd should not crash in step 3, it should report error when hotplug the 2nd time like:
"error: Failed to attach interface
error: internal error: network 'direct-macvtap' requires exclusive access to interfaces, but none are available"

Additional info:
After the crash, destroy the vm and start it again, check the status:
# virsh destroy rhel; virsh start rhel
Domain rhel destroyed

Domain rhel started
1.  No interface exists in the guest xml, and there is no connection in the network xml, this is expected:
# virsh dumpxml rhel | grep /interface
# (no output)
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>d618b264-945d-4e36-99a5-e6c1abef0eba</uuid>
  <forward dev='eno26' mode='passthrough'>
    <interface dev='eno26'/>
  </forward>
</network>
# ifconfig  -a  | grep macvtap
# (no output)

2. The connection should be released, and hotplug the first time should succeed, but it crash the libvirtd again:
# virsh attach-interface rhel network direct-macvtap
error: Disconnected from qemu:///system due to end of file
error: Failed to attach interface
error: End of file while reading data: Input/output error

Comment 1 Ján Tomko 2019-08-15 11:20:49 UTC
Please include a backtrace for crash reports:
https://libvirt.org/bugs.html#quality

Comment 2 yalzhang@redhat.com 2019-08-16 02:03:17 UTC
Created attachment 1604255 [details]
backtrace for libvirtd crash

Comment 3 yalzhang@redhat.com 2019-08-16 02:06:29 UTC
Created attachment 1604256 [details]
backtrace, output of br fu

Comment 4 Laine Stump 2019-08-16 02:42:49 UTC
Fix posted upstream:

https://www.redhat.com/archives/libvir-list/2019-August/msg00589.html

Comment 5 Laine Stump 2019-08-17 00:15:20 UTC
These two patches were pushed upstream. The first is necessary for the bugfix. The second is a small change that was made at the same time that will make any future patches to the same code easier to backport.

commit dac697e8d7d6d9a607e61caeeec06b259edf513f
Author: Laine Stump <laine>
Date:   Thu Aug 15 21:52:28 2019 -0400

    network: fix crash during cleanup from failure to allocate port
    
commit 39de732aa728ae9b8a9414ad08b8d0ee7ed02732
origin/HEAD)
Author: Laine Stump <laine>
Date:   Thu Aug 15 22:28:27 2019 -0400

    network: replace virSaveLastError() with virErrorPreserveLast()

Comment 8 yalzhang@redhat.com 2019-08-22 03:03:43 UTC
Test on libvirt-5.6.0-2.virtcov.el8.x86_64, the result is as expected except the network connection will clear after libvirtd restart, please help to check, Thank you!

1.
# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

# virsh attach-interface rhel network direct-macvtap --model virtio
Interface attached successfully

# virsh net-dumpxml direct-macvtap
<network connections='1'>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2' connections='1'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

2. Restart libvirtd will clear current the connection:
# systemctl restart libvirtd

# virsh net-dumpxml direct-macvtap
<network>
  <name>direct-macvtap</name>
  <uuid>3d3376a1-77b5-49e9-bbf1-c2c268fa8e02</uuid>
  <forward dev='eno27v2' mode='passthrough'>
    <interface dev='eno27v2'/>
    <interface dev='eno27v1'/>
  </forward>
</network>

# virsh attach-interface rhel network direct-macvtap --model virtio
error: Failed to attach interface
error: error creating macvtap interface macvtap1@eno27v2 (52:54:00:5e:ac:45): Invalid argument

Comment 9 Laine Stump 2019-08-26 04:24:07 UTC
Loss of the connection counts during libvirtd restart is also the result of the refactoring that added the virNetworkPort APIs, but it is a separate Bug. Please file a BZ for this (also check the connection count for other types of network to see if they are affected similarly).

Your testing does show that *this* BZ has been fixed though, so it can be moved to VERIFIED.

Comment 10 yalzhang@redhat.com 2019-08-27 02:28:36 UTC
Hi Laine, Thank you! File Bug 1745815 and set this one to be verified.

Comment 12 errata-xmlrpc 2019-11-06 07:18:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:3723