Bug 1595184

Summary: Libvirtd crashed when domifstat for interface type: user, udp
Product: Red Hat Enterprise Linux 7 Reporter: Fangge Jin <fjin>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.6CC: dyuan, lmen, mprivozn, xuzhang, yafu, yalzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.5.0-3.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1596512 (view as bug list) Environment:
Last Closed: 2018-10-30 09:56:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1596512    
Attachments:
Description Flags
backtrace of libvirtd none

Description Fangge Jin 2018-06-26 10:21:59 UTC
Created attachment 1454615 [details]
backtrace of libvirtd

Description of problem:
Libvirtd crashed when domifstat for interface type: user, udp

Version-Release number of selected component (if applicable):
libvirt-4.4.0-2.virtcov.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a guest with interface of type user/udp
# virsh dumpxml rhel7.5 --inactive
    <interface type='udp'>
      <mac address='52:54:00:22:c9:42'/>
      <source address='10.66.4.140' port='11116'>
        <local address='10.66.5.190' port='11115'/>
      </source>
      <model type='rtl8139'/>
...
    <interface type='user'>
      <mac address='00:11:22:33:44:55'/>
      <ip address='172.17.2.0' family='ipv4' prefix='24'/>
      <ip address='2001:db8:ac10:fd01::' family='ipv6' prefix='64'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>


2.Start guest

3.Get network interface stats by "virsh domifstat":
# virsh domiflist rhel7.5
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     br0        virtio      00:1a:4a:16:01:55
vnet1      bridge     host-bridge rtl8139     52:54:00:0e:24:fe
-          udp        -          rtl8139     52:54:00:22:c9:42
vnet2      network    default    virtio      52:54:00:c6:3b:95
-          user       -          rtl8139     00:11:22:33:44:55

# virsh domifstat rhel7.5 52:54:00:22:c9:42
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 52:54:00:22:c9:42
error: End of file while reading data: Input/output error

# virsh domifstat rhel7.5 00:11:22:33:44:55
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 00:11:22:33:44:55
error: End of file while reading data: Input/output error


Actual results:
Libvirtd crashed when domifstat for interface type: user, udp

Expected results:
No crash

Additional info:

Comment 2 Fangge Jin 2018-06-26 10:22:54 UTC
No crash when domifstat other interface by either interface name or mac addres

Comment 3 Fangge Jin 2018-07-05 07:24:20 UTC
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug

Comment 4 Fangge Jin 2018-07-05 07:24:37 UTC
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug

Comment 5 Michal Privoznik 2018-07-13 09:25:42 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-July/msg00827.html

Comment 8 yalzhang@redhat.com 2018-07-19 07:09:31 UTC
Test on libvirt-4.5.0-3.el7.x86_64, libvirtd do not crash, the issue is fixed.

1. Start the vm with multiple interfaces as below:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
-          user       -          rtl8139     00:11:22:33:44:55
-          udp        -          rtl8139     52:54:00:54:02:e5
-          mcast      -          rtl8139     52:54:00:29:ae:b7
-          server     -          rtl8139     52:54:00:22:c9:42
-          client     -          rtl8139     52:54:00:8b:c9:51

# pidof libvirtd
76350

2. both will report error as below except for the network type:
# virsh domifstat rhel 00:11:22:33:44:55
error: Failed to get interface stats rhel 00:11:22:33:44:55
error: internal error: Interface name not provided

# virsh domifstat rhel 52:54:00:54:02:e5
error: Failed to get interface stats rhel 52:54:00:54:02:e5
error: internal error: Interface name not provided
...

3. destroy the guest and configure the target name for them, then start vm, the error message will change, but libvirt not crash:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
user_target user       -          rtl8139     00:11:22:33:44:55
udp_target udp        -          rtl8139     52:54:00:54:02:e5
mcast_target mcast      -          rtl8139     52:54:00:29:ae:b7
server_target server     -          rtl8139     52:54:00:22:c9:42
client_target client     -          rtl8139     52:54:00:8b:c9:51

# virsh domifstat rhel  mcast_target
error: Failed to get interface stats rhel mcast_target
error: internal error: /proc/net/dev: Interface not found

# virsh domifstat rhel  52:54:00:22:c9:42
error: Failed to get interface stats rhel 52:54:00:22:c9:42
error: internal error: /proc/net/dev: Interface not found

# pidof libvirtd
76350

Comment 9 yalzhang@redhat.com 2018-07-19 08:01:54 UTC
Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to confirm, thank you!

2) And for the error message in comment 4, I think it is not related with this bug. Furthermore, I can not reproduce when restart libvirtd with interfaces in comment 8 or comment 1, so if you agree, I will move this bug to be verified.

3) I found the same error message here: https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html

I'm not sure if that patch is to fix this kind of issue, and I can not tell if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you please help to check? because I can still get the error with below scenario:

Start with non-exits nwfilter(100% reproducible)
1. prepare a guest with interface as below with not exists nwfitler:
 <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
      <filterref filter='no-ip'/>
    </interface>

# virsh start rhel
error: Failed to start domain rhel
error: internal error: referenced filter 'no-ip' is missing

# grep error /var/log/libvirt/libvirtd.log
2018-07-19 07:27:23.139+0000: 79106: error : virNWFilterObjListFindInstantiateFilter:201 : internal error: referenced filter 'no-ip' is missing
2018-07-19 07:27:23.162+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.167+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.169+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.171+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.172+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.173+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.173+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.175+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.177+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.179+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device

As this is a negative scenario, I'm not sure if it is a issue with such error message. Could you please help to confirm? Thank you very much!

Comment 10 Michal Privoznik 2018-07-23 12:01:09 UTC
(In reply to yalzhang@redhat.com from comment #9)
> Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to
> confirm, thank you!

Yes it is.

> 
> 2) And for the error message in comment 4, I think it is not related with
> this bug. Furthermore, I can not reproduce when restart libvirtd with
> interfaces in comment 8 or comment 1, so if you agree, I will move this bug
> to be verified.

Yes, it is unrelated. I'm trying to fix it upstream but it is different issue.

> 
> 3) I found the same error message here:
> https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html
> 
> I'm not sure if that patch is to fix this kind of issue, and I can not tell
> if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you
> please help to check? because I can still get the error with below scenario:
> 

This is a different issue (the same as in 2) and should be tracked separately.

Comment 11 yalzhang@redhat.com 2018-07-24 11:19:49 UTC
file Bug 1607831 to track the issue in comment 10 and move this one into verified.

Comment 13 errata-xmlrpc 2018-10-30 09:56:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113