Bug 1595184 - Libvirtd crashed when domifstat for interface type: user, udp
Summary: Libvirtd crashed when domifstat for interface type: user, udp
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1596512
TreeView+ depends on / blocked
 
Reported: 2018-06-26 10:21 UTC by Fangge Jin
Modified: 2018-10-30 09:57 UTC (History)
6 users (show)

Fixed In Version: libvirt-4.5.0-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1596512 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:56:58 UTC
Target Upstream Version:


Attachments (Terms of Use)
backtrace of libvirtd (14.25 KB, text/plain)
2018-06-26 10:21 UTC, Fangge Jin
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 None None None 2018-10-30 09:57:59 UTC

Description Fangge Jin 2018-06-26 10:21:59 UTC
Created attachment 1454615 [details]
backtrace of libvirtd

Description of problem:
Libvirtd crashed when domifstat for interface type: user, udp

Version-Release number of selected component (if applicable):
libvirt-4.4.0-2.virtcov.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a guest with interface of type user/udp
# virsh dumpxml rhel7.5 --inactive
    <interface type='udp'>
      <mac address='52:54:00:22:c9:42'/>
      <source address='10.66.4.140' port='11116'>
        <local address='10.66.5.190' port='11115'/>
      </source>
      <model type='rtl8139'/>
...
    <interface type='user'>
      <mac address='00:11:22:33:44:55'/>
      <ip address='172.17.2.0' family='ipv4' prefix='24'/>
      <ip address='2001:db8:ac10:fd01::' family='ipv6' prefix='64'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>


2.Start guest

3.Get network interface stats by "virsh domifstat":
# virsh domiflist rhel7.5
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     br0        virtio      00:1a:4a:16:01:55
vnet1      bridge     host-bridge rtl8139     52:54:00:0e:24:fe
-          udp        -          rtl8139     52:54:00:22:c9:42
vnet2      network    default    virtio      52:54:00:c6:3b:95
-          user       -          rtl8139     00:11:22:33:44:55

# virsh domifstat rhel7.5 52:54:00:22:c9:42
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 52:54:00:22:c9:42
error: End of file while reading data: Input/output error

# virsh domifstat rhel7.5 00:11:22:33:44:55
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 00:11:22:33:44:55
error: End of file while reading data: Input/output error


Actual results:
Libvirtd crashed when domifstat for interface type: user, udp

Expected results:
No crash

Additional info:

Comment 2 Fangge Jin 2018-06-26 10:22:54 UTC
No crash when domifstat other interface by either interface name or mac addres

Comment 3 Fangge Jin 2018-07-05 07:24:20 UTC
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug

Comment 4 Fangge Jin 2018-07-05 07:24:37 UTC
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug

Comment 5 Michal Privoznik 2018-07-13 09:25:42 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-July/msg00827.html

Comment 8 yalzhang@redhat.com 2018-07-19 07:09:31 UTC
Test on libvirt-4.5.0-3.el7.x86_64, libvirtd do not crash, the issue is fixed.

1. Start the vm with multiple interfaces as below:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
-          user       -          rtl8139     00:11:22:33:44:55
-          udp        -          rtl8139     52:54:00:54:02:e5
-          mcast      -          rtl8139     52:54:00:29:ae:b7
-          server     -          rtl8139     52:54:00:22:c9:42
-          client     -          rtl8139     52:54:00:8b:c9:51

# pidof libvirtd
76350

2. both will report error as below except for the network type:
# virsh domifstat rhel 00:11:22:33:44:55
error: Failed to get interface stats rhel 00:11:22:33:44:55
error: internal error: Interface name not provided

# virsh domifstat rhel 52:54:00:54:02:e5
error: Failed to get interface stats rhel 52:54:00:54:02:e5
error: internal error: Interface name not provided
...

3. destroy the guest and configure the target name for them, then start vm, the error message will change, but libvirt not crash:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
user_target user       -          rtl8139     00:11:22:33:44:55
udp_target udp        -          rtl8139     52:54:00:54:02:e5
mcast_target mcast      -          rtl8139     52:54:00:29:ae:b7
server_target server     -          rtl8139     52:54:00:22:c9:42
client_target client     -          rtl8139     52:54:00:8b:c9:51

# virsh domifstat rhel  mcast_target
error: Failed to get interface stats rhel mcast_target
error: internal error: /proc/net/dev: Interface not found

# virsh domifstat rhel  52:54:00:22:c9:42
error: Failed to get interface stats rhel 52:54:00:22:c9:42
error: internal error: /proc/net/dev: Interface not found

# pidof libvirtd
76350

Comment 9 yalzhang@redhat.com 2018-07-19 08:01:54 UTC
Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to confirm, thank you!

2) And for the error message in comment 4, I think it is not related with this bug. Furthermore, I can not reproduce when restart libvirtd with interfaces in comment 8 or comment 1, so if you agree, I will move this bug to be verified.

3) I found the same error message here: https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html

I'm not sure if that patch is to fix this kind of issue, and I can not tell if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you please help to check? because I can still get the error with below scenario:

Start with non-exits nwfilter(100% reproducible)
1. prepare a guest with interface as below with not exists nwfitler:
 <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
      <filterref filter='no-ip'/>
    </interface>

# virsh start rhel
error: Failed to start domain rhel
error: internal error: referenced filter 'no-ip' is missing

# grep error /var/log/libvirt/libvirtd.log
2018-07-19 07:27:23.139+0000: 79106: error : virNWFilterObjListFindInstantiateFilter:201 : internal error: referenced filter 'no-ip' is missing
2018-07-19 07:27:23.162+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.167+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.169+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.171+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.172+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.173+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.173+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.175+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.177+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.179+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device

As this is a negative scenario, I'm not sure if it is a issue with such error message. Could you please help to confirm? Thank you very much!

Comment 10 Michal Privoznik 2018-07-23 12:01:09 UTC
(In reply to yalzhang@redhat.com from comment #9)
> Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to
> confirm, thank you!

Yes it is.

> 
> 2) And for the error message in comment 4, I think it is not related with
> this bug. Furthermore, I can not reproduce when restart libvirtd with
> interfaces in comment 8 or comment 1, so if you agree, I will move this bug
> to be verified.

Yes, it is unrelated. I'm trying to fix it upstream but it is different issue.

> 
> 3) I found the same error message here:
> https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html
> 
> I'm not sure if that patch is to fix this kind of issue, and I can not tell
> if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you
> please help to check? because I can still get the error with below scenario:
> 

This is a different issue (the same as in 2) and should be tracked separately.

Comment 11 yalzhang@redhat.com 2018-07-24 11:19:49 UTC
file Bug 1607831 to track the issue in comment 10 and move this one into verified.

Comment 13 errata-xmlrpc 2018-10-30 09:56:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113


Note You need to log in before you can comment on or make changes to this bug.