RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1595184 - Libvirtd crashed when domifstat for interface type: user, udp
Summary: Libvirtd crashed when domifstat for interface type: user, udp
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: yalzhang@redhat.com
URL:
Whiteboard:
Depends On:
Blocks: 1596512
TreeView+ depends on / blocked
 
Reported: 2018-06-26 10:21 UTC by Fangge Jin
Modified: 2018-10-30 09:57 UTC (History)
6 users (show)

Fixed In Version: libvirt-4.5.0-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1596512 (view as bug list)
Environment:
Last Closed: 2018-10-30 09:56:58 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
backtrace of libvirtd (14.25 KB, text/plain)
2018-06-26 10:21 UTC, Fangge Jin 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:3113 0 None None None 2018-10-30 09:57:59 UTC

Description Fangge Jin 2018-06-26 10:21:59 UTC 
Created attachment 1454615 [details]
backtrace of libvirtd

Description of problem:
Libvirtd crashed when domifstat for interface type: user, udp

Version-Release number of selected component (if applicable):
libvirt-4.4.0-2.virtcov.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Prepare a guest with interface of type user/udp
# virsh dumpxml rhel7.5 --inactive
    <interface type='udp'>
      <mac address='52:54:00:22:c9:42'/>
      <source address='10.66.4.140' port='11116'>
        <local address='10.66.5.190' port='11115'/>
      </source>
      <model type='rtl8139'/>
...
    <interface type='user'>
      <mac address='00:11:22:33:44:55'/>
      <ip address='172.17.2.0' family='ipv4' prefix='24'/>
      <ip address='2001:db8:ac10:fd01::' family='ipv6' prefix='64'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </interface>


2.Start guest

3.Get network interface stats by "virsh domifstat":
# virsh domiflist rhel7.5
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      bridge     br0        virtio      00:1a:4a:16:01:55
vnet1      bridge     host-bridge rtl8139     52:54:00:0e:24:fe
-          udp        -          rtl8139     52:54:00:22:c9:42
vnet2      network    default    virtio      52:54:00:c6:3b:95
-          user       -          rtl8139     00:11:22:33:44:55

# virsh domifstat rhel7.5 52:54:00:22:c9:42
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 52:54:00:22:c9:42
error: End of file while reading data: Input/output error

# virsh domifstat rhel7.5 00:11:22:33:44:55
error: Disconnected from qemu:///system due to end of file
error: Failed to get interface stats rhel7.5 00:11:22:33:44:55
error: End of file while reading data: Input/output error


Actual results:
Libvirtd crashed when domifstat for interface type: user, udp

Expected results:
No crash

Additional info:
Comment 2 Fangge Jin 2018-06-26 10:22:54 UTC 
No crash when domifstat other interface by either interface name or mac addres
Comment 3 Fangge Jin 2018-07-05 07:24:20 UTC 
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug
Comment 4 Fangge Jin 2018-07-05 07:24:37 UTC 
When restart libvirtd with such interface, there is error in libvirtd.log:
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-04 10:04:48.870+0000: 16277: debug : virNWFilterBindingLookupByPortDev:584 : conn=0x7f5d00023de0, name=<null>
2018-07-04 10:04:48.870+0000: 16277: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL

The root cause for this may be same, please double confirm when verifying this bug
Comment 5 Michal Privoznik 2018-07-13 09:25:42 UTC 
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2018-July/msg00827.html
Comment 6 Michal Privoznik 2018-07-13 13:45:05 UTC 
To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-July/msg00342.html
Comment 8 yalzhang@redhat.com 2018-07-19 07:09:31 UTC 
Test on libvirt-4.5.0-3.el7.x86_64, libvirtd do not crash, the issue is fixed.

1. Start the vm with multiple interfaces as below:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
-          user       -          rtl8139     00:11:22:33:44:55
-          udp        -          rtl8139     52:54:00:54:02:e5
-          mcast      -          rtl8139     52:54:00:29:ae:b7
-          server     -          rtl8139     52:54:00:22:c9:42
-          client     -          rtl8139     52:54:00:8b:c9:51

# pidof libvirtd
76350

2. both will report error as below except for the network type:
# virsh domifstat rhel 00:11:22:33:44:55
error: Failed to get interface stats rhel 00:11:22:33:44:55
error: internal error: Interface name not provided

# virsh domifstat rhel 52:54:00:54:02:e5
error: Failed to get interface stats rhel 52:54:00:54:02:e5
error: internal error: Interface name not provided
...

3. destroy the guest and configure the target name for them, then start vm, the error message will change, but libvirt not crash:
# virsh domiflist rhel
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vnet0      network    default    virtio      52:54:00:49:92:db
user_target user       -          rtl8139     00:11:22:33:44:55
udp_target udp        -          rtl8139     52:54:00:54:02:e5
mcast_target mcast      -          rtl8139     52:54:00:29:ae:b7
server_target server     -          rtl8139     52:54:00:22:c9:42
client_target client     -          rtl8139     52:54:00:8b:c9:51

# virsh domifstat rhel  mcast_target
error: Failed to get interface stats rhel mcast_target
error: internal error: /proc/net/dev: Interface not found

# virsh domifstat rhel  52:54:00:22:c9:42
error: Failed to get interface stats rhel 52:54:00:22:c9:42
error: internal error: /proc/net/dev: Interface not found

# pidof libvirtd
76350
Comment 9 yalzhang@redhat.com 2018-07-19 08:01:54 UTC 
Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to confirm, thank you!

2) And for the error message in comment 4, I think it is not related with this bug. Furthermore, I can not reproduce when restart libvirtd with interfaces in comment 8 or comment 1, so if you agree, I will move this bug to be verified.

3) I found the same error message here: https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html

I'm not sure if that patch is to fix this kind of issue, and I can not tell if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you please help to check? because I can still get the error with below scenario:

Start with non-exits nwfilter(100% reproducible)
1. prepare a guest with interface as below with not exists nwfitler:
 <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
      <filterref filter='no-ip'/>
    </interface>

# virsh start rhel
error: Failed to start domain rhel
error: internal error: referenced filter 'no-ip' is missing

# grep error /var/log/libvirt/libvirtd.log
2018-07-19 07:27:23.139+0000: 79106: error : virNWFilterObjListFindInstantiateFilter:201 : internal error: referenced filter 'no-ip' is missing
2018-07-19 07:27:23.162+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.167+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.169+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.171+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.172+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.173+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.173+0000: 79106: error : virNWFilterBindingLookupByPortDev:589 : portdev in virNWFilterBindingLookupByPortDev must not be NULL
2018-07-19 07:27:23.175+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.177+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device
2018-07-19 07:27:23.179+0000: 79156: error : virNetDevSendEthtoolIoctl:3072 : ethtool ioctl error: No such device

As this is a negative scenario, I'm not sure if it is a issue with such error message. Could you please help to confirm? Thank you very much!
Comment 10 Michal Privoznik 2018-07-23 12:01:09 UTC 
(In reply to yalzhang from comment #9)
> Hi Michal, 1) I think the result on comment 8 is acceptable. Please help to
> confirm, thank you!

Yes it is.

> 
> 2) And for the error message in comment 4, I think it is not related with
> this bug. Furthermore, I can not reproduce when restart libvirtd with
> interfaces in comment 8 or comment 1, so if you agree, I will move this bug
> to be verified.

Yes, it is unrelated. I'm trying to fix it upstream but it is different issue.

> 
> 3) I found the same error message here:
> https://www.redhat.com/archives/libvir-list/2018-June/msg01348.html
> 
> I'm not sure if that patch is to fix this kind of issue, and I can not tell
> if that patch is merged downstream in libvirt-4.5.0-3.el7.x86_64. Could you
> please help to check? because I can still get the error with below scenario:
> 

This is a different issue (the same as in 2) and should be tracked separately.
Comment 11 yalzhang@redhat.com 2018-07-24 11:19:49 UTC 
file Bug 1607831 to track the issue in comment 10 and move this one into verified.
Comment 13 errata-xmlrpc 2018-10-30 09:56:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113
Note You need to log in before you can comment on or make changes to this bug.