RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1461270 - virsh domifstat tx_drop value is not correct for openvswitch vhostuser interfaces
Summary: virsh domifstat tx_drop value is not correct for openvswitch vhostuser interf...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.4
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Jingjing Shao
URL:
Whiteboard:
: 1624273 (view as bug list)
Depends On:
Blocks: 1627746
TreeView+ depends on / blocked
 
Reported: 2017-06-14 05:47 UTC by chhu
Modified: 2021-12-10 15:05 UTC (History)
10 users (show)

Fixed In Version: libvirt-3.7.0-1.el7
Doc Type: Bug Fix
Doc Text:
Previously, diagnostic commands for virtual machines (VMs), such as "virsh domifstat", in some cases incorrectly reported 0 as the value of certain statistics. This update modifies the libvirtd service to fetch these statistics one by one instead of in groups. As a result, reporting 0 values for VM statistics is significantly less likely.
Clone Of:
: 1627746 (view as bug list)
Environment:
Last Closed: 2018-04-10 10:48:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:0704 0 None None None 2018-04-10 10:50:02 UTC

Description chhu 2017-06-14 05:47:47 UTC
Version-Release number of selected component (if applicable):
libvirt-3.2.0-9.el7.x86_64
qemu-kvm-rhev-2.9.0-7.el7.x86_64
kernel-3.10.0-677.el7.x86_64


How reproducible:
100%

Steps to reproduce:
1. Start 2 guests with vhostuser

#virsh dumpxml r7-4t1| grep interface -A 6
    <interface type='vhostuser'>
      <mac address='52:54:00:50:1a:9b'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <target dev='vhost-user1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

#virsh dumpxml r7-4t2| grep interface -A 6
    <interface type='vhostuser'>
      <mac address='52:54:00:25:1b:fb'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user2' mode='client'/>
      <target dev='vhost-user2'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 8     r7-4t1                         running
 9     r7-4t2                         running

2. Configure the IP for vNIC
In guest1:
ifconfig eth0 192.168.100.1

In guest2:
# ifconfig eth0 192.168.100.2

3. On one guest ping the other one.
In guest1: ping 192.168.100.2
In guest2: ping 192.168.100.1

4.  Check the network statistics
# ovs-vsctl list interface
......
mtu                 : 1500
name                : "vhost-user2"
ofport              : 2
ofport_request      : []
options             : {}
other_config        : {}
statistics          : {rx_bytes=9637014, rx_errors=0, rx_packets=54317, tx_bytes=9615286, tx_dropped=18828, tx_packets=54239}
status              : {}
type                : dpdkvhostuser

# virsh domifstat r7-4t2 vhost-user2
vhost-user2 rx_bytes 9615286
vhost-user2 rx_packets 54239
vhost-user2 rx_errs 0
vhost-user2 rx_drop 0
vhost-user2 tx_bytes 9637014
vhost-user2 tx_packets 54317
vhost-user2 tx_errs 0
vhost-user2 tx_drop 0

# ovs-vsctl get Interface vhost-user2 statistics:rx_errors
0

# ovs-vsctl get Interface vhost-user2  statistics:tx_dropped
18828

5. Check the virsh domifstat, the tx_drop should be 18828, but it's 0.
# virsh domiflist r7-4t2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vhost-user2 vhostuser  -          virtio      52:54:00:25:1b:fb

# virsh domifstat r7-4t2 vhost-user2
vhost-user2 rx_bytes 9611986
vhost-user2 rx_packets 54217
vhost-user2 rx_errs 0
vhost-user2 rx_drop 0
vhost-user2 tx_bytes 9633776
vhost-user2 tx_packets 54296
vhost-user2 tx_errs 0
vhost-user2 tx_drop 0


Expected results:
In step5, the tx_drop, rx_errors should be the same as ovs-vsctl statistics.

Actual results:
In step5. the value of tx_drop is not correct.

Additional info:
# ovs-vsctl get Interface vhost-user2 statistics:rx_errors statistics:tx_errors
ovs-vsctl: no key "tx_errors" in Interface record "vhost-user2" column statistics
# ovs-vsctl get Interface vhost-user2 statistics:rx_dropped statistics:tx_droppped
ovs-vsctl: no key "rx_dropped" in Interface record "vhost-user2" column statistics

Comment 2 Michal Privoznik 2017-06-14 08:58:48 UTC
The problem here is libvirt issues one ovs-vsctl command trying to fetch all the data. However, upon error, no output is produced for libvirt to parse. Therefore we must break it into smaller pieces:

int
virNetDevOpenvswitchInterfaceStats(const char *ifname,
                                   virDomainInterfaceStatsPtr stats)
{
    ...
    cmd = virCommandNew(OVSVSCTL);
    virNetDevOpenvswitchAddTimeout(cmd);
    virCommandAddArgList(cmd, "get", "Interface", ifname,
                         "statistics:rx_errors",
                         "statistics:rx_dropped",
                         "statistics:tx_errors",
                         "statistics:tx_dropped", NULL);
    ...
}

Comment 3 Michal Privoznik 2017-06-22 13:45:32 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2017-June/msg00913.html

Comment 4 Michal Privoznik 2017-06-23 09:52:47 UTC
I've just pushed the patch upstream:

commit edaf13565712b431c6fd66a5c4a964af7759ce68
Author:     Michal Privoznik <mprivozn>
AuthorDate: Wed Jun 14 15:23:29 2017 +0200
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Jun 23 11:51:23 2017 +0200

    virNetDevOpenvswitchInterfaceStats: Be more forgiving when fetching stats
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1461270
    
    When fetching stats for a vhost-user type of interface, we run
    couple of ovs-vsctl commands and parse their output. However, not
    all stats exist at all times, for instance "rx_dropped" or
    "tx_errors" can be missing. Thing is, we ask for a bulk of
    statistics and if one of them is missing an error is reported
    instead of returning the rest. Since we ignore errors, we fail to
    set statistics. Fix this by asking for each piece alone.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v3.4.0-165-gedaf13565

Comment 6 yalzhang@redhat.com 2017-09-28 03:20:11 UTC
Test on below packages, the result is as expected.
libvirt-3.7.0-2.el7.x86_64
openvswitch-2.8.0-0.1.20170810git3631ed2.el7fdb.x86_64

1. Set 2 guests with vhostuser type interface, then ping each other
2. on host check that there is no rx_error, and the statistics are right.

# virsh domiflist vhost1 
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vhost-user1 vhostuser  -          virtio      52:54:00:69:33:29

# virsh domiflist vhost2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
vhost-user2 vhostuser  -          virtio      52:54:00:93:51:db

# virsh domifstat vhost1 vhost-user1
vhost-user1 rx_bytes 403278
vhost-user1 rx_packets 4188
vhost-user1 rx_drop 50
vhost-user1 tx_bytes 404706
vhost-user1 tx_packets 4206
vhost-user1 tx_errs 0
vhost-user1 tx_drop 0

# ovs-vsctl list interface
...
name                : "vhost-user1"
...
statistics          : {"rx_1024_to_1522_packets"=0, "rx_128_to_255_packets"=0, "rx_1523_to_max_packets"=0, "rx_1_to_64_packets"=125, "rx_256_to_511_packets"=0, "rx_512_to_1023_packets"=0, "rx_65_to_127_packets"=4082, rx_bytes=404706, rx_dropped=0, rx_errors=0, rx_packets=4206, tx_bytes=403278, tx_dropped=50, tx_packets=4188}

Comment 10 errata-xmlrpc 2018-04-10 10:48:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704

Comment 11 Michal Privoznik 2018-09-10 11:27:35 UTC
*** Bug 1624273 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.