Bug 1397940

Summary: Gathers network statistics of openvswitch vhostuser interfaces
Product: Red Hat Enterprise Linux 7 Reporter: Mehdi ABAAKOUK <mabaakou>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: chhu
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.4CC: ailan, atelang, chhu, dyuan, fbaudin, jdenemar, jsuchane, lhuang, mprivozn, rbalakri, xuzhang, yalzhang
Target Milestone: rcKeywords: Upstream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-3.0.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 17:19:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1459091    
Bug Blocks:    

Description Mehdi ABAAKOUK 2016-11-23 15:47:14 UTC
Description of problem:

Currently network statistics are retrieved from /proc/net/dev but that works only for interface seen by the kernel, when openvswitch and vhostuser network interface are used, no statistics are gathered while they are available via ab openvswitch call.


Expected: 

Statistics retrieved from openvswitch.

Comment 1 Mehdi ABAAKOUK 2016-11-23 15:49:22 UTC
Some discution have been started upstream: https://www.redhat.com/archives/libvir-list/2016-November/msg00915.html

Comment 4 Michal Privoznik 2016-12-09 09:42:35 UTC
I've just pushed patches upstream:

commit e0d893e86df2daa51b88fdb5f0195fec279da99f
Author:     Mehdi Abaakouk <sileht>
AuthorDate: Fri Nov 18 23:51:15 2016 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 9 10:28:07 2016 +0100

    Move virstat.c code to virnetdevtap.c
    
    This is just a code move of virstat.c to virnetdevtap.c

commit 9b6de7c506c1bfc76fdcb9a42f990b699fe18d2e
Author:     Mehdi Abaakouk <sileht>
AuthorDate: Fri Nov 18 23:51:14 2016 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 9 10:27:56 2016 +0100

    virstat: fix signature of virstat helper
    
    In preparation to the code move to virnetdevtap.c, this change:
    
    * renames virNetInterfaceStats to virNetDevTapInterfaceStats
    * changes 'path' to 'ifname', to use the same vocable as other
      method in virnetdevtap.c.
    * Add the attributes checker

commit 013df874db7e3e4b4f8a8fa25e068da776e0e2c7
Author:     Mehdi Abaakouk <sileht>
AuthorDate: Fri Nov 18 23:51:13 2016 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 9 10:23:09 2016 +0100

    Gathering vhostuser interface stats with ovs
    
    When vhostuser interfaces are used, the interface statistics
    are not available in /proc/net/dev.
    
    This change looks at the openvswitch interfaces statistics
    tables to provide this information for vhostuser interface.
    
    Note that in openvswitch world drop/error doesn't always make sense
    for some interface type. When these informations are not available we
    set them to 0 on the virDomainInterfaceStats.
    
    Signed-off-by: Michal Privoznik <mprivozn>

v2.5.0-57-ge0d893e86

Comment 6 chhu 2017-05-23 08:35:17 UTC
Hi,  Michal

I tried to verify this bug, however, when I use 'virsh domifstat <domain_name>',
I can't get the interface name.

# virsh domiflist r7-4t1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          vhostuser  -          virtio      52:54:01:93:55:db

And then, I can't get the statistics of this interface by `virsh domifstat <domain_name> <interface>`

Will you please help to have a look at this problem? Thank you!
You can find more details in below:


Try to verify the bug on packages:
libvirt-3.2.0-5.el7.x86_64
qemu-kvm-rhev-2.9.0-4.el7.x86_64
kernel-3.10.0-660.el7.x86_64

Test steps:
1. Prepare 2 guests like this:
guest1:
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  ......
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Broadwell</model>
    ......
    <feature policy='require' name='abm'/>
    <numa>
      <cell id='0' cpus='0-1' memory='1048576' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  ......
    <interface type='vhostuser'>
      <mac address='52:54:01:93:55:db'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user1' mode='client'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

guest2: with path='/var/run/openvswitch/vhost-user2'

2. Start the guests
# virsh start r7-4t1
Domain r7-4t1 started

# virsh start r7-4t2
Domain r7-4t2 started

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 6     r7-4t1                         running
 7     r7-4t2                         running

3. Configure the IP for vNIC
In guest1:
ifconfig eth0 192.168.100.1

In guest2:
# ifconfig eth0 192.168.100.2

4. On one guest ping the other one.
In guest1: ping 192.168.100.2
In guest2: ping 192.168.100.1

5. Check the network statistics
# ovs-vsctl list interface
_uuid               : 8725f0de-e8ae-4ac2-abba-0a9ae716c4a2
admin_state         : down
......
mac                 : []
mac_in_use          : "00:00:00:00:00:00"
mtu                 : 1500
name                : "vhost-user1"
......
statistics          : {rx_bytes=301030, rx_errors=0, rx_packets=1859, tx_bytes=290924, tx_dropped=1, tx_packets=1830}
status              : {}
type                : dpdkvhostuser

_uuid               : a73ae67e-0d16-415a-8c6a-5132e4766d94
admin_state         : down
......
mac                 : []
mac_in_use          : "00:00:00:00:00:00"
mtu                 : 1500
name                : "vhost-user2"
......
statistics          : {rx_bytes=291266, rx_errors=0, rx_packets=1831, tx_bytes=292630, tx_dropped=50, tx_packets=1809}
status              : {}
type                : dpdkvhostuser

#ovs-ofctl dump-flows ovsbr0
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=90456.579s, table=0, n_packets=1843, n_bytes=297950, idle_age=2, hard_age=65534, in_port=1 actions=output:2
 cookie=0x0, duration=90449.118s, table=0, n_packets=1807, n_bytes=287698, idle_age=0, hard_age=65534, in_port=2 actions=output:1

6. Check the virsh domiflist: no Interface is listed
#virsh domiflist r7-4t1
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          vhostuser  -          virtio      52:54:01:93:55:db

#virsh domiflist r7-4t2
Interface  Type       Source     Model       MAC
-------------------------------------------------------
-          vhostuser  -          virtio      52:59:21:13:11:db

7. Can't run virsh domifstat as don't know the interface name.

# virsh domifstat r7-4t1
error: command 'domifstat' requires <interface> option

# virsh domifstat r7-4t1 eth0
error: Failed to get interface stats r7-4t1 eth0
error: invalid argument: invalid path, 'eth0' is not a known interface

Comment 7 Michal Privoznik 2017-05-24 12:11:32 UTC
(In reply to chhu from comment #6)
> Hi,  Michal
> 
> I tried to verify this bug, however, when I use 'virsh domifstat
> <domain_name>',
> I can't get the interface name.
> 
> # virsh domiflist r7-4t1
> Interface  Type       Source     Model       MAC
> -------------------------------------------------------
> -          vhostuser  -          virtio      52:54:01:93:55:db
> 
> And then, I can't get the statistics of this interface by `virsh domifstat
> <domain_name> <interface>`

virsh domifstat r7-4t1 vhost-user1   should do the trick.

Michal

Comment 8 chhu 2017-05-25 03:05:36 UTC
(In reply to Michal Privoznik from comment #7)
> (In reply to chhu from comment #6)
> > Hi,  Michal
> > 
> > I tried to verify this bug, however, when I use 'virsh domifstat
> > <domain_name>',
> > I can't get the interface name.
> > 
> > # virsh domiflist r7-4t1
> > Interface  Type       Source     Model       MAC
> > -------------------------------------------------------
> > -          vhostuser  -          virtio      52:54:01:93:55:db
> > 
> > And then, I can't get the statistics of this interface by `virsh domifstat
> > <domain_name> <interface>`
> 
> virsh domifstat r7-4t1 vhost-user1   should do the trick.
> 
> Michal


Hi, Michal

I tried `virsh domifstat r7-4t1 vhost-user1 or vhost-user or eth0`. 
They are not work.

How about file a new bug about:
"virsh domiflist return NULL for the vhostuser interface name and source". 

And then set this one blocked by that bug? Waiting for your reply, thank you!


More detail as below:
# virsh domifstat r7-4t1 vhostuser1
error: Failed to get interface stats r7-4t1 vhostuser1
error: invalid argument: invalid path, 'vhostuser1' is not a known interface


# virsh domifstat r7-4t1 vhostuser
error: Failed to get interface stats r7-4t1 vhostuser
error: invalid argument: invalid path, 'vhostuser' is not a known interface

# virsh domifstat r7-4t1 eth0
error: Failed to get interface stats r7-4t1 eth0
error: invalid argument: invalid path, 'eth0' is not a known interface


Regards,
chhu

Comment 9 chhu 2017-06-06 09:31:23 UTC
(In reply to chhu from comment #8)
> (In reply to Michal Privoznik from comment #7)
> > (In reply to chhu from comment #6)
> > > Hi,  Michal
> > > 
> > > I tried to verify this bug, however, when I use 'virsh domifstat
> > > <domain_name>',
> > > I can't get the interface name.
> > > 
> > > # virsh domiflist r7-4t1
> > > Interface  Type       Source     Model       MAC
> > > -------------------------------------------------------
> > > -          vhostuser  -          virtio      52:54:01:93:55:db
> > > 
> > > And then, I can't get the statistics of this interface by `virsh domifstat
> > > <domain_name> <interface>`
> > 
> > virsh domifstat r7-4t1 vhost-user1   should do the trick.
> > 
> > Michal
> 
> 
> Hi, Michal
> 
> I tried `virsh domifstat r7-4t1 vhost-user1 or vhost-user or eth0`. 
> They are not work.
> 
> How about file a new bug about:
> "virsh domiflist return NULL for the vhostuser interface name and source". 
> 
> And then set this one blocked by that bug? Waiting for your reply, thank you!
> 
> 
> More detail as below:
> # virsh domifstat r7-4t1 vhostuser1
> error: Failed to get interface stats r7-4t1 vhostuser1
> error: invalid argument: invalid path, 'vhostuser1' is not a known interface
> 
> 
> # virsh domifstat r7-4t1 vhostuser
> error: Failed to get interface stats r7-4t1 vhostuser
> error: invalid argument: invalid path, 'vhostuser' is not a known interface
> 
> # virsh domifstat r7-4t1 eth0
> error: Failed to get interface stats r7-4t1 eth0
> error: invalid argument: invalid path, 'eth0' is not a known interface
> 
> 
> Regards,
> chhu

Thanks Michal's confirm, file bug1459091 for this issue.

Comment 18 errata-xmlrpc 2017-08-01 17:19:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 19 errata-xmlrpc 2017-08-01 23:59:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846