Bug 495958 - xentop doesn't report vif statistics on bonded interfaces
xentop doesn't report vif statistics on bonded interfaces
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.4
All Linux
low Severity medium
: rc
: ---
Assigned To: Michal Novotny
Virtualization Bugs
:
Depends On:
Blocks: 514499
  Show dependency treegraph
 
Reported: 2009-04-15 13:47 EDT by Adam Stokes
Modified: 2014-02-02 17:37 EST (History)
11 users (show)

See Also:
Fixed In Version: xen-3.0.3-107.el5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-01-13 17:16:40 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/proc/net/dev of a system without VLAN (2.31 KB, text/plain)
2009-06-19 13:27 EDT, Rik van Riel
no flags Details
Xenstat network-bonding patch (10.41 KB, patch)
2009-11-30 10:59 EST, Michal Novotny
no flags Details | Diff
Xenstat network-bonding patch version 2 (9.20 KB, patch)
2009-12-02 08:22 EST, Michal Novotny
no flags Details | Diff

  None (edit)
Comment 6 Justin M. Forbes 2009-05-29 11:12:51 EDT
Still looking for the cause, but it appears to be a symptom of the dom0 not having an interface on the vlans being used.
Comment 9 Rik van Riel 2009-06-19 13:17:29 EDT
Digging through the network code, it looks like the statistics for vlan devices should show up in /proc/net/vlan/<device name> and not in /proc/net/dev

I do not know whether making those statistics also show up in /proc/net/dev (or adding them to the "main" device statistics) would break anything.  Herbert?
Comment 10 Rik van Riel 2009-06-19 13:25:13 EDT
FWIW, the vif0.X interfaces show all zeroes on non-VLANed configurations, too.

The vifX.0 interfaces are the ones that have actual statistics on them.
Comment 11 Herbert Xu 2009-06-19 13:25:40 EDT
Rik, do you mean the TX/RX counters? They certainly do show up in /proc/net/dev for VLAN devices.  In fact the attached /proc/net/dev contains the VLAN devices bond1.115 and bond1.116.
Comment 12 Rik van Riel 2009-06-19 13:27:45 EDT
Created attachment 348685 [details]
/proc/net/dev of a system without VLAN

Note that the vethX and vif0.X interfaces have all-zeroed statistics.

The interfaces that accumulate statistics are vifX.0 - those seem to be missing from the /proc/net/dev on the customer system.
Comment 13 Herbert Xu 2009-06-19 13:28:56 EDT
vif0.X are Xen netloop devices, vifX.0 are Xen netback devices.  Both should have non-zero counters if traffic is passing through them.
Comment 14 Herbert Xu 2009-06-19 13:30:41 EDT
Rik, vifX.0 only exist if you have guests running.  I think the original complaint is about dom0 stats...
Comment 18 Rik van Riel 2009-07-09 15:45:47 EDT
This is a feature request for xentop.
Comment 25 Michal Novotny 2009-11-10 07:49:19 EST
Well, I did testing using version Xen 3.0.3-96.el5 with kernel-xen-2.6.18-172.el5 and I am getting following results...

      NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR
SSID
  Domain-0 -----r         49    0.9    7673936   91.5   no limit       n/a     4    5        0        0    0        0        0        0
   0
Net0 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes        0pkts        0err        0drop
Net1 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes        0pkts        0err        0drop
Net2 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes        0pkts        0err        0drop
Net3 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes        0pkts        0err        0drop
Net4 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes        0pkts        0err        0drop
rhel5-32pv --b---         11    0.0     524152    6.2     524288       6.3     1    1        9        6    0        0        0        0
   0
Net0 RX:     6369bytes       94pkts        0err        0drop  TX:     9644bytes       58pkts        0err        0drop

I did bond0 configuration from manual at https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-networkscripts-interfaces.html . It's showing bond0 and pbond0 devices...

I am having:
(vif-script vif-bridge)
(network-bridge 'network-bridge-bonding')
in my xend-config.sxp set up and I think it's fine because it's showing statistics for rhel5-32pv guest...

In xen domain configuration file I tried to use both virbr0 and bond0 devices and the same results...

Is this OK for DomU? Just for clarification...

Also, I think this BZ is about dom0 stats like stated in comment #0. The problem here is that `ifconfig` doesn't show any vif devices for dom0 and therefore it's showing 0s in dom0 stats... That's one thing, second thing here is that since bond device is common both dom0 and domUs I am afraid there is no way we can get real dom0 *only* stats because all the traffic is going through this bond device (for both dom0 and domU) and there is no interface to get stats for dom0 only...

Herbert, somebody told me that you're a good guy to ask about networking issues, what's your opinion about that? Is there some way to get dom0 stats if we use bonding devices ?

Thanks,
Michal
Comment 26 Michal Novotny 2009-11-11 11:11:50 EST
(In reply to comment #25)
> Well, I did testing using version Xen 3.0.3-96.el5 with
> kernel-xen-2.6.18-172.el5 and I am getting following results...
> 
>       NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%)
> VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR
> SSID
>   Domain-0 -----r         49    0.9    7673936   91.5   no limit       n/a    
> 4    5        0        0    0        0        0        0
>    0
> Net0 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes
>        0pkts        0err        0drop
> Net1 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes
>        0pkts        0err        0drop
> Net2 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes
>        0pkts        0err        0drop
> Net3 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes
>        0pkts        0err        0drop
> Net4 RX:        0bytes        0pkts        0err        0drop  TX:        0bytes
>        0pkts        0err        0drop
> rhel5-32pv --b---         11    0.0     524152    6.2     524288       6.3    
> 1    1        9        6    0        0        0        0
>    0
> Net0 RX:     6369bytes       94pkts        0err        0drop  TX:     9644bytes
>       58pkts        0err        0drop
> 
> I did bond0 configuration from manual at
> https://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/ref-guide/s1-networkscripts-interfaces.html
> . It's showing bond0 and pbond0 devices...
> 
> I am having:
> (vif-script vif-bridge)
> (network-bridge 'network-bridge-bonding')
> in my xend-config.sxp set up and I think it's fine because it's showing
> statistics for rhel5-32pv guest...
> 
> In xen domain configuration file I tried to use both virbr0 and bond0 devices
> and the same results...
> 
> Is this OK for DomU? Just for clarification...
> 
> Also, I think this BZ is about dom0 stats like stated in comment #0. The
> problem here is that `ifconfig` doesn't show any vif devices for dom0 and
> therefore it's showing 0s in dom0 stats... That's one thing, second thing here
> is that since bond device is common both dom0 and domUs I am afraid there is no
> way we can get real dom0 *only* stats because all the traffic is going through
> this bond device (for both dom0 and domU) and there is no interface to get
> stats for dom0 only...
> 
> Herbert, somebody told me that you're a good guy to ask about networking
> issues, what's your opinion about that? Is there some way to get dom0 stats if
> we use bonding devices ?
> 
> Thanks,
> Michal  

Well, not everything was set up right yesterday so I altered my setup but I am seeing the same result. Also, I don't know whether there is a way to get vif device for dom0 statistics when using bonding interfaces. I did try to investigate it but I found nothing...

Michal
Comment 27 Herbert Xu 2009-11-27 07:17:12 EST
Michal, you can always take the bonding device's total counters, subtract the guest counters to arrive at the dom0's share.
Comment 28 Michal Novotny 2009-11-27 07:21:29 EST
(In reply to comment #27)
> Michal, you can always take the bonding device's total counters, subtract the
> guest counters to arrive at the dom0's share.  

Herbert, I don't think this way is clear but unfortunately this solution is the only we can have. Or is there any direct way how to get dom0 stats somewhere from /proc tree or something?

Michal
Comment 29 Herbert Xu 2009-11-27 07:55:19 EST
Without using a netloop device in dom0, this is the best you can do short of adding a packet/byte counter using netfilter or similar.
Comment 30 Herbert Xu 2009-11-27 07:56:05 EST
Actually I take that back, you should be able to use the bridge device's counters, no?
Comment 31 Michal Novotny 2009-11-27 07:58:36 EST
(In reply to comment #30)
> Actually I take that back, you should be able to use the bridge device's
> counters, no?  

Well, but if 2 guests are communicating with each other? It should be in bridge statistics as well, right?

Michal
Comment 32 Michal Novotny 2009-11-27 08:23:04 EST
(In reply to comment #31)
> (In reply to comment #30)
> > Actually I take that back, you should be able to use the bridge device's
> > counters, no?  
> 
> Well, but if 2 guests are communicating with each other? It should be in bridge
> statistics as well, right?
> 
> Michal  

One more thing, we discussed this with Jirka and if we use formula (Tx is just the example, it should be for all the stats):

 Tx(dom0) = Tx(bridge) - sum( Tx(all the guests) )

we can get all the domain statistics and subtract it from the bridge statistics but the problem here is that if guest is destroyed/shut down we don't see the statistics in /proc/net/dev and therefore it would appear that this traffic was generated by dom0 because the entries for not existing domains are no longer in /proc/net/dev.

Any idea how to solve this one ?

Thanks,
Michal
Comment 33 Herbert Xu 2009-11-27 08:37:11 EST
(In reply to comment #31)
>
> Well, but if 2 guests are communicating with each other? It should be in bridge
> statistics as well, right?

Only traffic terminating on the bridge device's MAC should show up in its counters.
Comment 34 Michal Novotny 2009-11-27 08:52:51 EST
(In reply to comment #33)
> (In reply to comment #31)
> >
> > Well, but if 2 guests are communicating with each other? It should be in bridge
> > statistics as well, right?
> 
> Only traffic terminating on the bridge device's MAC should show up in its
> counters.  

Ok, but the real problem here is described in comment #32.

Michal
Comment 35 Michal Novotny 2009-11-27 09:00:52 EST
(In reply to comment #33)
> (In reply to comment #31)
> >
> > Well, but if 2 guests are communicating with each other? It should be in bridge
> > statistics as well, right?
> 
> Only traffic terminating on the bridge device's MAC should show up in its
> counters.  

So, does this mean that statistics for bonding device (bond0) are statistics for dom0?

Thanks,
Michal
Comment 36 Herbert Xu 2009-11-27 09:11:58 EST
(In reply to comment #35)
>
> So, does this mean that statistics for bonding device (bond0) are statistics
> for dom0?

If bond0 is the bridge then yes.
Comment 37 Michal Novotny 2009-11-30 10:59:01 EST
Created attachment 374790 [details]
Xenstat network-bonding patch

Thanks for the clarification Herbert. Considering this I was able to create a patch that checks whether we have dom0 statistics (as we don't have *only* when bonding device is used) and if there are none we get the statistics of bridge interface (ie. bond0 in example we discussed above). It's been tested on x86_64 machine with both bonding enabled and disabled.

The very same patch (ie. with no modifications) were sent upstream for consideration too.
Comment 39 Michal Novotny 2009-12-02 08:22:55 EST
Created attachment 375426 [details]
Xenstat network-bonding patch version 2

This is new patch for this one...
Comment 45 errata-xmlrpc 2011-01-13 17:16:40 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0031.html

Note You need to log in before you can comment on or make changes to this bug.