Bug 2172578

Summary: Networking receive/transmit_bytes metrics values are swapped
Product: Red Hat Enterprise Linux 8 Reporter: Michal Privoznik <mprivozn>
Component: libvirtAssignee: Michal Privoznik <mprivozn>
Status: CLOSED ERRATA QA Contact: yalzhang <yalzhang>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.8CC: akalenyu, dzheng, haizhao, ibezukh, jdenemar, jsuchane, jvilaca, mprivozn, ngavrilo, sradco, stirabos, virt-maint, yalzhang, ymankad
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-8.0.0-18.module+el8.8.0+18287+112bb4e6 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2169168
: 2173976 2173977 2192909 (view as bug list) Environment:
Last Closed: 2023-05-16 08:18:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2172574    
Bug Blocks: 2169168, 2173976, 2173977, 2192909    
Deadline: 2023-02-27   

Description Michal Privoznik 2023-02-22 15:25:18 UTC
+++ This bug was initially created as a clone of Bug #2169168 +++

Description of problem:
Networking receive/transmit_bytes metrics values are swapped

Version-Release number of selected component (if applicable):
CNV 4.12.1

How reproducible:
100%

Steps to Reproduce:
1. Create VM
2. Download large image

Actual results:
transmit goes up

Expected results:
receive goes up

Additional info:
Originally reported in issue https://github.com/kubevirt/kubevirt/issues/9129
Note this is fixed in main:
https://github.com/kubevirt/kubevirt/issues/9129#issuecomment-1415844928
Since libvirt was bumped to a version that contains the fix.


[fedora@simple-vm ~]$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[fedora@simple-vm ~]$ ls -l Fedora-Cloud-Base-37-1.7.x86_64.qcow2 
-rw-r--r--. 1 fedora fedora 492830720 Feb 12 12:36 Fedora-Cloud-Base-37-1.7.x86_64.qcow2

Observe metrics:
kubevirt_vmi_network_transmit_bytes_total: 518623611
kubevirt_vmi_network_receive_bytes_total: 1030624


--- Additional comment from Igor Bezukh on 2023-02-16 17:31:37 CET ---

Hi Michal,

Can you please assist us with backporting https://gitlab.com/libvirt/libvirt/-/commit/0862cb3ce46253a58ca02d36b2b6a6397a60bfc7 
to the libvirt release that is in use in 4.12?

TIA
Igor

--- Additional comment from Jaroslav Suchanek on 2023-02-17 10:53:59 CET ---

(In reply to Igor Bezukh from comment #2)
> Hi,
> 
> There are plans to move to UBI 9 (RHEL 9 based) in 4.12
> But currently 4.12 consumes UBI 8.6. So we will have to ask Libvirt folks to
> do 2 backports for us - one to libvirt 8.0.0 for RHEL 8.6 and one to libvirt
> 8.5.0 for RHEL 9

Can you please elaborate more, why you need it in RHEL-9.0, which is not being used by CNV? Backporting to RHEL-8.6.0 should be fine. RHEL-9.2 already contain the requested fix.

Thanks.

--- Additional comment from Igor Bezukh on 2023-02-21 10:49:31 CET ---

Hi Jaroslav,

Let me correct myself - we will need the backport for RHEL-8.6.0, we don't plan to switch our base containers to be RHEL-9 based in OCPV 4.12

Is there any action from our side regarding whats need to be done about the backport to RHEL-8.6.0?

Comment 3 yalzhang@redhat.com 2023-02-23 04:49:44 UTC
Reproduce this bug on libvirt-8.0.0-17.module+el8.8.0+18133+90800a0a.x86_64
# ip tuntap add mode tap name mytap0
# ip link set mytap0 up
# ip link set mytap0 master br0
# virsh dumpxml rhel | grep /interface -B6
   <interface type='ethernet'>
     <target dev='mytap0' managed='no'/>
     <model type='virtio'/>
   </interface>

Login guest, and run:
# curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L

Check the statistics on guest and host:
On guest:
# ip -s l show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:88:8b:23 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     517891022  357101      0     225       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      20557310  253394      0       0       0       0

On host:
# virsh domifstat rhel mytap0
mytap0 rx_bytes 20557310
mytap0 rx_packets 253394
mytap0 rx_errs 0
mytap0 rx_drop 0
mytap0 tx_bytes 517891112
mytap0 tx_packets 357102
mytap0 tx_errs 0
mytap0 tx_drop 0
The domifstat command shows the statistics from the host view, which is unexpected. 

While with bridge type interface as below, the domifstat command shows the statistics from the guest point of view, which is as expected.
# virsh dumpxml rhel | grep /interface -B7
    <interface type='bridge'>
      <mac address='52:54:00:41:7c:2c'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Download the file on guest, then check the statistics on host and guest:
On host:
# virsh domifstat rhel vnet1
vnet1 rx_bytes 509510064
vnet1 rx_packets 227805
vnet1 rx_errs 0
vnet1 rx_drop 0
vnet1 tx_bytes 14086386
vnet1 tx_packets 178018
vnet1 tx_errs 0
vnet1 tx_drop 0

On guest:
# ip -s l show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:41:7c:2c brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     509509914  227803      0    1155       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      14086386  178018

Comment 4 yalzhang@redhat.com 2023-02-23 04:52:58 UTC
From the test result above, the "virsh domifstat" can not catch the RX_dropped value, is that acceptable? 

On host, I have checked the rx_dropped file, it's also 0. 
# cat /sys/devices/virtual/net/vnet1/statistics/rx_dropped 
0

Comment 7 Michal Privoznik 2023-02-27 14:36:51 UTC
(In reply to yalzhang from comment #4)
> From the test result above, the "virsh domifstat" can not catch the
> RX_dropped value, is that acceptable? 
> 
> On host, I have checked the rx_dropped file, it's also 0. 
> # cat /sys/devices/virtual/net/vnet1/statistics/rx_dropped 
> 0

Yes it is. Libivirt does nothing more than read this value. If kernel reports it as zero there's not much we can do.

Comment 14 yalzhang@redhat.com 2023-03-05 00:27:24 UTC
Test on libvirt-8.0.0-18.module+el8.8.0+18287+112bb4e6.x86_64 with the scenarios in bz2172574#c4  bz2172574#c5, the result is as expected.

Comment 17 errata-xmlrpc 2023-05-16 08:18:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: virt:rhel and virt-devel:rhel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:2757