Bug 2192909

+++ This bug was initially created as a clone of Bug #2172578 +++

+++ This bug was initially created as a clone of Bug #2169168 +++

Description of problem:
Networking receive/transmit_bytes metrics values are swapped

Version-Release number of selected component (if applicable):
CNV 4.12.1

How reproducible:
100%

Steps to Reproduce:
1. Create VM
2. Download large image

Actual results:
transmit goes up

Expected results:
receive goes up

Additional info:
Originally reported in issue https://github.com/kubevirt/kubevirt/issues/9129
Note this is fixed in main:
https://github.com/kubevirt/kubevirt/issues/9129#issuecomment-1415844928
Since libvirt was bumped to a version that contains the fix.


[fedora@simple-vm ~]$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[fedora@simple-vm ~]$ ls -l Fedora-Cloud-Base-37-1.7.x86_64.qcow2 
-rw-r--r--. 1 fedora fedora 492830720 Feb 12 12:36 Fedora-Cloud-Base-37-1.7.x86_64.qcow2

Observe metrics:
kubevirt_vmi_network_transmit_bytes_total: 518623611
kubevirt_vmi_network_receive_bytes_total: 1030624


--- Additional comment from Igor Bezukh on 2023-02-16 17:31:37 CET ---

Hi Michal,

Can you please assist us with backporting https://gitlab.com/libvirt/libvirt/-/commit/0862cb3ce46253a58ca02d36b2b6a6397a60bfc7 
to the libvirt release that is in use in 4.12?

TIA
Igor

--- Additional comment from Jaroslav Suchanek on 2023-02-17 10:53:59 CET ---

(In reply to Igor Bezukh from comment #2)
> Hi,
> 
> There are plans to move to UBI 9 (RHEL 9 based) in 4.12
> But currently 4.12 consumes UBI 8.6. So we will have to ask Libvirt folks to
> do 2 backports for us - one to libvirt 8.0.0 for RHEL 8.6 and one to libvirt
> 8.5.0 for RHEL 9

Can you please elaborate more, why you need it in RHEL-9.0, which is not being used by CNV? Backporting to RHEL-8.6.0 should be fine. RHEL-9.2 already contain the requested fix.

Thanks.

--- Additional comment from Igor Bezukh on 2023-02-21 10:49:31 CET ---

Hi Jaroslav,

Let me correct myself - we will need the backport for RHEL-8.6.0, we don't plan to switch our base containers to be RHEL-9 based in OCPV 4.12

Is there any action from our side regarding whats need to be done about the backport to RHEL-8.6.0?

--- Additional comment from Michal Privoznik on 2023-02-22 17:36:31 CET ---

To POST:

https://gitlab.com/redhat/rhel/src/libvirt/-/merge_requests/96

Scratch build to test:

https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=50933128

http://brew-task-repos.usersys.redhat.com/repos/scratch/mprivozn/libvirt/8.0.0/18.el8_rc.a9a218e899/

--- Additional comment from Michal Privoznik on 2023-02-22 17:37:28 CET ---

Requesting z-stream, per comment 0.

--- Additional comment from yalzhang on 2023-02-23 05:49:44 CET ---

Reproduce this bug on libvirt-8.0.0-17.module+el8.8.0+18133+90800a0a.x86_64
# ip tuntap add mode tap name mytap0
# ip link set mytap0 up
# ip link set mytap0 master br0
# virsh dumpxml rhel | grep /interface -B6
   <interface type='ethernet'>
     <target dev='mytap0' managed='no'/>
     <model type='virtio'/>
   </interface>

Login guest, and run:
# curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L

Check the statistics on guest and host:
On guest:
# ip -s l show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:88:8b:23 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     517891022  357101      0     225       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      20557310  253394      0       0       0       0

On host:
# virsh domifstat rhel mytap0
mytap0 rx_bytes 20557310
mytap0 rx_packets 253394
mytap0 rx_errs 0
mytap0 rx_drop 0
mytap0 tx_bytes 517891112
mytap0 tx_packets 357102
mytap0 tx_errs 0
mytap0 tx_drop 0
The domifstat command shows the statistics from the host view, which is unexpected. 

While with bridge type interface as below, the domifstat command shows the statistics from the guest point of view, which is as expected.
# virsh dumpxml rhel | grep /interface -B7
    <interface type='bridge'>
      <mac address='52:54:00:41:7c:2c'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Download the file on guest, then check the statistics on host and guest:
On host:
# virsh domifstat rhel vnet1
vnet1 rx_bytes 509510064
vnet1 rx_packets 227805
vnet1 rx_errs 0
vnet1 rx_drop 0
vnet1 tx_bytes 14086386
vnet1 tx_packets 178018
vnet1 tx_errs 0
vnet1 tx_drop 0

On guest:
# ip -s l show enp1s0
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 52:54:00:41:7c:2c brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     509509914  227803      0    1155       0       0 
    TX:  bytes packets errors dropped carrier collsns           
      14086386  178018

--- Additional comment from yalzhang on 2023-02-23 05:52:58 CET ---

From the test result above, the "virsh domifstat" can not catch the RX_dropped value, is that acceptable? 

On host, I have checked the rx_dropped file, it's also 0. 
# cat /sys/devices/virtual/net/vnet1/statistics/rx_dropped 
0

--- Additional comment from Yash Mankad on 2023-02-23 17:21:18 CET ---

Hi Yalan - could you please set the ITM, and Michal - could you please request exception? and add the justification for the RHEL Program.

We discussed this on the ZStream call today and it is good to go from a zstream+ perspectve.
However, we will need to wait for 8.8 exception+ approval before I can approve the zstream.

--- Additional comment from RHEL Program Management on 2023-02-27 10:46:04 CET ---

This BZ requests changes which would normally be forbidden in this product release's current phase (EUS, E4S, TUS, AMC, etc.). Generally this means that the requested changes intentionally introduce new features, or include a rebase from upstream (which may also include new features), or constitute a significant risk of regression for existing users.

To obtain approval for this BZ to proceed, the assignee must follow the RHEL BU "Policy Exception Framework" (https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=RHELPLAN&title=Policy+Exceptions) which involves creating a Jira issue for the RHEL BU to review and discuss. 

When the issue has been created, the assignee should add it to this BZ as a Link. The assignee should also add the relevant Z-stream Program Manager as a watcher on the issue (the program manager's name can be found on the Product Pages 'People' page for the product and version involved).

When the Program Manager is notified that the RHEL BU has made a decision, the BZ will be updated as follows:

* If the exception was granted, feature_ack+ and zstream+ will be set on the BZ. This will set the remaining flags to allow the BZ to proceed through the normal workflow.

* If the exception was not granted, exception- will be set on the BZ. This will stop the workflow, and then the BZ can be closed with 'EOL' as the resolution.

--- Additional comment from Michal Privoznik on 2023-02-27 15:36:51 CET ---

(In reply to yalzhang from comment #4)
> From the test result above, the "virsh domifstat" can not catch the
> RX_dropped value, is that acceptable? 
> 
> On host, I have checked the rx_dropped file, it's also 0. 
> # cat /sys/devices/virtual/net/vnet1/statistics/rx_dropped 
> 0

Yes it is. Libivirt does nothing more than read this value. If kernel reports it as zero there's not much we can do.

--- Additional comment from Michal Privoznik on 2023-02-27 16:32:18 CET ---

(In reply to RHEL Program Management from comment #6)
> This BZ requests changes which would normally be forbidden in this product
> release's current phase (EUS, E4S, TUS, AMC, etc.). Generally this means
> that the requested changes intentionally introduce new features, or include
> a rebase from upstream (which may also include new features), or constitute
> a significant risk of regression for existing users.

Kevin, help me out here please. This bug is neither rebase, nor backport of an RFE. I want to backport a simple, two line bug fix (one line is moved) - see link in comment #0.

> 
> To obtain approval for this BZ to proceed, the assignee must follow the RHEL
> BU "Policy Exception Framework"
> (https://docs.engineering.redhat.com/pages/viewpage.
> action?spaceKey=RHELPLAN&title=Policy+Exceptions) which involves creating a
> Jira issue for the RHEL BU to review and discuss. 
> 

Do I really need to undergo this process?

> When the issue has been created, the assignee should add it to this BZ as a
> Link. The assignee should also add the relevant Z-stream Program Manager as
> a watcher on the issue (the program manager's name can be found on the
> Product Pages 'People' page for the product and version involved).
> 
> When the Program Manager is notified that the RHEL BU has made a decision,
> the BZ will be updated as follows:
> 
> * If the exception was granted, feature_ack+ and zstream+ will be set on the
> BZ. This will set the remaining flags to allow the BZ to proceed through the
> normal workflow.

I mean, we do not need feature_ack+. We do need zstream+ though (because a layered product needs this fix in RHEL-8.6.0).
Thanks in advance.

--- Additional comment from Kevin P. Fleming on 2023-02-27 16:54:52 CET ---

Unfortunately this is an artifact of the situation we have today, where we use the 'exception' flag to mean two unrelated things. In this case exception? has been set in order to request a schedule exception for inclusion into 8.8 GA, but the BRE rules also see that when zstream? is set and assume that the exception request is for the Z-stream. It is not.

I'll go ahead and set zstream+, and the exception request for 8.8 can proceed as intended.

--- Additional comment from RHEL Program Management on 2023-02-27 16:55:00 CET ---

This BZ has been approved for cloning.

The BZ can be now cloned by everyone with the self-service cloning tool https://watson.engineering.redhat.com/rules

For more information regarding ZStream and cloning please visit https://docs.google.com/document/d/1yL8iTHjxyQ7sI-fC4PcPjpOOyfF5ECGnK-B7r_QRZm4/edit#

--- Additional comment from RHEL Program Management Team on 2023-02-28 15:58:51 CET ---

This bug has been copied as 8.7.0 stream bug#2173976 and now must be resolved in the current update release, blocker flag set.

This bug has been copied as 8.6.0 stream bug#2173977 and now must be resolved in the current update release, blocker flag set.

--- Additional comment from errata-xmlrpc on 2023-03-02 11:19:37 CET ---

This bug has been added to advisory RHEA-2022:102625 by Jiri Denemark (jdenemar)

--- Additional comment from errata-xmlrpc on 2023-03-02 11:19:49 CET ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHEA-2022:102625-03
https://errata.devel.redhat.com/advisory/102625

--- Additional comment from yalzhang on 2023-03-05 01:27:24 CET ---

Test on libvirt-8.0.0-18.module+el8.8.0+18287+112bb4e6.x86_64 with the scenarios in bz2172574#c4  bz2172574#c5, the result is as expected.