Description of problem: Networking receive/transmit_bytes metrics values are swapped Version-Release number of selected component (if applicable): CNV 4.12.1 How reproducible: 100% Steps to Reproduce: 1. Create VM 2. Download large image Actual results: transmit goes up Expected results: receive goes up Additional info: Originally reported in issue https://github.com/kubevirt/kubevirt/issues/9129 Note this is fixed in main: https://github.com/kubevirt/kubevirt/issues/9129#issuecomment-1415844928 Since libvirt was bumped to a version that contains the fix. [fedora@simple-vm ~]$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [fedora@simple-vm ~]$ ls -l Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -rw-r--r--. 1 fedora fedora 492830720 Feb 12 12:36 Fedora-Cloud-Base-37-1.7.x86_64.qcow2 Observe metrics: kubevirt_vmi_network_transmit_bytes_total: 518623611 kubevirt_vmi_network_receive_bytes_total: 1030624 Manifests: apiVersion: cdi.kubevirt.io/v1beta1 kind: DataVolume metadata: name: simple-dv spec: source: http: url: "http://.../Fedora-Cloud-Base-34-1.2.x86_64.qcow2" pvc: storageClassName: nfs accessModes: - ReadWriteOnce resources: requests: storage: 8Gi apiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: simple-vm namespace: default spec: running: true template: metadata: labels: {kubevirt.io/domain: simple-vm, kubevirt.io/vm: simple-vm} spec: domain: devices: disks: - disk: {bus: virtio} name: dv-disk - disk: {bus: virtio} name: cloudinitdisk resources: requests: {memory: 2048M} volumes: - dataVolume: {name: simple-dv} name: dv-disk - cloudInitNoCloud: userData: | #cloud-config password: fedora chpasswd: { expire: False } name: cloudinitdisk Also reproduces with virsh domstats since this is a libvirt issue (virsh domstats default_simple-vm | grep -e tx -e rx from launcher pod)
Can we bump Libvirt version for CNV 4.12?
Hi, There are plans to move to UBI 9 (RHEL 9 based) in 4.12 But currently 4.12 consumes UBI 8.6. So we will have to ask Libvirt folks to do 2 backports for us - one to libvirt 8.0.0 for RHEL 8.6 and one to libvirt 8.5.0 for RHEL 9
Hi Michal, Can you please assist us with backporting https://gitlab.com/libvirt/libvirt/-/commit/0862cb3ce46253a58ca02d36b2b6a6397a60bfc7 to the libvirt release that is in use in 4.12? TIA Igor
(In reply to Igor Bezukh from comment #2) > Hi, > > There are plans to move to UBI 9 (RHEL 9 based) in 4.12 > But currently 4.12 consumes UBI 8.6. So we will have to ask Libvirt folks to > do 2 backports for us - one to libvirt 8.0.0 for RHEL 8.6 and one to libvirt > 8.5.0 for RHEL 9 Can you please elaborate more, why you need it in RHEL-9.0, which is not being used by CNV? Backporting to RHEL-8.6.0 should be fine. RHEL-9.2 already contain the requested fix. Thanks.
Hi Jaroslav, Let me correct myself - we will need the backport for RHEL-8.6.0, we don't plan to switch our base containers to be RHEL-9 based in OCPV 4.12 Is there any action from our side regarding whats need to be done about the backport to RHEL-8.6.0?
Backports done, clearing needinfo.
Moving to ON_QA as we should consume now a fixed libvirt version.
The fixed libvirt is going to be available only later on, let's move back to MODIFIED and let's retarget this to CNV 4.12.3
QA: verified manually version v4.12.3-53 libvirt 8.0.0 metric output before downloading a large image: kubevirt_vmi_network_receive_bytes_total = 807184 kubevirt_vmi_network_transmit_bytes_total = 1142113887 output from virsh domstats: net.0.rx.bytes=807184 net.0.tx.bytes=1142113887 download a large image from inside the VM: [fedora@fedora-foh8ouye2h8my7wu ~]$ curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L % Total % Received % Xferd Average Speed Time Time Time Currentpub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 470M 100 470M 0 0 11.3M 0 0:00:41 0:00:41 --:--:-- 11.6M [fedora@fedora-foh8ouye2h8my7wu ~]$ metric output after downloading a large image: kubevirt_vmi_network_receive_bytes_total = 1188660 kubevirt_vmi_network_transmit_bytes_total = 1659433675 output from virsh domstats: net.0.rx.bytes=1188660 net.0.tx.bytes=1659433675
yes, right: with #2172578 we got the fix backported to el8.8 but on OpenShift Virtualization 4.12.z we are going to stay on el8.6 so I also filed https://bugzilla.redhat.com/show_bug.cgi?id=2192909
According to #2173977 the fix is already in el8.6 but with libvirt-8.0.0-5.8.module+el8.6.0+18297+b0096f4a while we are consuming libvirt-8.0.0-5.module+el8.6.0+14495+7194fa43.
Moving to ON_QE as underlying bug is Verified.
QA: Tested v4.12.4-37 Fixed metric output before downloading a large image: kubevirt_vmi_network_receive_bytes_total = 1679393181 kubevirt_vmi_network_transmit_bytes_total = 3189215 download a large image from inside the VM: curl -O https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.qcow2 -L % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 470M 100 470M 0 0 22.3M 0 0:00:20 0:00:20 --:--:-- 15.5M metric output after downloading a large image: kubevirt_vmi_network_receive_bytes_total = 2198780576 kubevirt_vmi_network_transmit_bytes_total = 4703974
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 4.12.4 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:3889