Bug 2003432

+++ This bug was initially created as a clone of Bug #1981946 +++

Description of problem:
The ip address and fqdn are getting to the ovirt-engine. The interface shows an "!" and says that the latest guest addition needs to be installed and running on the guest OS even though it is installed.

The vagrant plugin we use queries the API to sort out what IP it needs to communicate with vm's that are being brought up by our development team and fails because it can't sort out what IP the vm has.

Some vm's seem to show the correct information while others based on the same template do not. Migrating a vm from one host to another sometimes caused the ip to get into the system.

Version-Release number of selected component (if applicable):


How reproducible:
In my case, it is very easy to reproduce, but the issue does not always happen. If the server has been rebooted or vdsm has been restarted recently then things seem to work OK for a while.

Steps to Reproduce:
Up a vm from a previously known, good template.

Actual results:
IP intermittently does not appear in ovirt-engine.

Expected results:
IP and FQDN should be properly sent to ovirt-engine as qemu-guest-agent is installed

Additional info:
This affecting vms based on templates running Ubuntu 18.04/20.04 and Redhat 7/8.

--- Additional comment from RHEL Program Management on 2021-07-14 06:30:04 UTC ---

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

--- Additional comment from Tomáš Golembiovský on 2021-07-14 10:24:51 UTC ---

There is no obvious issue on VDSM side. The VMs are queried regularly but there is no guest info in the stats.
Once you notice this behavior again on the VM could you please share output of the following commands:

    virsh guestinfo noel-ovirt-biaa-11
    virsh domifaddr noel-ovirt-biaa-11 --source agent
    virsh qemu-agent-command noel-ovirt-biaa-11 '{"execute":"guest-network-get-interfaces"}'

Hopefully this will give us some clues.

--- Additional comment from GervaisdeM on 2021-07-15 20:59:54 UTC ---

The vm (noel-ovirt-biaa-11) that was not displaying an IP is now.

I ran the requested commands on several vms that were not displaying an IP in the web interface The last vm was one I spun up with vagrant and it is currently hanging and will eventually timeout.
Vagrant run stage: ==> i8manage_nopuppet_test_ov: Waiting for VM to get an IP address... 

Hope this is not too much, but ran this on multiple nodes with vms running different OS's in case there is a clue in there somewhere.

Thanks,
Gervais

[root ~]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # guestinfo noel-ovirt-dhs-22
user.count          : 0
os.id               : ubuntu
os.name             : Ubuntu
os.pretty-name      : Ubuntu 20.04.2 LTS
os.version          : 20.04.2 LTS (Focal Fossa)
os.version-id       : 20.04
os.machine          : x86_64
os.kernel-release   : 5.4.0-66-generic
os.kernel-version   : #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021
timezone.name       : UTC
timezone.offset     : 0
hostname            : noel-ovirt-dhs-22
fs.count            : 8
fs.0.name           : sda2
fs.0.mountpoint     : /
fs.0.fstype         : ext4
fs.0.total-bytes    : 500507766784
fs.0.used-bytes     : 28232536064
fs.0.disk.count     : 1
fs.0.disk.0.alias   : sda
fs.0.disk.0.serial  : 0QEMU_QEMU_HARDDISK_8a9e76e7-6827-4248-9dd4-29c3d1c9771c
fs.0.disk.0.device  : /dev/sda2
fs.1.name           : sda2
fs.1.mountpoint     : /sftp/sftp_user/ingest_data
fs.1.fstype         : ext4
fs.1.total-bytes    : 500507766784
fs.1.used-bytes     : 28232536064
fs.1.disk.count     : 1
fs.1.disk.0.alias   : sda
fs.1.disk.0.serial  : 0QEMU_QEMU_HARDDISK_8a9e76e7-6827-4248-9dd4-29c3d1c9771c
fs.1.disk.0.device  : /dev/sda2
fs.2.name           : loop0
fs.2.mountpoint     : /snap/core18/2074
fs.2.fstype         : squashfs
fs.2.total-bytes    : 58195968
fs.2.used-bytes     : 58195968
fs.2.disk.count     : 0
fs.3.name           : loop1
fs.3.mountpoint     : /snap/core18/1880
fs.3.fstype         : squashfs
fs.3.total-bytes    : 57671680
fs.3.used-bytes     : 57671680
fs.3.disk.count     : 0
fs.4.name           : loop2
fs.4.mountpoint     : /snap/lxd/16099
fs.4.fstype         : squashfs
fs.4.total-bytes    : 74842112
fs.4.used-bytes     : 74842112
fs.4.disk.count     : 0
fs.5.name           : loop3
fs.5.mountpoint     : /snap/lxd/20326
fs.5.fstype         : squashfs
fs.5.total-bytes    : 70909952
fs.5.used-bytes     : 70909952
fs.5.disk.count     : 0
fs.6.name           : loop5
fs.6.mountpoint     : /snap/snapd/12398
fs.6.fstype         : squashfs
fs.6.total-bytes    : 33947648
fs.6.used-bytes     : 33947648
fs.6.disk.count     : 0
fs.7.name           : loop4
fs.7.mountpoint     : /snap/snapd/8542
fs.7.fstype         : squashfs
fs.7.total-bytes    : 31457280
fs.7.used-bytes     : 31457280
fs.7.disk.count     : 0

virsh # domifaddr noel-ovirt-dhs-22 --source agent
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
 -          -                    ipv6         ::1/128
 enp1s0     56:6f:65:8b:00:09    ipv4         192.168.173.25/16
 -          -                    ipv6         fe80::546f:65ff:fe8b:9/64

virsh # qemu-agent-command noel-ovirt-dhs-22 '{"execute":"guest-network-get-interfaces"}'
{"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":853428,"tx-errs":0,"rx-bytes":427632008,"rx-dropped":0,"rx-packets":853428,"rx-errs":0,"tx-bytes":427632008,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"enp1s0","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"192.168.173.25","prefix":16},{"ip-address-type":"ipv6","ip-address":"fe80::546f:65ff:fe8b:9","prefix":64}],"statistics":{"tx-packets":62180,"tx-errs":0,"rx-bytes":128789706,"rx-dropped":171968,"rx-packets":1166301,"rx-errs":0,"tx-bytes":20379261,"tx-dropped":0},"hardware-address":"56:6f:65:8b:00:09"}]}

virsh # ^C
[root ~]# exit
Connection to ovirt1 closed.



ssh ovirt2
Last login: Thu Jul 15 19:49:35 2021 from 192.168.168.1

  node status: OK
  See `nodectl check` for more information

Admin Console: https://10.10.1.2:9090/ or https://192.168.168.11:9090/ or https://10.10.2.2:9090/


  node status: OK
  See `nodectl check` for more information

Admin Console: https://10.10.1.2:9090/ or https://192.168.168.11:9090/ or https://10.10.2.2:9090/


Welcome to ovirt2.dgi 
[root ~]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # guestinfo alexander-2004
user.count          : 0
os.id               : ubuntu
os.name             : Ubuntu
os.pretty-name      : Ubuntu 20.04.2 LTS
os.version          : 20.04.2 LTS (Focal Fossa)
os.version-id       : 20.04
os.machine          : x86_64
os.kernel-release   : 5.4.0-66-generic
os.kernel-version   : #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021
timezone.name       : UTC
timezone.offset     : 0
hostname            : vagrantbase-ubuntu2004
fs.count            : 7
fs.0.name           : sda2
fs.0.mountpoint     : /
fs.0.fstype         : ext4
fs.0.total-bytes    : 500507766784
fs.0.used-bytes     : 15814242304
fs.0.disk.count     : 1
fs.0.disk.0.alias   : sda
fs.0.disk.0.serial  : 0QEMU_QEMU_HARDDISK_c1bcd399-bbcb-4758-a8a1-4632d79e5d1e
fs.0.disk.0.device  : /dev/sda2
fs.1.name           : loop1
fs.1.mountpoint     : /snap/lxd/16099
fs.1.fstype         : squashfs
fs.1.total-bytes    : 74842112
fs.1.used-bytes     : 74842112
fs.1.disk.count     : 0
fs.2.name           : loop0
fs.2.mountpoint     : /snap/core18/1880
fs.2.fstype         : squashfs
fs.2.total-bytes    : 57671680
fs.2.used-bytes     : 57671680
fs.2.disk.count     : 0
fs.3.name           : loop2
fs.3.mountpoint     : /snap/snapd/8542
fs.3.fstype         : squashfs
fs.3.total-bytes    : 31457280
fs.3.used-bytes     : 31457280
fs.3.disk.count     : 0
fs.4.name           : loop3
fs.4.mountpoint     : /snap/snapd/12398
fs.4.fstype         : squashfs
fs.4.total-bytes    : 33947648
fs.4.used-bytes     : 33947648
fs.4.disk.count     : 0
fs.5.name           : loop4
fs.5.mountpoint     : /snap/core18/2074
fs.5.fstype         : squashfs
fs.5.total-bytes    : 58195968
fs.5.used-bytes     : 58195968
fs.5.disk.count     : 0
fs.6.name           : loop5
fs.6.mountpoint     : /snap/lxd/20326
fs.6.fstype         : squashfs
fs.6.total-bytes    : 70909952
fs.6.used-bytes     : 70909952
fs.6.disk.count     : 0

virsh # domifaddr alexander-2004 --source agent
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
 -          -                    ipv6         ::1/128
 enp1s0     56:6f:65:8b:00:0c    ipv4         192.168.173.28/16
 -          -                    ipv6         fe80::546f:65ff:fe8b:c/64

virsh # qemu-agent-command alexander-2004 '{"execute":"guest-network-get-interfaces"}'
{"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":637,"tx-errs":0,"rx-bytes":60896,"rx-dropped":0,"rx-packets":637,"rx-errs":0,"tx-bytes":60896,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"enp1s0","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"192.168.173.28","prefix":16},{"ip-address-type":"ipv6","ip-address":"fe80::546f:65ff:fe8b:c","prefix":64}],"statistics":{"tx-packets":52758,"tx-errs":0,"rx-bytes":433033164,"rx-dropped":447839,"rx-packets":2944392,"rx-errs":0,"tx-bytes":6164774,"tx-dropped":0},"hardware-address":"56:6f:65:8b:00:0c"}]}

virsh # exit

[root ~]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # guestinfo alexandria2
user.count          : 0
os.id               : centos
os.name             : CentOS Stream
os.pretty-name      : CentOS Stream 8
os.version          : 8
os.version-id       : 8
os.machine          : x86_64
os.kernel-release   : 4.18.0-315.el8.x86_64
os.kernel-version   : #1 SMP Mon Jun 28 19:09:44 UTC 2021
timezone.name       : ADT
timezone.offset     : -10800
hostname            : alexandria.dgi
fs.count            : 2
fs.0.name           : dm-0
fs.0.mountpoint     : /
fs.0.fstype         : xfs
fs.0.total-bytes    : 29827903488
fs.0.used-bytes     : 3050864640
fs.0.disk.count     : 1
fs.0.disk.0.alias   : vda
fs.0.disk.0.serial  : d4b74174-3778-4fee-9
fs.0.disk.0.device  : /dev/vda2
fs.1.name           : vda1
fs.1.mountpoint     : /boot
fs.1.fstype         : ext4
fs.1.total-bytes    : 952840192
fs.1.used-bytes     : 299028480
fs.1.disk.count     : 1
fs.1.disk.0.alias   : vda
fs.1.disk.0.serial  : d4b74174-3778-4fee-9
fs.1.disk.0.device  : /dev/vda1
disk.count          : 6
disk.0.name         : /dev/vda1
disk.0.partition    : yes
disk.0.dependency.count: 1
disk.0.dependency.0.name: /dev/vda
disk.1.name         : /dev/vda2
disk.1.partition    : yes
disk.1.dependency.count: 1
disk.1.dependency.0.name: /dev/vda
disk.2.name         : /dev/vda
disk.2.partition    : no
disk.2.serial       : d4b74174-3778-4fee-9
disk.2.alias        : vda
disk.3.name         : /dev/dm-0
disk.3.partition    : no
disk.3.dependency.count: 1
disk.3.dependency.0.name: /dev/vda2
disk.3.guest_alias  : cl-root
disk.4.name         : /dev/sr0
disk.4.partition    : no
disk.4.serial       : QEMU_DVD-ROM_QM00005
disk.4.alias        : sdc
disk.5.name         : /dev/dm-1
disk.5.partition    : no
disk.5.dependency.count: 1
disk.5.dependency.0.name: /dev/vda2
disk.5.guest_alias  : cl-swap

virsh # domifaddr alexandria2 --source agent
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
 -          -                    ipv6         ::1/128
 enp1s0     00:50:56:bf:47:53    ipv4         192.168.180.135/16
 -          -                    ipv6         fe80::9b49:b62d:1800:49d3/64

virsh # qemu-agent-command alexandria2 '{"execute":"guest-network-get-interfaces"}'
{"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":652,"tx-errs":0,"rx-bytes":56859,"rx-dropped":0,"rx-packets":652,"rx-errs":0,"tx-bytes":56859,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"enp1s0","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"192.168.180.135","prefix":16},{"ip-address-type":"ipv6","ip-address":"fe80::9b49:b62d:1800:49d3","prefix":64}],"statistics":{"tx-packets":755882,"tx-errs":0,"rx-bytes":2107436985,"rx-dropped":95063,"rx-packets":2266605,"rx-errs":0,"tx-bytes":97774500,"tx-dropped":0},"hardware-address":"00:50:56:bf:47:53"}]}

ssh ovirt2          
Last login: Thu Jul 15 20:19:19 2021 from 192.168.168.1

  node status: OK
  See `nodectl check` for more information

Admin Console: https://10.10.1.2:9090/ or https://192.168.168.11:9090/ or https://10.10.2.2:9090/

^[[A
  node status: OK
  See `nodectl check` for more information

Admin Console: https://10.10.1.2:9090/ or https://192.168.168.11:9090/ or https://10.10.2.2:9090/


Welcome to ovirt2.dgi 
[root ~]# virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # guestinfo gervais-ovirt-i8manage-test
user.count          : 0
os.id               : ubuntu
os.name             : Ubuntu
os.pretty-name      : Ubuntu 20.04.2 LTS
os.version          : 20.04.2 LTS (Focal Fossa)
os.version-id       : 20.04
os.machine          : x86_64
os.kernel-release   : 5.4.0-66-generic
os.kernel-version   : #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021
timezone.name       : UTC
timezone.offset     : 0
hostname            : vagrantbase-ubuntu2004
fs.count            : 4
fs.0.name           : sda2
fs.0.mountpoint     : /
fs.0.fstype         : ext4
fs.0.total-bytes    : 500507766784
fs.0.used-bytes     : 4842360832
fs.0.disk.count     : 1
fs.0.disk.0.alias   : sda
fs.0.disk.0.serial  : 0QEMU_QEMU_HARDDISK_eafaac44-c1ca-4ee4-81f4-2b0db5a88f46
fs.0.disk.0.device  : /dev/sda2
fs.1.name           : loop2
fs.1.mountpoint     : /snap/lxd/16099
fs.1.fstype         : squashfs
fs.1.total-bytes    : 74842112
fs.1.used-bytes     : 74842112
fs.1.disk.count     : 0
fs.2.name           : loop0
fs.2.mountpoint     : /snap/core18/1880
fs.2.fstype         : squashfs
fs.2.total-bytes    : 57671680
fs.2.used-bytes     : 57671680
fs.2.disk.count     : 0
fs.3.name           : loop1
fs.3.mountpoint     : /snap/snapd/8542
fs.3.fstype         : squashfs
fs.3.total-bytes    : 31457280
fs.3.used-bytes     : 31457280
fs.3.disk.count     : 0

virsh # domifaddr gervais-ovirt-i8manage-test --source agent
 Name       MAC address          Protocol     Address
-------------------------------------------------------------------------------
 lo         00:00:00:00:00:00    ipv4         127.0.0.1/8
 -          -                    ipv6         ::1/128
 enp1s0     56:6f:65:8b:00:67    ipv4         192.168.173.119/16
 -          -                    ipv6         fe80::546f:65ff:fe8b:67/64

virsh # qemu-agent-command gervais-ovirt-i8manage-test '{"execute":"guest-network-get-interfaces"}'
{"return":[{"name":"lo","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"127.0.0.1","prefix":8},{"ip-address-type":"ipv6","ip-address":"::1","prefix":128}],"statistics":{"tx-packets":132,"tx-errs":0,"rx-bytes":10980,"rx-dropped":0,"rx-packets":132,"rx-errs":0,"tx-bytes":10980,"tx-dropped":0},"hardware-address":"00:00:00:00:00:00"},{"name":"enp1s0","ip-addresses":[{"ip-address-type":"ipv4","ip-address":"192.168.173.119","prefix":16},{"ip-address-type":"ipv6","ip-address":"fe80::546f:65ff:fe8b:67","prefix":64}],"statistics":{"tx-packets":11345,"tx-errs":0,"rx-bytes":122338065,"rx-dropped":66,"rx-packets":80174,"rx-errs":0,"tx-bytes":926907,"tx-dropped":0},"hardware-address":"56:6f:65:8b:00:67"}]}

--- Additional comment from GervaisdeM on 2021-07-23 12:51:41 UTC ---

@

--- Additional comment from GervaisdeM on 2021-07-23 12:58:10 UTC ---

Apologies for the above comment.

This issue is still huge for our organization. Please let me know if there is anything else I can do to help move this issue along.

--- Additional comment from Tomáš Golembiovský on 2021-07-27 16:46:28 UTC ---

I went through the provided data several times and I still don't see where could the stats get lost. The only thing I noticed that the VMs were migrated to that host (and not started fresh). Do you experience the behavior only after a migration?

--- Additional comment from GervaisdeM on 2021-07-27 19:38:07 UTC ---

Hi Tomáš,

It does not work on boxes that are started fresh on a host. If I put a host into maintenance mode and then activate it, vm's started on that host show an IP properly. If I migrate a vm that is not showing its IP to the newly reactivated host, it displays its IP.

Hosts that have active VM's running for any length of time eventually no longer consistently display their IP's in the interface. The odd one or two show an IP. It is not always the same VM that continues to display an IP and I can not find a pattern.

If I migrate a vm off of the host that was recently put into maintenance mode and activated to a host that has been running untouched for some time, the vm immediately stops displaying its IP.

This includes the HostedEngine.

--- Additional comment from GervaisdeM on 2021-07-28 18:16:30 UTC ---

My HostedEngine is still running CentOS 8.

Does this have any bearing on this issue?
Should I move it to CentOS Stream?

--- Additional comment from Arik on 2021-08-08 16:23:48 UTC ---

We fail to reproduce this issue and while we have the relevant logs, it's still quite difficult to understand what's going on.
Assuming there are still VMs with no guest agent information (ip addresses and fqdn) in your environment, could you please pick one of them and:
1. Provide the output of getStats
2. Specify the exact time getStats was triggered
3. Provide VDSM log from that host that covers this point in time and few hours back

--- Additional comment from GervaisdeM on 2021-08-09 14:48:32 UTC ---

The issue definitely continues. We have 74 vm's running in our cluster and about 1/3 have no IP and FQDN being displayed in the oVirt engine. 

We have 3 hosts in our cluster (ovirt1, ovirt2, ovirt3 - I know crazy naming scheme). I pushed most of the vm's off to ovirt2 and ovirt3. oVirt1 has only 8 vm's running on it and they all show the correct information in the engine. If a new box is brought up by one of our developers, it almost always comes up on oVirt1 (because it has the least vm's) and it does come up ok and display the correct information in the ovirt-engine. If for some reason, the box starts on ovirt2 or ovirt3, I can force it to migrate to ovirt1 and it will display the IP and fqdn properly. As soon as I migrate that new vm to ovirt2 or ovirt3 (the hosts with the majority of the vms running on them), the ip and fqdn are not longer displayed in the engine interface.

I'm attaching vdsm logs and output of getStats as requested. the file is called ovirt-logs-2021-08-09.zip
getStats was run at Mon Aug  9 13:21:37 UTC 2021

--- Additional comment from GervaisdeM on 2021-08-09 14:50:19 UTC ---



--- Additional comment from Tomáš Golembiovský on 2021-08-10 06:41:53 UTC ---

I may have already asked this, but I cannot find this in a bug. Are all the hosts up to date? What are the versions of vdsm, libvirt and qemu?

In previous attachment you posted a getStats for a VM on broken host (as requested). For comparison, could you please share also result of getStats for the same VM but running on ovirt1 when it has all the guest information visible in engine?

It pretty much looks like a host issue, but let's double check it is not related to bug 1901572. Please try clearing the hash for NICs info in engine, like in:

    https://bugzilla.redhat.com/show_bug.cgi?id=1901572#c16

Will this fix the issue or change anything?

--- Additional comment from GervaisdeM on 2021-08-10 20:19:10 UTC ---

Clearing the hash did not seem to have any effect.

Output from getStats on same vm after migrating it to ovirt1

```
[root ~]# vdsm-client VM getStats vmID=72ae23f6-7487-42e1-9ceb-dcf189a6f264
[
    {
        "acpiEnable": "true",
        "appsList": [
            "kernel-3.10.0-1160.21.1.el7.x86_64",
            "qemu-guest-agent-2.12.0"
        ],
        "balloonInfo": {
            "balloon_cur": "8388608",
            "balloon_max": "8388608",
            "balloon_min": "8388608",
            "balloon_target": "8388608",
            "ballooning_enabled": true
        },
        "clientIp": "",
        "cpuSys": "0.27",
        "cpuUsage": "6530000000",
        "cpuUser": "0.44",
        "disks": {
            "sdc": {
                "apparentsize": "0",
                "flushLatency": "0",
                "readBytes": "0",
                "readLatency": "0",
                "readOps": "0",
                "readRate": "0.0",
                "truesize": "0",
                "writeLatency": "0",
                "writeOps": "0",
                "writeRate": "0.0",
                "writtenBytes": "0"
            },
            "vda": {
                "apparentsize": "200540160",
                "flushLatency": "1413051.5",
                "imageID": "563abb58-e91a-44da-9382-9a850fff0c4e",
                "readBytes": "0",
                "readLatency": "0",
                "readOps": "0",
                "readRate": "0.0",
                "truesize": "214777344",
                "writeLatency": "21563795.0",
                "writeOps": "1",
                "writeRate": "102.40785782986511",
                "writtenBytes": "1536"
            }
        },
        "disksUsage": [],
        "displayInfo": [
            {
                "ipAddress": "192.168.168.13",
                "port": "5909",
                "tlsPort": "",
                "type": "vnc"
            }
        ],
        "elapsedTime": "962703",
        "guestCPUCount": -1,
        "guestFQDN": "gervais-centos7-test-php56.dgi",
        "guestIPs": "",
        "guestName": "gervais-centos7-test-php56.dgi",
        "guestOs": "3.10.0-1160.21.1.el7.x86_64",
        "guestOsInfo": {
            "arch": "x86_64",
            "codename": "",
            "distribution": "CentOS Linux",
            "kernel": "3.10.0-1160.21.1.el7.x86_64",
            "type": "linux",
            "version": "7"
        },
        "guestTimezone": {
            "offset": -180,
            "zone": "ADT"
        },
        "hash": "-5327289600703475597",
        "kvmEnable": "true",
        "memUsage": "10",
        "memoryStats": {
            "majflt": 0,
            "mem_free": "7491276",
            "mem_total": "8006768",
            "mem_unused": "7491276",
            "minflt": 2,
            "pageflt": 2,
            "swap_in": 0,
            "swap_out": 0
        },
        "monitorResponse": "0",
        "netIfaces": [
            {
                "hw": "00:00:00:00:00:00",
                "inet": [
                    "127.0.0.1"
                ],
                "inet6": [
                    "::1"
                ],
                "name": "lo"
            },
            {
                "hw": "56:6f:65:8b:00:36",
                "inet": [
                    "192.168.173.70"
                ],
                "inet6": [
                    "fe80::5547:5c32:3c7c:842e"
                ],
                "name": "eth0"
            }
        ],
        "network": {
            "vnet26": {
                "macAddr": "56:6f:65:8b:00:36",
                "name": "vnet26",
                "rx": "28134",
                "rxDropped": "0",
                "rxErrors": "0",
                "sampleTime": 523369.025767464,
                "speed": "1000",
                "state": "unknown",
                "tx": "940",
                "txDropped": "0",
                "txErrors": "0"
            }
        },
        "session": "Unknown",
        "status": "Up",
        "statusTime": "2670852671",
        "timeOffset": "0",
        "username": "",
        "vcpuCount": "1",
        "vcpuPeriod": 100000,
        "vcpuQuota": "-1",
        "vmId": "72ae23f6-7487-42e1-9ceb-dcf189a6f264",
        "vmJobs": {},
        "vmName": "gervais-centos7-test-php56",
        "vmType": "kvm"
    }
]
```

Everything is up to date:

```
[root ~]# dnf clean all ; dnf upgrade
99 files removed
Ceph packages for x86_64                                                                                                                      176 kB/s | 283 kB     00:01    
dell-system-update_independent                                                                                                                144 kB/s | 150 kB     00:01    
dell-system-update_dependent                                                                                                                  155 kB/s | 195 kB     00:01    
oVirt Node Optional packages from CentOS Stream 8 - BaseOS                                                                                    6.7 MB/s | 8.8 MB     00:01    
oVirt Node Optional packages from CentOS Stream 8 - AppStream                                                                                 522 kB/s |  12 MB     00:24    
Latest oVirt 4.4 Release                                                                                                                      1.3 MB/s | 2.2 MB     00:01    
Extra Packages for Enterprise Linux 8 - x86_64                                                                                                7.6 MB/s |  10 MB     00:01    
CentOS-8 - Gluster 8                                                                                                                          157 kB/s |  64 kB     00:00    
virtio-win builds roughly matching what will be shipped in upcoming RHEL                                                                      102 kB/s |  66 kB     00:00    
Copr repo for EL8_collection owned by sbonazzo                                                                                                615 kB/s | 402 kB     00:00    
Copr repo for gluster-ansible owned by sac                                                                                                     28 kB/s | 7.3 kB     00:00    
Copr repo for ovsdbapp owned by mdbarroso                                                                                                     6.9 kB/s | 2.0 kB     00:00    
Advanced Virtualization CentOS Stream packages for x86_64                                                                                     230 kB/s | 153 kB     00:00    
CentOS-8 Stream - oVirt 4.4                                                                                                                   1.0 MB/s | 928 kB     00:00    
CentOS-8 - OpsTools - collectd                                                                                                                287 kB/s | 148 kB     00:00    
CentOS-8 - NFV OpenvSwitch                                                                                                                     92 kB/s |  45 kB     00:00    
OpenStack Train Repository                                                                                                                    661  B/s | 257  B     00:00    
Dependencies resolved.
Nothing to do.
Complete!
```

Versions I am running:

```
[root ~]# rpm -qa | grep vdsm
vdsm-client-4.40.70.6-1.el8.noarch
vdsm-common-4.40.70.6-1.el8.noarch
vdsm-http-4.40.70.6-1.el8.noarch
vdsm-4.40.70.6-1.el8.x86_64
vdsm-gluster-4.40.70.6-1.el8.x86_64
vdsm-network-4.40.70.6-1.el8.x86_64
vdsm-jsonrpc-4.40.70.6-1.el8.noarch
vdsm-python-4.40.70.6-1.el8.noarch
vdsm-api-4.40.70.6-1.el8.noarch
vdsm-yajsonrpc-4.40.70.6-1.el8.noarch

[root ~]# rpm -qa libvirt
libvirt-7.4.0-1.el8s.x86_64

[root ~]# rpm -qa | grep qemu
qemu-kvm-block-ssh-5.2.0-16.el8s.x86_64
qemu-kvm-block-gluster-5.2.0-16.el8s.x86_64
qemu-guest-agent-5.2.0-16.el8s.x86_64
qemu-img-5.2.0-16.el8s.x86_64
qemu-kvm-common-5.2.0-16.el8s.x86_64
qemu-kvm-block-iscsi-5.2.0-16.el8s.x86_64
libvirt-daemon-driver-qemu-7.4.0-1.el8s.x86_64
qemu-kvm-ui-opengl-5.2.0-16.el8s.x86_64
qemu-kvm-block-rbd-5.2.0-16.el8s.x86_64
qemu-kvm-ui-spice-5.2.0-16.el8s.x86_64
qemu-kvm-block-curl-5.2.0-16.el8s.x86_64
qemu-kvm-5.2.0-16.el8s.x86_64
ipxe-roms-qemu-20181214-8.git133f4c47.el8.noarch
qemu-kvm-core-5.2.0-16.el8s.x86_64
```

--- Additional comment from GervaisdeM on 2021-09-01 14:07:14 UTC ---

I've noticed that there have been some small updates to packages and have applied them. None of them have made any difference to the issue described on this ticket. 

Is there anything I can do on my end? Should I do a reinstall of my hosts from the UI in the ovirt-engine? Could that make the situation worse?

--- Additional comment from Tomáš Golembiovský on 2021-09-02 08:51:30 UTC ---

(In reply to GervaisdeM from comment #14)
> I've noticed that there have been some small updates to packages and have
> applied them. None of them have made any difference to the issue described
> on this ticket. 
> 
> Is there anything I can do on my end? Should I do a reinstall of my hosts
> from the UI in the ovirt-engine? Could that make the situation worse?

Yes, I wanted to suggest reinstalling the host. So if you can please try that.

Also, compare configuration in /etc/vdsm if there are any differences between the host that works OK and the faulty hosts.

--- Additional comment from GervaisdeM on 2021-09-07 20:25:50 UTC ---

I haven't gotten to reinstalling yet. Will try to one evening this week. Before I do, should I have any concerns about reinstalling? My cluster is hyperconverged with all three servers being being part of the underlying gluster filesystems that I need for vm's and hosted engine. Anything I need to be aware of before running the reinstall?

All three hosts in my cluster are affected by this issue. The only difference in my /etc/vdsm directories on my servers is the vdsm.id

Example:
[root ~]# scp -r ovirt1:/etc/vdsm .
svdsm.logger.conf                                                                                                                           100%  734   124.2KB/s   00:00    
mom.conf                                                                                                                                    100% 3149     3.9MB/s   00:00    
vdsm                                                                                                                                        100%  159   177.3KB/s   00:00    
00-defines.policy                                                                                                                           100%  323    18.6KB/s   00:00    
04-cputune.policy                                                                                                                           100% 2276     3.1MB/s   00:00    
02-balloon.policy                                                                                                                           100% 5369     6.2MB/s   00:00    
05-iotune.policy                                                                                                                            100% 1105     1.6MB/s   00:00    
03-ksm.policy                                                                                                                               100% 4651   621.0KB/s   00:00    
01-parameters.policy                                                                                                                        100%    1     1.8KB/s   00:00    
vdsm.conf                                                                                                                                   100%   56    94.5KB/s   00:00    
logger.conf                                                                                                                                 100% 2167     3.2MB/s   00:00    
vdsm.id                                                                                                                                     100%   36    49.5KB/s   00:00    
[root ~]# diff -r /etc/vdsm vdsm
diff -r /etc/vdsm/vdsm.id vdsm/vdsm.id
1c1
< 4c4c4544-0056-3510-8056-c2c04f383432
\ No newline at end of file
---
> 4c4c4544-0056-3410-8057-c2c04f383432
\ No newline at end of file

--- Additional comment from GervaisdeM on 2021-09-10 14:36:18 UTC ---

There was another minor upgrade showing for my hosts... I applied the update to all of my hosts and then ran the reinstall one at a time for all three oVirt hosts.

This is exactly what I did:

1. Put first server into maintenance mode.
2. Selected Installation -> Reinstall from the menu
3. Left everything as the default, except set Hosted Engine -> Deploy

I did that for all three ovirt hosts.

My issue remains exactly the same. Is there something in my process above that I should change?