Description of problem: RHV-M is not displaying the guest agent information of the HostedEngine VM. The data is collected correctly at the host and it's in the database, but it's not displayed. Version-Release number of selected component (if applicable): ovirt-engine-4.2.7.4-0.1.el7ev.noarch How reproducible: Always in this environment. Steps to Reproduce: 1. In the GUI, check the IP address of the HostedEngine VM Actual results: It's empty Expected results: IP address Additional info: The guest agent information is collected at the host without problems: /var/log/vdsm/vdsm.log 2020-10-23 11:19:21,612+0000 INFO (jsonrpc/2) [api.virt] START getStats() from=::1,43322, vmId=2f42f1ba-3425-45e1-9254-f5683e9d319f (api:46) 2020-10-23 11:19:21,613+0000 INFO (jsonrpc/2) [api.virt] FINISH getStats return={'status': {'message': 'Done', 'code': 0}, 'statsList': [{'displayInfo': [{'tlsPort': '5900', 'ipAddress': '0', 'type': u'spice', 'port': '-1'}], 'memUsage': '30', 'acpiEnable': 'true', 'guestFQDN': u'rhvm.example.com', 'vmId': '2f42f1ba-3425-45e1-9254-f5683e9d319f', 'session': 'Unknown', 'displaySecurePort': '5900', 'timeOffset': '0', 'balloonInfo': {'balloon_max': '16777216', 'balloon_min': '0', 'balloon_target': '16777216', 'balloon_cur': '16777216'}, 'disksUsage': [{u'path': u'/', u'total': '64412704768', u'fs': u'xfs', u'used': '11033858048'}], 'network': {'vnet0': {'macAddr': u'aa:bb:cc:dd:ee:ff', 'rxDropped': '0', 'tx': '5676426027', 'rxErrors': '0', 'txDropped': '0', 'rx': '653179338', 'txErrors': '0', 'state': 'unknown', 'sampleTime': 4450647.84, 'speed': '1000', 'name': 'vnet0'}}, 'vmJobs': {}, 'cpuUser': '6.01', 'elapsedTime': '161374', 'memoryStats': {'swap_out': '0', 'majflt': '0', 'mem_cached': '682748', 'mem_free': '10930624', 'mem_buffers': '0', 'swap_in': '0', 'pageflt': '358', 'mem_total': '16264528', 'mem_unused': '10930624'}, 'cpuSys': '1.07', 'appsList': (u'kernel-3.10.0-514.21.1.el7', u'cloud-init-18.2-1.el7_6.1', u'ovirt-guest-agent-common-1.0.14-3.el7ev', u'kernel-3.10.0-957.el7', u'kernel-3.10.0-514.16.1.el7'), 'guestOs': u'3.10.0-957.el7.x86_64', 'vmName': 'HostedEngine', 'displayType': 'qxl', 'vcpuCount': '4', 'clientIp': u'', 'hash': '-1609165203876588203', 'guestCPUCount': 4, 'vmType': 'kvm', 'displayIp': '0', 'cpuUsage': '1428770000000', 'vcpuPeriod': 100000L, 'displayPort': '-1', 'guestTimezone': {u'zone': u'zoneinfo/GMT', u'offset': 0}, 'vcpuQuota': '-1', 'statusTime': '4450647840', 'kvmEnable': u'true', 'disks': {'vda': {'readLatency': '0', 'flushLatency': '80043', 'readRate': '0.0', 'writeRate': '46486.4576377', 'writtenBytes': '5688982016', 'truesize': '64424509440', 'apparentsize': '64424509440', 'readOps': '7986', 'writeLatency': '1382774', 'imageID': u'7260cc92-6d69-405a-8682-21c22fe67872', 'readBytes': '423696384', 'writeOps': '435610'}, 'hdc': {'readLatency': '0', 'flushLatency': '0', 'readRate': '0.0', 'writeRate': '0.0', 'writtenBytes': '0', 'truesize': '0', 'apparentsize': '0', 'readOps': '0', 'writeLatency': '0', 'readBytes': '0', 'writeOps': '0'}}, 'monitorResponse': '0', 'guestOsInfo': {u'kernel': u'3.10.0-957.el7.x86_64', u'arch': u'x86_64', u'version': u'7.6', u'distribution': u'Red Hat Enterprise Linux Server', u'type': u'linux', u'codename': u'Maipo'}, 'username': u'None', 'guestName': u'rhvm.example.com', 'status': 'Up', 'lastLogin': 1603385669.287476, 'guestIPs': u'192.168.1.2', 'guestContainers': [], 'netIfaces': [{u'inet6': [u'fe80::abcd'], u'hw': u'aa:bb:cc:dd:ee:ff', u'inet': [u'192.168.1.2'], u'name': u'eth0'}]}]} from=::1,43322, vmId=2f42f1ba-3425-45e1-9254-f5683e9d319f (api:52) The guest IP address is in the database: engine=> select vm_id,interface_name,ipv4_addresses from vm_guest_agent_interfaces where vm_id in (select vm_guid from vms where vm_name='HostedEngine'); vm_id | interface_name | ipv4_addresses --------------------------------------+----------------+---------------- 2f42f1ba-3425-45e1-9254-f5683e9d319f | eth0 | 192.168.1.2
(In reply to Juan Orti Alcaine from comment #0) > Version-Release number of selected component (if applicable): > ovirt-engine-4.2.7.4-0.1.el7ev.noarch Please upgrade the manager to the current GA version, 4.4. Other than that, this shouldn't be blocking, right? It's supposedly the IP you're using to access the portal in a first place.
Does this still reproduce in 4.4?
Doesn't reproduced on rhvm-4.4.4.2-0.1.el8ev.noarch.
(In reply to Nikolai Sednev from comment #6) > Doesn't reproduced on rhvm-4.4.4.2-0.1.el8ev.noarch. Double checked now on fresh and latest version:4.4.3.12-0.1.el8ev and IP address is not reported for the engine VM. Screenshot is attached.
Created attachment 1736973 [details] Screenshot from 2020-12-06 16-38-45.png
I used the qemu-guest-agent-4.2.0-34.module+el8.3.0+7976+077be4ec.x86_64 on the engine.
Nikolai, Please try with downgrading to http://pulp-read.dist.prod.ext.phx2.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/q/qemu-kvm-2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64.rpm
alma03 ~]# yum downgrade -y http://pulp-read.dist.prod.ext.phx2.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/Packages/q/qemu-kvm-2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64.rpm Updating Subscription Management repositories. Unable to read consumer identity This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Last metadata expiration check: 0:05:12 ago on Mon 14 Dec 2020 11:45:09 AM IST. qemu-kvm-2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64.rpm 122 kB/s | 102 kB 00:00 Error: Problem: conflicting requests - nothing provides qemu-kvm-block-curl = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 - nothing provides qemu-kvm-block-gluster = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 - nothing provides qemu-kvm-block-iscsi = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 - nothing provides qemu-kvm-block-rbd = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 - nothing provides qemu-kvm-block-ssh = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 - nothing provides qemu-kvm-core = 15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4 needed by qemu-kvm-15:2.12.0-99.module+el8.2.0+7988+c1d02dbb.4.x86_64 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Created attachment 1743449 [details] HE444Screenshot Looks ok with RHV-M 4.4.4.6
Not working on these components: rhvm-4.4.4.6-0.1.el8ev.noarch qemu-guest-agent-4.2.0-34.module+el8.3.0+9094+80c20d97.2.x86_64 Screenshot attached.
Created attachment 1743454 [details] Screenshot from 2020-12-31 18-01-00.png
what do you get with: select vm_ip from vms where vm_name='HostedEngine'; when the IP is not shown? if that returns an empty string, does it change by executing: update vm_dynamic set guest_agent_nics_hash = NULL where vm_guid in (select vm_guid from vm_static where vm_name='HostedEngine'); ?
engine=# select vm_ip from vms where vm_name='HostedEngine'; vm_ip ------- (1 row) engine-# update vm_dynamic set guest_agent_nics_hash = NULL where vm_guid in (select vm_guid from vm_static where vm_name='HostedEngine'); ERROR: syntax error at or near "vm_ip" LINE 1: vm_ip ^ engine=#
I just reran it again now and got these: nsednev-he-1 ~]# su - postgres -c psql psql (12.5) Type "help" for help. postgres=# \c engine You are now connected to database "engine" as user "postgres". engine=# select vm_ip from vms where vm_name='HostedEngine'; vm_ip ------------------------------------------------------------------------ 10.35.92.51 2620:52:0:235c:216:3eff:fe7b:b853 fe80::216:3eff:fe7b:b853 (1 row) engine=# update vm_dynamic set guest_agent_nics_hash = NULL where vm_guid in (select vm_guid from vm_static where vm_name='HostedEngine'); UPDATE 1 engine=# For some reason the engine now is reporting the IP as appears in the attachment.
Created attachment 1745971 [details] Screenshot from 2021-01-10 11-19-52.png
Looks like "update vm_dynamic set guest_agent_nics_hash = NULL where vm_guid in (select vm_guid from vm_static where vm_name='HostedEngine');" has triggered the change.
Thanks Nikolai So it appears like we have some kind of race or we don't clear the guest_agent_nics_hash field in some cases
(In reply to Arik from comment #21) > So it appears like we have some kind of race or we don't clear the guest_agent_nics_hash field in some cases Actually looking again at comment 18, I see that vm_ip was set in the database before the hash was cleared so clearing the hash had no impact in this case - what is displayed in the screenshot attached in comment 19 must have been displayed also right before the hash was cleared
I suspected that it's the same as bz 1981946 where the hosted engine VM is also mentioned but I managed to reproduce the scenario described in comment 0 in which the NICs are set with IP address(es) and vm_ip is not displayed: 1. the hosted engine VM is stopped on a first host and moves to Down state 2. the agent starts the hosted engine VM on a second host 3. ovirt-engine detects the VM as Down on the first host and calls 'destroy' + set the VM with Down state in the database which means vm_ip + guest_agent_nics_hash also get cleared 4. ovirt-engine detects the VM "unexpectedly" on the second host and therefore saves it in the database with the new guest_agent_nics_hash and without updating vm_ip 5. as a result, the information from the guest agent on the NICs exists but vm_ip is never set as the reported hash never changes again
Version 4.4.8.3-0.10.el8ev works fine.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHV Manager (ovirt-engine) [ovirt-4.4.8]), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3460
Due to QE capacity we are not going to cover this issue in our automation