Description of problem: Failed to read metadata from storage after adding additional host. Version-Release number of selected component (if applicable): Host: qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64 ovirt-vmconsole-host-1.0.0-1.el7ev.noarch ovirt-host-deploy-1.4.1-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.2.x86_64 vdsm-4.17.13-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 ovirt-setup-lib-1.0.0-1.el7ev.noarch ovirt-vmconsole-1.0.0-1.el7ev.noarch ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch mom-0.5.1-1.el7ev.noarch ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015 How reproducible: 100% Steps to Reproduce: 1.Try adding additional host to HE environment. 2. 3. Actual results: Third in my case host was not shown within the "hosted-engine --vm-status" and this error was received: - "Dec 15 16:06:48 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata' ". Expected results: Additional host should be seen via "hosted-engine --vm-status" and should see the storage metadata. Additional info: Sosreport from the host is attached.
Created attachment 1106040 [details] sosreport from host
The host passed the https://bugzilla.redhat.com/show_bug.cgi?id=1291234 and was added in to the HE-WEBUI, was clearly seen there as active, but not even single VM could be migrated to it. Dec 15 16:06:21 seal09.qa.lab.tlv.redhat.com ovirt-ha-agent[32609]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Request failed: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f'' - trying to restart agent Dec 15 16:06:38 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f' # hosted-engine --vm-status Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 117, in <module> if not status_checker.print_status(): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 60, in print_status all_host_stats = ha_cli.get_all_host_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats return self.get_all_stats(self.StatModes.HOST) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats stats = broker.get_stats_from_storage(service) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 232, in get_stats_from_storage result = self._checked_communicate(request) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 260, in _checked_communicate .format(message or response)) ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata'
If both hosts rebooted, then third host being shown via "hosted-engine --vm-status".
Closing this bug according to comment 3. it looks like a environment issue, please reopen if you see this again.
(In reply to Fabian Deutsch from comment #4) > Closing this bug according to comment 3. > > it looks like a environment issue, please reopen if you see this again. It's not an environment issue, because the first time you're adding a new host, third in my case, it's not being shown if running "hosted-engine --vm-status", until previously existing hosts rebooted. Have you tried reproducing this while adding third host?