Bug 1291736
| Summary: | Failed to read metadata from storage after adding additional host | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-hosted-engine-setup | Reporter: | Nikolai Sednev <nsednev> | ||||
| Component: | Plugins.General | Assignee: | Simone Tiraboschi <stirabos> | ||||
| Status: | CLOSED WORKSFORME | QA Contact: | Nikolai Sednev <nsednev> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 1.3.1.3 | CC: | bugs, didi, fdeutsch, gklein, lveyde, mavital, rmartins, sbonazzo, stirabos | ||||
| Target Milestone: | ovirt-3.6.2 | Flags: | gklein:
ovirt-3.6.z?
gklein: blocker? rule-engine: planning_ack? sbonazzo: devel_ack+ rule-engine: testing_ack? |
||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | integration | ||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2015-12-17 14:22:34 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1291234 | ||||||
| Attachments: |
|
||||||
Created attachment 1106040 [details]
sosreport from host
The host passed the https://bugzilla.redhat.com/show_bug.cgi?id=1291234 and was added in to the HE-WEBUI, was clearly seen there as active, but not even single VM could be migrated to it. Dec 15 16:06:21 seal09.qa.lab.tlv.redhat.com ovirt-ha-agent[32609]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Request failed: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f'' - trying to restart agent Dec 15 16:06:38 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f' # hosted-engine --vm-status Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 117, in <module> if not status_checker.print_status(): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 60, in print_status all_host_stats = ha_cli.get_all_host_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats return self.get_all_stats(self.StatModes.HOST) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats stats = broker.get_stats_from_storage(service) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 232, in get_stats_from_storage result = self._checked_communicate(request) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 260, in _checked_communicate .format(message or response)) ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata' If both hosts rebooted, then third host being shown via "hosted-engine --vm-status". Closing this bug according to comment 3. it looks like a environment issue, please reopen if you see this again. (In reply to Fabian Deutsch from comment #4) > Closing this bug according to comment 3. > > it looks like a environment issue, please reopen if you see this again. It's not an environment issue, because the first time you're adding a new host, third in my case, it's not being shown if running "hosted-engine --vm-status", until previously existing hosts rebooted. Have you tried reproducing this while adding third host? |
Description of problem: Failed to read metadata from storage after adding additional host. Version-Release number of selected component (if applicable): Host: qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64 ovirt-vmconsole-host-1.0.0-1.el7ev.noarch ovirt-host-deploy-1.4.1-1.el7ev.noarch libvirt-client-1.2.17-13.el7_2.2.x86_64 vdsm-4.17.13-1.el7ev.noarch sanlock-3.2.4-2.el7_2.x86_64 ovirt-setup-lib-1.0.0-1.el7ev.noarch ovirt-vmconsole-1.0.0-1.el7ev.noarch ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch mom-0.5.1-1.el7ev.noarch ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch Linux version 3.10.0-327.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015 How reproducible: 100% Steps to Reproduce: 1.Try adding additional host to HE environment. 2. 3. Actual results: Third in my case host was not shown within the "hosted-engine --vm-status" and this error was received: - "Dec 15 16:06:48 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC) OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata' ". Expected results: Additional host should be seen via "hosted-engine --vm-status" and should see the storage metadata. Additional info: Sosreport from the host is attached.