Bug 1291736 - Failed to read metadata from storage after adding additional host
Failed to read metadata from storage after adding additional host
Status: CLOSED WORKSFORME
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.General (Show other bugs)
1.3.1.3
x86_64 Linux
unspecified Severity urgent (vote)
: ovirt-3.6.2
: ---
Assigned To: Simone Tiraboschi
Nikolai Sednev
integration
:
Depends On:
Blocks: 1291234
  Show dependency treegraph
 
Reported: 2015-12-15 09:12 EST by Nikolai Sednev
Modified: 2016-02-01 07:57 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-17 09:22:34 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
gklein: ovirt‑3.6.z?
gklein: blocker?
rule-engine: planning_ack?
sbonazzo: devel_ack+
rule-engine: testing_ack?


Attachments (Terms of Use)
sosreport from host (7.17 MB, application/x-xz)
2015-12-15 09:15 EST, Nikolai Sednev
no flags Details

  None (edit)
Description Nikolai Sednev 2015-12-15 09:12:55 EST
Description of problem:
Failed to read metadata from storage after adding additional host.


Version-Release number of selected component (if applicable):
Host:
qemu-kvm-rhev-2.3.0-31.el7_2.5.x86_64
ovirt-vmconsole-host-1.0.0-1.el7ev.noarch
ovirt-host-deploy-1.4.1-1.el7ev.noarch
libvirt-client-1.2.17-13.el7_2.2.x86_64
vdsm-4.17.13-1.el7ev.noarch
sanlock-3.2.4-2.el7_2.x86_64
ovirt-setup-lib-1.0.0-1.el7ev.noarch
ovirt-vmconsole-1.0.0-1.el7ev.noarch
ovirt-hosted-engine-ha-1.3.3.3-1.el7ev.noarch
mom-0.5.1-1.el7ev.noarch
ovirt-hosted-engine-setup-1.3.1.3-1.el7ev.noarch
Linux version 3.10.0-327.el7.x86_64 (mockbuild@x86-034.build.eng.bos.redhat.com) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Oct 29 17:29:29 EDT 2015


How reproducible:
100%

Steps to Reproduce:
1.Try adding additional host to HE environment.
2.
3.

Actual results:
Third in my case host was not shown within the "hosted-engine --vm-status" and this error was received: - "Dec 15 16:06:48 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata
                                                                     Traceback (most recent call last):
                                                                       File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type
                                                                         f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)
                                                                     OSError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata' ".

Expected results:
Additional host should be seen via "hosted-engine --vm-status" and should see the storage metadata.

Additional info:
Sosreport from the host is attached.
Comment 1 Nikolai Sednev 2015-12-15 09:15 EST
Created attachment 1106040 [details]
sosreport from host
Comment 2 Nikolai Sednev 2015-12-15 09:18:58 EST
The host passed the https://bugzilla.redhat.com/show_bug.cgi?id=1291234 and was added in to the HE-WEBUI, was clearly seen there as active, but not even single VM could be migrated to it.


Dec 15 16:06:21 seal09.qa.lab.tlv.redhat.com ovirt-ha-agent[32609]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Request failed: failed to read metadata: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f'' - trying to restart agent

Dec 15 16:06:38 seal09.qa.lab.tlv.redhat.com ovirt-ha-broker[32618]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to read metadata from /var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f
                                                                     Traceback (most recent call last):
                                                                       File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 129, in get_raw_stats_for_service_type
                                                                         f = os.open(path, direct_flag | os.O_RDONLY | os.O_SYNC)
                                                                     OSError: [Errno 2] No such file or directory: '/var/run/vdsm/storage/8ac16336-7e92-48ae-838e-6bae32d28a5a/d092da14-ac6d-41a6-aa0a-8285d852ad76/3de4c99d-5690-4531-a96d-88a1266e942f'

# hosted-engine --vm-status
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 117, in <module>
    if not status_checker.print_status():
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 60, in print_status
    all_host_stats = ha_cli.get_all_host_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats
    return self.get_all_stats(self.StatModes.HOST)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats
    stats = broker.get_stats_from_storage(service)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 232, in get_stats_from_storage
    result = self._checked_communicate(request)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 260, in _checked_communicate
    .format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/10.35.64.11:_vol_RHEV_Virt_nsednev__he__bugs__1__3__6_/8ac16336-7e92-48ae-838e-6bae32d28a5a/ha_agent/hosted-engine.metadata'
Comment 3 Nikolai Sednev 2015-12-15 11:40:00 EST
If both hosts rebooted, then third host being shown via "hosted-engine --vm-status".
Comment 4 Fabian Deutsch 2015-12-17 09:22:34 EST
Closing this bug according to comment 3.

it looks like a environment issue, please reopen if you see this again.
Comment 5 Nikolai Sednev 2015-12-20 01:17:46 EST
(In reply to Fabian Deutsch from comment #4)
> Closing this bug according to comment 3.
> 
> it looks like a environment issue, please reopen if you see this again.

It's not an environment issue, because the first time you're adding a new host, third in my case, it's not being shown if running "hosted-engine --vm-status", until previously existing hosts rebooted. Have you tried reproducing this while adding third host?

Note You need to log in before you can comment on or make changes to this bug.