Bug 1267337 - Hosted engine fails to complete cleanly, can't see hosted engine VM, and can't import HE storage domain.
Summary: Hosted engine fails to complete cleanly, can't see hosted engine VM, and can'...
Keywords:
Status: CLOSED DUPLICATE of bug 1269768
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.Block
Version: 1.3.0
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: ---
: ---
Assignee: Simone Tiraboschi
QA Contact: Ilanit Stein
URL:
Whiteboard: integration
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-09-29 17:10 UTC by Christopher Miersma
Modified: 2015-10-14 15:53 UTC (History)
8 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2015-10-14 15:53:24 UTC
oVirt Team: ---
Embargoed:
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
Output from hosted-engine setup (8.88 KB, text/plain)
2015-09-29 17:10 UTC, Christopher Miersma
no flags Details
Logs from the engine VM in this case (487.10 KB, application/x-gzip)
2015-09-29 17:10 UTC, Christopher Miersma
no flags Details
Logs from the host (14.66 MB, application/x-gzip)
2015-09-29 17:11 UTC, Christopher Miersma
no flags Details
Host logs (2.45 MB, application/x-gzip)
2015-10-09 16:42 UTC, Christopher Miersma
no flags Details
Engine logs with the latest patches (1.33 MB, application/x-gzip)
2015-10-09 16:44 UTC, Christopher Miersma
no flags Details

Description Christopher Miersma 2015-09-29 17:10:06 UTC
Created attachment 1078435 [details]
Output from hosted-engine setup

Description of problem:


Version-Release number of selected component (if applicable): 3.6 release candidate


How reproducible:
This has happened to me twice, once in the beta and again in 3.6rc1

Steps to Reproduce:
1. Install hosted-engine-setup on CentOS 7.
2. Run hosted-engine-setup with Fibre Channel storage
3. Build hosted-engine VM using an ISO image stored locally on the host.

Actual results:
The hosted-engine setup ends with error messages including:
"The VDSM host was found in a failed state. Please check engine and bootstrap installation log." I am able to complete the installation, by using the "reinstall" option on the host in the web interface. At that point, the interface gives the message: "The Hosted Engine Storage Domain doesn't no exist. It should be imported into the setup." When I go to import it, the interface cannot find the volume on which the storage domain resides. I am able to add additional LUNs as storage domains and create VMs, and the HE Storage Domain LUN appears in those interfaces, greyed out.

Expected results:
The hosted-engine should complete cleanly, and either automatically import the storage domain on which the HE VM resides or allow you to import it via the web interface. In addition, the engine VM should appear in the interface.

Additional info:
This could be related to these bugs, but I'm not sure: https://bugzilla.redhat.com/show_bug.cgi?id=1254748
https://bugzilla.redhat.com/show_bug.cgi?id=1221956
https://bugzilla.redhat.com/show_bug.cgi?id=1261996

Comment 1 Christopher Miersma 2015-09-29 17:10:43 UTC
Created attachment 1078436 [details]
Logs from the engine VM in this case

Comment 2 Christopher Miersma 2015-09-29 17:11:23 UTC
Created attachment 1078437 [details]
Logs from the host

Comment 3 Sandro Bonazzola 2015-10-07 15:07:29 UTC
2015-09-28 14:49:53 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:246 stderr: dd: failed to open ‘/rhev/data-center/mnt/blockSD/da105b33-9ef8-4cbd-bb51-916c223ed218/images/551a55b1-cae1-419c-b8bb-944fad65a594/ba9d6d41-407b-4014-ae4d-0fd407c0f451’: Permission denied

2015-09-28 14:49:53 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 149, in _closeup_create_tar
    dest
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py", line 249, in create_heconfimage
    raise RuntimeError('Unable to write HEConfImage')
RuntimeError: Unable to write HEConfImage
2015-09-28 14:49:53 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Closing up': Unable to write HEConfImage

Can you please provide the output of "ausearch -m avc -ts 28/09/2015" ?

Can you retry disabling selinux ?

Can you provide the output of: "rpm -qv selinux-policy-targeted" ?

Comment 4 Christopher Miersma 2015-10-08 13:45:10 UTC
(In reply to Sandro Bonazzola from comment #3)
> 2015-09-28 14:49:53 DEBUG
> otopi.plugins.ovirt_hosted_engine_setup.storage.heconf
> heconflib.create_heconfimage:246 stderr: dd: failed to open
> ‘/rhev/data-center/mnt/blockSD/da105b33-9ef8-4cbd-bb51-916c223ed218/images/
> 551a55b1-cae1-419c-b8bb-944fad65a594/ba9d6d41-407b-4014-ae4d-0fd407c0f451’:
> Permission denied
> 
> 2015-09-28 14:49:53 DEBUG otopi.context context._executeMethod:156 method
> exception
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in
> _executeMethod
>     method['method']()
>   File
> "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-
> setup/storage/heconf.py", line 149, in _closeup_create_tar
>     dest
>   File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py",
> line 249, in create_heconfimage
>     raise RuntimeError('Unable to write HEConfImage')
> RuntimeError: Unable to write HEConfImage
> 2015-09-28 14:49:53 ERROR otopi.context context._executeMethod:165 Failed to
> execute stage 'Closing up': Unable to write HEConfImage
> 
> Can you please provide the output of "ausearch -m avc -ts 28/09/2015" ?
> 
> Can you retry disabling selinux ?
> 
> Can you provide the output of: "rpm -qv selinux-policy-targeted" ?

Unfortunately, I no longer have that particular install up and running, as I have rebuilt on the same hardware. However, I can guarantee that I was running selinux in Permissive mode on both the host and engine VMs. In addition, the new setup has the same issue, but I can't find that error message above in the logs.

Comment 5 Christopher Miersma 2015-10-09 16:42:51 UTC
Created attachment 1081391 [details]
Host logs

I've retried a clean install with the latest build from the snapshots repo. Everything else is as clean as possible, but the hosted-engine domain still does not appear for import. SELinux is in Permissive mode throughout. If you're searching through these logs, the UUID for the hosted engine storage domain is 78501422-e863-47e2-986d-39e11577dc50.

Comment 6 Christopher Miersma 2015-10-09 16:44:36 UTC
Created attachment 1081392 [details]
Engine logs with the latest patches

This is from a new clean installation.

Comment 7 Simone Tiraboschi 2015-10-14 15:53:24 UTC

*** This bug has been marked as a duplicate of bug 1269768 ***


Note You need to log in before you can comment on or make changes to this bug.