Bug 1254671 - Fail HE install when closing the install
Summary: Fail HE install when closing the install
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-08-18 15:47 UTC by Dudi Maroshi
Modified: 2016-03-09 19:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-03-09 19:14:32 UTC
oVirt Team: Integration
Target Upstream Version:

Attachments (Terms of Use)
log of install and failure (412.77 KB, text/plain)
2015-08-18 15:48 UTC, Dudi Maroshi 		no flags Details
SOS report after hosted engine setup fails (8.50 MB, application/x-xz)
2015-08-26 07:04 UTC, Dudi Maroshi 		no flags Details
Readable SELinux denial reports (7.09 KB, text/plain)
2015-08-26 07:45 UTC, Dudi Maroshi 		no flags Details
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0375 normal SHIPPED_LIVE ovirt-hosted-engine-setup bug fix and enhancement update 2016-03-09 23:48:34 UTC
oVirt gerrit 45812 master MERGED sharedconf: always read/write the shared configuration as VDSM user Never
oVirt gerrit 45815 master MERGED setup: relying on heconflib from ovirt-hosted-engine-ha Never
oVirt gerrit 45838 ovirt-hosted-engine-setup-1.3 MERGED setup: relying on heconflib from ovirt-hosted-engine-ha Never

Description Dudi Maroshi 2015-08-18 15:47:13 UTC 
Description of problem:
When manually installing hosted-engine.
When completing and finalizing the configuration, installer fails with message:
Failed to execute stage 'Closing up': Unable to write HEConfImage

------------------log extract-----------------------
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:210 saving on: /rhev/data-center/mnt
/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0/images/6e10fc9f-e4b0-4773-9*
*FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:219 executing: 'dd if=/tmp/tmpw4Ca9v
.tar of=/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0
/images/6e10fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5 bs=4k'
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:229 stdout: 
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:230 stderr: dd: failed to open ‘/rhe
v/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0/images/6e10
fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5’: Permission denied

2015-08-18 16:28:24 DEBUG otopi.context context._executeMethod:155 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 145, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 150, in _closeup_create_tar
    dest
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/heconflib.py", line 233, in create_heconfimage
    raise RuntimeError('Unable to write HEConfImage')
RuntimeError: Unable to write HEConfImage
2015-08-18 16:28:24 ERROR otopi.context context._executeMethod:164 Failed to execute stage 'Closing up': Unable to write HEConfImage
2
---------------end log extract----------------------


Version-Release number of selected component (if applicable):
Host RHEL 7.2 with ovirt repo 3.6
Engine RHEL 6.7 with ovirt repo 3.6



How reproducible:
Repeated twice, after increasing hard disk space.

Steps to Reproduce:
1.Install host on RHEL 7.6 with ovirt-hosted-engine 3.6
2.Start installing with hosted-engine --deploy
3.Follow the instruction to install and configure ovirt-engine VM with RHEL 6.7 and ovirt-engine 3.6
4.At last closing stage the configuration fails.

Actual results:
Fails

Expected results:
Success.

Additional info:
Attached log.
Comment 1 Dudi Maroshi 2015-08-18 15:48:42 UTC 
Created attachment 1064367 [details]
log of install and failure
Comment 2 Yaniv Lavi 2015-08-20 11:14:47 UTC 
Can you please attach full SOS report from the host?
Comment 3 Simone Tiraboschi 2015-08-20 17:45:35 UTC 
Dudi, if the host is still available could you also please check SELinux denials?
Comment 4 Dudi Maroshi 2015-08-26 07:04:30 UTC 
Created attachment 1067164 [details]
SOS report after hosted engine setup fails
Comment 5 Dudi Maroshi 2015-08-26 07:45:20 UTC 
Created attachment 1067169 [details]
Readable SELinux denial reports

A human readable SELinux denial log
#sealert -a /var/log/audit/audit.log > /tmp/selinux.log
Contains a suspicious denial at the 3rd and last alert, a chrony lock alert.
Comment 9 Simone Tiraboschi 2015-09-07 14:25:37 UTC 
Ok, found it.
NFS configuration is preventing root to write while hosted-engine-setup is executed as root.
It should be an issue with the recommended NFS configuration but let's fix to have it writing as vdsm user.
Comment 11 Ivan Bulatovic 2015-09-19 22:27:31 UTC 
I still see this on iscsi storage domain (ovirt-hosted-engine-ha-20150909150556), node rhel7.1, engine appliance is used, 3.6-snapshot repos:

2015-09-20 00:04:18 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconf._closeup_create_tar:122 Saving hosted-engine configuration on the shared storage domain
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:188 temp tar file: /tmp/tmpyL69wk.tar
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:223 saving on: /rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:235 executing: 'sudo -u vdsm dd if=/tmp/tmpyL69wk.tar of=/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e bs=4k'
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:245 stdout: 
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:246 stderr: dd: failed to open ‘/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e’: Permission denied

2015-09-20 00:04:18 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 149, in _closeup_create_tar
    dest
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py", line 249, in create_heconfimage
    raise RuntimeError('Unable to write HEConfImage')
RuntimeError: Unable to write HEConfImage
2015-09-20 00:04:18 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Closing up': Unable to write HEConfImage

selinux is in permissive mode.
Comment 12 Simone Tiraboschi 2015-09-20 15:00:59 UTC 
(In reply to Ivan Bulatovic from comment #11)
> I still see this on iscsi storage domain

It's different but related issue, https://gerrit.ovirt.org/#/c/46344/ will fix also that will fix it
Comment 13 Nikolai Sednev 2015-11-25 08:14:08 UTC 
Works for me on these components (performed deployment over NFS SD):
# rpm -qa vdsm libvirt-client sanlock qemu-kvm-rhev mom ovirt*
ovirt-vmconsole-host-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch
ovirt-release36-002-2.noarch
ovirt-engine-sdk-python-3.6.0.4-0.2.20151123.gita2f81ed.el7.centos.noarch
sanlock-3.2.4-1.el7.x86_64
ovirt-setup-lib-1.0.1-0.0.master.20151119123055.gitfa908be.el7.centos.noarch
qemu-kvm-rhev-2.3.0-31.el7_2.3.x86_64
ovirt-hosted-engine-ha-1.3.3-0.0.master.20151118145556.20151118145552.git71b535e.el7.noarch
ovirt-vmconsole-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch
ovirt-release36-snapshot-002-2.noarch
libvirt-client-1.2.17-13.el7.x86_64
ovirt-hosted-engine-setup-1.3.1-0.0.master.20151118143825.gitc013638.el7.centos.noarch
ovirt-host-deploy-1.4.2-0.0.master.20151122153544.gitfc808fc.el7.noarch
vdsm-4.17.10.1-0.el7ev.noarch
mom-0.5.1-2.el7.noarch
Comment 15 errata-xmlrpc 2016-03-09 19:14:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0375.html
Note You need to log in before you can comment on or make changes to this bug.