Bug 1254671 - Fail HE install when closing the install
Summary: Fail HE install when closing the install
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ovirt-3.6.0-rc
: 3.6.0
Assignee: Simone Tiraboschi
QA Contact: Nikolai Sednev
Depends On:
TreeView+ depends on / blocked
Reported: 2015-08-18 15:47 UTC by Dudi Maroshi
Modified: 2016-03-09 19:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2016-03-09 19:14:32 UTC
oVirt Team: Integration
Target Upstream Version:

Attachments (Terms of Use)
log of install and failure (412.77 KB, text/plain)
2015-08-18 15:48 UTC, Dudi Maroshi
no flags Details
SOS report after hosted engine setup fails (8.50 MB, application/x-xz)
2015-08-26 07:04 UTC, Dudi Maroshi
no flags Details
Readable SELinux denial reports (7.09 KB, text/plain)
2015-08-26 07:45 UTC, Dudi Maroshi
no flags Details

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:0375 0 normal SHIPPED_LIVE ovirt-hosted-engine-setup bug fix and enhancement update 2016-03-09 23:48:34 UTC
oVirt gerrit 45812 0 master MERGED sharedconf: always read/write the shared configuration as VDSM user Never
oVirt gerrit 45815 0 master MERGED setup: relying on heconflib from ovirt-hosted-engine-ha Never
oVirt gerrit 45838 0 ovirt-hosted-engine-setup-1.3 MERGED setup: relying on heconflib from ovirt-hosted-engine-ha Never

Description Dudi Maroshi 2015-08-18 15:47:13 UTC
Description of problem:
When manually installing hosted-engine.
When completing and finalizing the configuration, installer fails with message:
Failed to execute stage 'Closing up': Unable to write HEConfImage

------------------log extract-----------------------
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:210 saving on: /rhev/data-center/mnt
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:219 executing: 'dd if=/tmp/tmpw4Ca9v
.tar of=/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0
/images/6e10fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5 bs=4k'
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:229 stdout: 
2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:230 stderr: dd: failed to open ‘/rhe
fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5’: Permission denied

2015-08-18 16:28:24 DEBUG otopi.context context._executeMethod:155 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 145, in _executeMethod
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 150, in _closeup_create_tar
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/heconflib.py", line 233, in create_heconfimage
    raise RuntimeError('Unable to write HEConfImage')
RuntimeError: Unable to write HEConfImage
2015-08-18 16:28:24 ERROR otopi.context context._executeMethod:164 Failed to execute stage 'Closing up': Unable to write HEConfImage
---------------end log extract----------------------

Version-Release number of selected component (if applicable):
Host RHEL 7.2 with ovirt repo 3.6
Engine RHEL 6.7 with ovirt repo 3.6

How reproducible:
Repeated twice, after increasing hard disk space.

Steps to Reproduce:
1.Install host on RHEL 7.6 with ovirt-hosted-engine 3.6
2.Start installing with hosted-engine --deploy
3.Follow the instruction to install and configure ovirt-engine VM with RHEL 6.7 and ovirt-engine 3.6
4.At last closing stage the configuration fails.

Actual results:

Expected results:

Additional info:
Attached log.

Comment 1 Dudi Maroshi 2015-08-18 15:48:42 UTC
Created attachment 1064367 [details]
log of install and failure

Comment 2 Yaniv Lavi 2015-08-20 11:14:47 UTC
Can you please attach full SOS report from the host?

Comment 3 Simone Tiraboschi 2015-08-20 17:45:35 UTC
Dudi, if the host is still available could you also please check SELinux denials?

Comment 4 Dudi Maroshi 2015-08-26 07:04:30 UTC
Created attachment 1067164 [details]
SOS report after hosted engine setup fails

Comment 5 Dudi Maroshi 2015-08-26 07:45:20 UTC
Created attachment 1067169 [details]
Readable SELinux denial reports

A human readable SELinux denial log
#sealert -a /var/log/audit/audit.log > /tmp/selinux.log
Contains a suspicious denial at the 3rd and last alert, a chrony lock alert.

Comment 9 Simone Tiraboschi 2015-09-07 14:25:37 UTC
Ok, found it.
NFS configuration is preventing root to write while hosted-engine-setup is executed as root.
It should be an issue with the recommended NFS configuration but let's fix to have it writing as vdsm user.

Comment 11 Ivan Bulatovic 2015-09-19 22:27:31 UTC
I still see this on iscsi storage domain (ovirt-hosted-engine-ha-20150909150556), node rhel7.1, engine appliance is used, 3.6-snapshot repos:

2015-09-20 00:04:18 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconf._closeup_create_tar:122 Saving hosted-engine configuration on the shared storage domain
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:188 temp tar file: /tmp/tmpyL69wk.tar
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:223 saving on: /rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:235 executing: 'sudo -u vdsm dd if=/tmp/tmpyL69wk.tar of=/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e bs=4k'
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:245 stdout: 
2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:246 stderr: dd: failed to open ‘/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e’: Permission denied

2015-09-20 00:04:18 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 149, in _closeup_create_tar
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py", line 249, in create_heconfimage
    raise RuntimeError('Unable to write HEConfImage')
RuntimeError: Unable to write HEConfImage
2015-09-20 00:04:18 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Closing up': Unable to write HEConfImage

selinux is in permissive mode.

Comment 12 Simone Tiraboschi 2015-09-20 15:00:59 UTC
(In reply to Ivan Bulatovic from comment #11)
> I still see this on iscsi storage domain

It's different but related issue, https://gerrit.ovirt.org/#/c/46344/ will fix also that will fix it

Comment 13 Nikolai Sednev 2015-11-25 08:14:08 UTC
Works for me on these components (performed deployment over NFS SD):
# rpm -qa vdsm libvirt-client sanlock qemu-kvm-rhev mom ovirt*

Comment 15 errata-xmlrpc 2016-03-09 19:14:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.