Description of problem: When manually installing hosted-engine. When completing and finalizing the configuration, installer fails with message: Failed to execute stage 'Closing up': Unable to write HEConfImage ------------------log extract----------------------- 2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:210 saving on: /rhev/data-center/mnt /multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0/images/6e10fc9f-e4b0-4773-9* *FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5 2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:219 executing: 'dd if=/tmp/tmpw4Ca9v .tar of=/rhev/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0 /images/6e10fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5 bs=4k' 2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:229 stdout: 2015-08-18 16:28:24 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:230 stderr: dd: failed to open ‘/rhe v/data-center/mnt/multipass.eng.lab.tlv.redhat.com:_export_images_rnd_dmaroshi_sd-hosted-engine-1/908a3d41-5f28-405b-bbb5-c9ff3e8faec0/images/6e10 fc9f-e4b0-4773-9**FILTERED**7-5a0dae**FILTERED**bf0c/3d54fd38-47a6-4e2c-a9f2-e0b49ff8cbb5’: Permission denied 2015-08-18 16:28:24 DEBUG otopi.context context._executeMethod:155 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 145, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 150, in _closeup_create_tar dest File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/heconflib.py", line 233, in create_heconfimage raise RuntimeError('Unable to write HEConfImage') RuntimeError: Unable to write HEConfImage 2015-08-18 16:28:24 ERROR otopi.context context._executeMethod:164 Failed to execute stage 'Closing up': Unable to write HEConfImage 2 ---------------end log extract---------------------- Version-Release number of selected component (if applicable): Host RHEL 7.2 with ovirt repo 3.6 Engine RHEL 6.7 with ovirt repo 3.6 How reproducible: Repeated twice, after increasing hard disk space. Steps to Reproduce: 1.Install host on RHEL 7.6 with ovirt-hosted-engine 3.6 2.Start installing with hosted-engine --deploy 3.Follow the instruction to install and configure ovirt-engine VM with RHEL 6.7 and ovirt-engine 3.6 4.At last closing stage the configuration fails. Actual results: Fails Expected results: Success. Additional info: Attached log.
Created attachment 1064367 [details] log of install and failure
Can you please attach full SOS report from the host?
Dudi, if the host is still available could you also please check SELinux denials?
Created attachment 1067164 [details] SOS report after hosted engine setup fails
Created attachment 1067169 [details] Readable SELinux denial reports A human readable SELinux denial log #sealert -a /var/log/audit/audit.log > /tmp/selinux.log Contains a suspicious denial at the 3rd and last alert, a chrony lock alert.
Ok, found it. NFS configuration is preventing root to write while hosted-engine-setup is executed as root. It should be an issue with the recommended NFS configuration but let's fix to have it writing as vdsm user.
I still see this on iscsi storage domain (ovirt-hosted-engine-ha-20150909150556), node rhel7.1, engine appliance is used, 3.6-snapshot repos: 2015-09-20 00:04:18 INFO otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconf._closeup_create_tar:122 Saving hosted-engine configuration on the shared storage domain 2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:188 temp tar file: /tmp/tmpyL69wk.tar 2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:223 saving on: /rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e 2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:235 executing: 'sudo -u vdsm dd if=/tmp/tmpyL69wk.tar of=/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e bs=4k' 2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:245 stdout: 2015-09-20 00:04:18 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.heconf heconflib.create_heconfimage:246 stderr: dd: failed to open ‘/rhev/data-center/mnt/blockSD/6176900b-7fa1-4cac-8986-7a3d8e567468/images/1bb973fe-cd4c-4c14-ace0-ef37b26cf5a9/39749656-5b91-451f-853d-386d0d27005e’: Permission denied 2015-09-20 00:04:18 DEBUG otopi.context context._executeMethod:156 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 146, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/heconf.py", line 149, in _closeup_create_tar dest File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/heconflib.py", line 249, in create_heconfimage raise RuntimeError('Unable to write HEConfImage') RuntimeError: Unable to write HEConfImage 2015-09-20 00:04:18 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Closing up': Unable to write HEConfImage selinux is in permissive mode.
(In reply to Ivan Bulatovic from comment #11) > I still see this on iscsi storage domain It's different but related issue, https://gerrit.ovirt.org/#/c/46344/ will fix also that will fix it
Works for me on these components (performed deployment over NFS SD): # rpm -qa vdsm libvirt-client sanlock qemu-kvm-rhev mom ovirt* ovirt-vmconsole-host-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch ovirt-release36-002-2.noarch ovirt-engine-sdk-python-3.6.0.4-0.2.20151123.gita2f81ed.el7.centos.noarch sanlock-3.2.4-1.el7.x86_64 ovirt-setup-lib-1.0.1-0.0.master.20151119123055.gitfa908be.el7.centos.noarch qemu-kvm-rhev-2.3.0-31.el7_2.3.x86_64 ovirt-hosted-engine-ha-1.3.3-0.0.master.20151118145556.20151118145552.git71b535e.el7.noarch ovirt-vmconsole-1.0.1-0.0.master.20151105234454.git3e5d52e.el7.noarch ovirt-release36-snapshot-002-2.noarch libvirt-client-1.2.17-13.el7.x86_64 ovirt-hosted-engine-setup-1.3.1-0.0.master.20151118143825.gitc013638.el7.centos.noarch ovirt-host-deploy-1.4.2-0.0.master.20151122153544.gitfc808fc.el7.noarch vdsm-4.17.10.1-0.el7ev.noarch mom-0.5.1-2.el7.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0375.html