Bug 1459167

Summary: [HE] - failed to deploy hosted engine since the selinux changed.
Product: [oVirt] ovirt-appliance Reporter: Kobi Hakimi <khakimi>
Component: GeneralAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED CURRENTRELEASE QA Contact: Nikolai Sednev <nsednev>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1CC: bugs, stirabos
Target Milestone: ovirt-4.1.3Keywords: AutomationBlocker, Triaged
Target Release: 4.1Flags: rule-engine: ovirt-4.1+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-06 13:55:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Integration RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Kobi Hakimi 2017-06-06 12:53:40 UTC
Description of problem:
[HE] - failed to deploy hosted engine since the selinux changed.

Version-Release number of selected component (if applicable):
rhvm-appliance-4.1.20170605.0-1.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Run the command: hosted-engine --deploy
   and fill all the relevant data

Actual results:
The installation failed with error as you can see below in Additional info.

Expected results:
To deploy the hosted engine without any errors.

Additional info:

          |- [ INFO  ] Stage: Misc configuration
[ ERROR ] Engine setup got stuck on the appliance
[ ERROR ] Failed to execute stage 'Closing up': Engine setup is stalled on the appliance since 1800 seconds ago. Please check its log on the appliance. 
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20170606125355.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170606120927-9607yg.log



The error from /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20170606120927-9607yg.log:

2017-06-06 12:53:55 ERROR otopi.plugins.gr_he_common.engine.health health._closeup:139 Engine setup got stuck on the appliance
2017-06-06 12:53:55 DEBUG otopi.context context._executeMethod:142 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 132, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-common/engine/health.py", line 146, in _closeup
    ).format(since=TIMEOUT*nTimeout5)
RuntimeError: Engine setup is stalled on the appliance since 1800 seconds ago.
Please check its log on the appliance.

2017-06-06 12:53:55 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Closing up': Engine setup is stalled on the appliance since 1800 seconds ago.
Please check its log on the appliance.




in the ovirt-engine-setup log file I can see the following error:

2017-06-06 05:23:51 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1179 dbenv: {'OVESETUP_DB/database': 'engine', 'OVESETUP_DB/host': 'localhost', 'OVESETUP_DB/user': 'engine', 'OV
ESETUP_DB/secured': False, 'OVESETUP_DB/securedHostValidation': False, 'OVESETUP_DB/password': '**FILTERED**', 'OVESETUP_DB/port': 5432}
2017-06-06 05:23:51 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:212 Database: 'None', Statement: '
                select count(*) as count
                from pg_catalog.pg_tables
                where schemaname = 'public';
            ', args: {}
2017-06-06 05:23:51 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:217 Creating own connection
2017-06-06 05:23:51 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1211 database connection failed
Traceback (most recent call last):
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 1209, in getCredentials
    ] = self.isNewDatabase()
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 405, in isNewDatabase
    transaction=False,
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 226, in execute
    database=database,
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 160, in connect
    sslmode=sslmode,
  File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: Connection refused
        Is the server running on host "localhost" (::1) and accepting
        TCP/IP connections on port 5432?
could not connect to server: Connection refused
        Is the server running on host "localhost" (127.0.0.1) and accepting
        TCP/IP connections on port 5432?

Comment 1 Nikolai Sednev 2017-06-18 12:02:28 UTC
Successfully deployed HE over NFS on these components on host:
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch
ovirt-vmconsole-1.0.4-1.el7ev.noarch
vdsm-4.19.19-1.el7ev.x86_64
ovirt-hosted-engine-setup-2.1.3.1-1.el7ev.noarch
sanlock-3.5.0-1.el7.x86_64
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-vmconsole-host-1.0.4-1.el7ev.noarch
ovirt-host-deploy-1.6.6-1.el7ev.noarch
mom-0.5.9-1.el7ev.noarch
ovirt-hosted-engine-ha-2.1.3-1.el7ev.noarch
ovirt-setup-lib-1.1.3-1.el7ev.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7ev.noarch
qemu-kvm-rhev-2.9.0-10.el7.x86_64
libvirt-client-3.2.0-10.el7.x86_64
Linux version 3.10.0-514.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Wed Oct 19 11:24:13 EDT 2016
Linux 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)

Used rhvm-appliance-4.1.20170608.0-1.el7.noarch.