Bug 1304514 - hosted-engine --deploy additional host installation fails when using iptables
hosted-engine --deploy additional host installation fails when using iptables
Status: CLOSED NOTABUG
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Network (Show other bugs)
1.3.2.3
x86_64 Linux
unspecified Severity urgent (vote)
: ---
: ---
Assigned To: Yedidyah Bar David
Ilanit Stein
:
Depends On:
Blocks: 1277010
  Show dependency treegraph
 
Reported: 2016-02-03 16:22 EST by Charlie Inglese
Modified: 2017-05-11 05:23 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1304445
Environment:
Last Closed: 2016-02-04 14:18:55 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?


Attachments (Terms of Use)
iptables -nvL (3.81 KB, text/plain)
2016-02-03 16:22 EST, Charlie Inglese
no flags Details
/var/log/vdsm/mom.log (3.09 KB, text/plain)
2016-02-03 16:24 EST, Charlie Inglese
no flags Details
/var/log/vdsm/vdsm.log (334.44 KB, text/plain)
2016-02-03 16:26 EST, Charlie Inglese
no flags Details
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup.log (479.37 KB, text/plain)
2016-02-03 16:27 EST, Charlie Inglese
no flags Details
ovirt-host-deploy log (624.24 KB, text/plain)
2016-02-04 14:20 EST, Charlie Inglese
no flags Details

  None (edit)
Description Charlie Inglese 2016-02-03 16:22:33 EST
Created attachment 1120905 [details]
iptables -nvL

Description of problem:
During initial installation of oVirt Hosted Engine using the appliance and answer file, if firewalld is selected as the OVEHOSTED_NETWORK/firewallManager (e.g. OVEHOSTED_NETWORK/firewallManager=str:firewalld), addition of the initial oVirt host fails.

Version-Release number of selected component (if applicable):
glusterfs-3.7.6-1.el7.x86_64
glusterfs-api-3.7.6-1.el7.x86_64
glusterfs-cli-3.7.6-1.el7.x86_64
glusterfs-client-xlators-3.7.6-1.el7.x86_64
glusterfs-fuse-3.7.6-1.el7.x86_64
glusterfs-geo-replication-3.7.6-1.el7.x86_64
glusterfs-libs-3.7.6-1.el7.x86_64
glusterfs-server-3.7.6-1.el7.x86_64
libgovirt-0.3.3-1.el7.x86_64
ovirt-engine-appliance-3.6-20160126.1.el7.centos.noarch
ovirt-engine-sdk-python-3.6.2.1-1.el7.centos.noarch
ovirt-host-deploy-1.4.1-1.el7.centos.noarch
ovirt-hosted-engine-ha-1.3.3.7-1.el7.centos.noarch
ovirt-hosted-engine-setup-1.3.2.3-1.el7.centos.noarch
ovirt-setup-lib-1.0.1-1.el7.centos.noarch
ovirt-vmconsole-1.0.0-1.el7.centos.noarch
ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
vdsm-4.17.18-0.el7.centos.noarch
vdsm-cli-4.17.18-0.el7.centos.noarch
vdsm-gluster-4.17.18-0.el7.centos.noarch
vdsm-hook-vmfex-dev-4.17.18-0.el7.centos.noarch
vdsm-infra-4.17.18-0.el7.centos.noarch
vdsm-jsonrpc-4.17.18-0.el7.centos.noarch
vdsm-python-4.17.18-0.el7.centos.noarch
vdsm-xmlrpc-4.17.18-0.el7.centos.noarch
vdsm-yajsonrpc-4.17.18-0.el7.centos.noarch


How reproducible:
Everytime

Steps to Reproduce:
1. Install oVirt appliance
2. Once oVirt HE successfully installed on initial node, begin install on freshly imaged (CentOS 7.2) additional (second) node
3. On second node: systemctl stop firewalld; setenforce 0; yum install -y ovirt-hosted-engine-setup; hosted-engine --deploy

Actual results:
1. [ERROR] Failed to execute stage 'Closing up': VDSM did not start within 120 seconds. 
2. [ERROR] Hosted Engine deployment failed: this system is not reliable, please check the issue, fix and redeploy.

Expected results:
1. Additional oVirt host is added to pool successfully.

Additional info:
SELinux permissive mode
Comment 1 Charlie Inglese 2016-02-03 16:24 EST
Created attachment 1120907 [details]
/var/log/vdsm/mom.log

hosted-engine --deploy additional host installation fails when using iptables
Comment 2 Charlie Inglese 2016-02-03 16:26 EST
Created attachment 1120908 [details]
/var/log/vdsm/vdsm.log

hosted-engine --deploy additional host installation fails when using iptables
Comment 3 Charlie Inglese 2016-02-03 16:27 EST
Created attachment 1120909 [details]
/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup.log

hosted-engine --deploy additional host installation fails when using iptables
Comment 4 Charlie Inglese 2016-02-03 16:32:49 EST
Comment on attachment 1120905 [details]
iptables -nvL

view of iptables rules after the setup fails/exits
Comment 5 Charlie Inglese 2016-02-03 16:41:08 EST
(In reply to Charlie Inglese from comment #0)
> Created attachment 1120905 [details]
> iptables -nvL
> 
> Description of problem:
> During initial installation of oVirt Hosted Engine using the appliance and
> answer file, if firewalld is selected as the
> OVEHOSTED_NETWORK/firewallManager (e.g.
> OVEHOSTED_NETWORK/firewallManager=str:firewalld), addition of the initial
> oVirt host fails.
> 
> Version-Release number of selected component (if applicable):
> glusterfs-3.7.6-1.el7.x86_64
> glusterfs-api-3.7.6-1.el7.x86_64
> glusterfs-cli-3.7.6-1.el7.x86_64
> glusterfs-client-xlators-3.7.6-1.el7.x86_64
> glusterfs-fuse-3.7.6-1.el7.x86_64
> glusterfs-geo-replication-3.7.6-1.el7.x86_64
> glusterfs-libs-3.7.6-1.el7.x86_64
> glusterfs-server-3.7.6-1.el7.x86_64
> libgovirt-0.3.3-1.el7.x86_64
> ovirt-engine-appliance-3.6-20160126.1.el7.centos.noarch
> ovirt-engine-sdk-python-3.6.2.1-1.el7.centos.noarch
> ovirt-host-deploy-1.4.1-1.el7.centos.noarch
> ovirt-hosted-engine-ha-1.3.3.7-1.el7.centos.noarch
> ovirt-hosted-engine-setup-1.3.2.3-1.el7.centos.noarch
> ovirt-setup-lib-1.0.1-1.el7.centos.noarch
> ovirt-vmconsole-1.0.0-1.el7.centos.noarch
> ovirt-vmconsole-host-1.0.0-1.el7.centos.noarch
> vdsm-4.17.18-0.el7.centos.noarch
> vdsm-cli-4.17.18-0.el7.centos.noarch
> vdsm-gluster-4.17.18-0.el7.centos.noarch
> vdsm-hook-vmfex-dev-4.17.18-0.el7.centos.noarch
> vdsm-infra-4.17.18-0.el7.centos.noarch
> vdsm-jsonrpc-4.17.18-0.el7.centos.noarch
> vdsm-python-4.17.18-0.el7.centos.noarch
> vdsm-xmlrpc-4.17.18-0.el7.centos.noarch
> vdsm-yajsonrpc-4.17.18-0.el7.centos.noarch
> 
> 
> How reproducible:
> Everytime
> 
> Steps to Reproduce:
> 1. Install oVirt appliance
> 2. Once oVirt HE successfully installed on initial node, begin install on
> freshly imaged (CentOS 7.2) additional (second) node
> 3. On second node: systemctl stop firewalld; setenforce 0; yum install -y
> ovirt-hosted-engine-setup; hosted-engine --deploy
> 
> Actual results:
> 1. [ERROR] Failed to execute stage 'Closing up': VDSM did not start within
> 120 seconds. 
> 2. [ERROR] Hosted Engine deployment failed: this system is not reliable,
> please check the issue, fix and redeploy.
> 
> Expected results:
> 1. Additional oVirt host is added to pool successfully.
> 
> Additional info:
> SELinux permissive mode

Description of problem:
Installation of additional oVirt HE node fails because vdsm connection refused due to iptables configuration.
Comment 6 Yedidyah Bar David 2016-02-04 03:14:57 EST
(In reply to Charlie Inglese from comment #0)
> Created attachment 1120905 [details]
> iptables -nvL
> 
> Description of problem:
> During initial installation of oVirt Hosted Engine using the appliance and
> answer file, if firewalld is selected as the
> OVEHOSTED_NETWORK/firewallManager (e.g.
> OVEHOSTED_NETWORK/firewallManager=str:firewalld), addition of the initial
> oVirt host fails.

Summary says iptables, here you mention firewalld. Which?

Isn't this a duplicate of bug 1304445?
Comment 7 Charlie Inglese 2016-02-04 14:18:55 EST
After more troubleshooting, it appears that this isn't iptables related. When using GlusterFS (OVESETUP_CONFIG/applicationMode=both), the additional nodes require gluster RPMs to be installed prior to oVirt setup being executed.

I'm attaching the ovirt-host-deploy log from ovirt-engine showing the error being encountered.

2016-02-04 08:30:44 DEBUG otopi.context context._executeMethod:142 Stage closeup METHOD otopi.plugins.ovirt_host_deploy.gluster.packages.Plugin._closeup
2016-02-04 08:30:44 INFO otopi.plugins.ovirt_host_deploy.gluster.packages packages._closeup:92 Starting gluster
2016-02-04 08:30:44 DEBUG otopi.plugins.otopi.services.systemd systemd.state:145 stopping service glusterd
2016-02-04 08:30:44 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:828 execute: ('/bin/systemctl', 'stop', 'glusterd.service'), executable='None', cwd='None', env=None
2016-02-04 08:30:44 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:878 execute-result: ('/bin/systemctl', 'stop', 'glusterd.service'), rc=5
2016-02-04 08:30:44 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:936 execute-output: ('/bin/systemctl', 'stop', 'glusterd.service') stdout:

2016-02-04 08:30:44 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:941 execute-output: ('/bin/systemctl', 'stop', 'glusterd.service') stderr:
Failed to stop glusterd.service: Unit glusterd.service not loaded.

2016-02-04 08:30:44 DEBUG otopi.context context._executeMethod:156 method exception
Traceback (most recent call last):
  File "/tmp/ovirt-F0vOHFRmv9/pythonlib/otopi/context.py", line 146, in _executeMethod
    method['method']()
  File "/tmp/ovirt-F0vOHFRmv9/otopi-plugins/ovirt-host-deploy/gluster/packages.py", line 94, in _closeup
    self.services.state('glusterd', state)
  File "/tmp/ovirt-F0vOHFRmv9/otopi-plugins/otopi/services/systemd.py", line 156, in state
    service=name,
RuntimeError: Failed to stop service 'glusterd'
2016-02-04 08:30:44 ERROR otopi.context context._executeMethod:165 Failed to execute stage 'Closing up': Failed to stop service 'glusterd'
Comment 8 Charlie Inglese 2016-02-04 14:20 EST
Created attachment 1121183 [details]
ovirt-host-deploy log

Note You need to log in before you can comment on or make changes to this bug.