Bug 1281417

Summary: vdsm host can not be added with firewalld enabled
Product: [oVirt] vdsm Reporter: Fabian Deutsch <fdeutsch>
Component: CoreAssignee: Dan Kenigsberg <danken>
Status: CLOSED NOTABUG QA Contact: Aharon Canan <acanan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.17.10CC: alonbl, bugs, fdeutsch
Target Milestone: ---Flags: rule-engine: planning_ack?
rule-engine: devel_ack?
rule-engine: testing_ack?
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-13 15:16:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Log from the failed attempt
none
firewalld and iptables tatus before/during host addition
none
Log from another failed attempt none

Description Fabian Deutsch 2015-11-12 13:42:00 UTC
Description of problem:
I can not add a freshly instaleld centos7 host with an (by default) enabelde firewalld.

Version-Release number of selected component (if applicable):
vdsm-4.17.10.1-0.el7.centos.noarch
centos-release-7-1.1503.el7.centos.2.8.x86_64

How reproducible:
Always 

Steps to Reproduce:
1. Install host with centos 7
2. Install vdsm on host and ovirt-host-deploy-offline
3. Add host to engine from Engine side

Actual results:
Host does not come up

Expected results:
Host comes up

Additional info:
vdsm is starting on the host, but the firewall prevents commnuication
Stopping firewalld solves the issue

Comment 1 Fabian Deutsch 2015-11-12 13:54:24 UTC
Alon, so should this bug rather be on host-deploy?

Comment 2 Alon Bar-Lev 2015-11-12 14:06:44 UTC
I do not understand, why you install anything on host whole host-deploy is doing this for you.

Comment 3 Fabian Deutsch 2015-11-12 14:16:19 UTC
I'm using host-deploy-offline, and in that case I need to have vdsm pre-installed - as you know.

Comment 4 Alon Bar-Lev 2015-11-12 14:44:24 UTC
(In reply to Fabian Deutsch from comment #3)
> I'm using host-deploy-offline, and in that case I need to have vdsm
> pre-installed - as you know.

if this is a node and you configure the host-deploy not to enforce firewall, you should take care of firewall your-self.

Comment 5 Fabian Deutsch 2015-11-12 14:50:17 UTC
This actually comes u pin next-ge Node work.

Because host-deploy is taking care of it, host-deploy-offline should take care of it in the offline case.

Comment 6 Fabian Deutsch 2015-11-12 14:51:10 UTC
Would it be possible to run all of host-deploy's plugins at install time in the offline case?

Comment 7 Alon Bar-Lev 2015-11-12 14:54:31 UTC
(In reply to Fabian Deutsch from comment #6)
> Would it be possible to run all of host-deploy's plugins at install time in
> the offline case?

so this is something new, stop open bugs and start open rfes, or perform proper design process.

legacy node had set its firewall settings correctly out of the box, engine hardcoded disabled firewall enforcement if target is node.

you can either keep this behaviour, identify as standard host or define a new type of host and modify engine and host-deploy behaviour to handle this new type of host.

Comment 8 Fabian Deutsch 2015-11-12 14:59:14 UTC
(In reply to Alon Bar-Lev from comment #7)
> (In reply to Fabian Deutsch from comment #6)
> > Would it be possible to run all of host-deploy's plugins at install time in
> > the offline case?
> 
> so this is something new, stop open bugs and start open rfes, or perform
> proper design process.
> 
> legacy node had set its firewall settings correctly out of the box,

Which has always been a problem.

> engine
> hardcoded disabled firewall enforcement if target is node.

Can you point me to that code? I can not find any host and firewalld specific code in host-deploy

> you can either keep this behaviour, identify as standard host or define a
> new type of host and modify engine and host-deploy behaviour to handle this
> new type of host.

That is what I am aiming at.

Comment 9 Alon Bar-Lev 2015-11-12 15:03:42 UTC
(In reply to Fabian Deutsch from comment #8)
> (In reply to Alon Bar-Lev from comment #7)
> > (In reply to Fabian Deutsch from comment #6)
> > > Would it be possible to run all of host-deploy's plugins at install time in
> > > the offline case?
> > 
> > so this is something new, stop open bugs and start open rfes, or perform
> > proper design process.
> > 
> > legacy node had set its firewall settings correctly out of the box,
> 
> Which has always been a problem.

I am unsure why, if you have a pre-defined environment there should be no change in firewall settings as well.

> 
> > engine
> > hardcoded disabled firewall enforcement if target is node.
> 
> Can you point me to that code? I can not find any host and firewalld
> specific code in host-deploy

InstallVdsInternalCommand

            if (parameters.getOverrideFirewall()) {
                switch (getVds().getVdsType()) {
                    case VDS:
                        deploy.addUnit(new VdsDeployIptablesUnit());
                    break;
                    case oVirtNode:
                        log.warn(
                            "Installation of Host {} will ignore Firewall Override option, since it is not supported for Host type {}",
                            getVds().getName(),
                            getVds().getVdsType().name()
                        );
                    break;

> > you can either keep this behaviour, identify as standard host or define a
> > new type of host and modify engine and host-deploy behaviour to handle this
> > new type of host.
> 
> That is what I am aiming at.

"That" has three options.

Comment 10 Fabian Deutsch 2015-11-13 14:44:17 UTC
Created attachment 1093690 [details]
Log from the failed attempt

Some summary: When I first add the host, then firewalld does not get disabled (see the lgos). On the second try however, firewalld is getting disabled and everything is well.

Comment 11 Fabian Deutsch 2015-11-13 14:44:54 UTC
Let me note: This is all happening on a plain and fresh centos7 host (I even dropped ovirt-host-deploy-offline)

Comment 12 Alon Bar-Lev 2015-11-13 14:47:03 UTC
(In reply to Fabian Deutsch from comment #10)
> Created attachment 1093690 [details]
> Log from the failed attempt
> 
> Some summary: When I first add the host, then firewalld does not get
> disabled (see the lgos). On the second try however, firewalld is getting
> disabled and everything is well.

2015-11-13 14:36:18 DEBUG otopi.context context.dumpEnvironment:510 ENV NETWORK/iptablesEnable=bool:'False'
2015-11-13 14:36:18 DEBUG otopi.context context.dumpEnvironment:510 ENV NETWORK/iptablesRules=NoneType:'None'

iptables/firewall was not enabled, ovirt-host-deploy did not touch any setting.

Comment 13 Fabian Deutsch 2015-11-13 14:51:20 UTC
Hm. I actually saw that firewalld.service was running on that host. Let me try to gather more logs.

Comment 14 Alon Bar-Lev 2015-11-13 14:58:49 UTC
(In reply to Fabian Deutsch from comment #13)
> Hm. I actually saw that firewalld.service was running on that host. Let me
> try to gather more logs.

enforcing firewall setting is done at engine side while adding host.

Comment 15 Fabian Deutsch 2015-11-13 15:00:21 UTC
Created attachment 1093692 [details]
firewalld and iptables tatus before/during host addition

This attachement shows that firewalld was running and an iptables service is not there.

Comment 16 Fabian Deutsch 2015-11-13 15:02:10 UTC
Created attachment 1093694 [details]
Log from another failed attempt

From this attachment:

2015-11-13 14:53:42 DEBUG otopi.context context.dumpEnvironment:500 ENVIRONMENT DUMP - BEGIN
2015-11-13 14:53:42 DEBUG otopi.context context.dumpEnvironment:510 ENV NETWORK/firewalldAvailable=bool:'False'
2015-11-13 14:53:42 DEBUG otopi.context context.dumpEnvironment:510 ENV NETWORK/firewalldDisableServices=list:'[]'
2015-11-13 14:53:42 DEBUG otopi.context context.dumpEnvironment:510 ENV NETWORK/firewalldEnable=bool:'False'
2015-11-13 14:53:42 DEBUG otopi.context context.dumpEnvironment:514 ENVIRONMENT DUMP - END

from the host side logs I see:

[root@test_tier_1_integrationsanity-node-ci ~]# service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Fr 2015-11-13 14:52:51 UTC; 2min 59s ago
 Main PID: 508 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─508 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Nov 13 14:52:51 test_tier_1_integrationsanity-node-ci.localdomain systemd[1]: Starting firewalld - dynamic firewall daemon...
Nov 13 14:52:51 test_tier_1_integrationsanity-node-ci.localdomain systemd[1]: Started firewalld - dynamic firewall daemon.


Which essentially shows that firewalld was started before the host was added but in the engine logs firewalld appears as not running IIUIC.

Comment 17 Fabian Deutsch 2015-11-13 15:16:48 UTC
After discussion on IRC: I am using "ovirt-shell add host" to add a host.
This command does _not_ include the firewall configuration - and that is why firewalld is not disabled in that flow.