Bug 1029020 - [ovirt-setup] engine-setup does not configure firewall properly on upgrade
[ovirt-setup] engine-setup does not configure firewall properly on upgrade
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine-setup (Show other bugs)
3.3.0
Unspecified Unspecified
urgent Severity urgent
: ---
: 3.3.0
Assigned To: Yedidyah Bar David
Jiri Belka
integration
: Triaged
Depends On:
Blocks: 3.3rc1
  Show dependency treegraph
 
Reported: 2013-11-11 08:40 EST by Jiri Belka
Modified: 2014-01-21 17:27 EST (History)
11 users (show)

See Also:
Fixed In Version: is27
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs, configuration (241.10 KB, application/x-xz)
2013-11-12 07:11 EST, Jiri Belka
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 20737 None None None Never
oVirt gerrit 22181 None None None Never

  None (edit)
Description Jiri Belka 2013-11-11 08:40:57 EST
Description of problem:
ovirt-websocket-proxy is not configured during engine-setup (as the rpm is part of 'rhevm' "meta" package).

but port for accessing it is not opened (6100/tcp).

# lsof -Pnp `pgrep -f 'python.*websocket'` | grep LISTEN
ovirt-web 26428 ovirt    5u  IPv4            1385611      0t0     TCP *:6100 (LISTEN)

-%-
                                       option_value                                        
-------------------------------------------------------------------------------------------
 
 # libvirt tls
 -A INPUT -p tcp --dport 16514 -j ACCEPT
 
 # guest consoles
 -A INPUT -p tcp -m multiport --dports 5634:6166 -j ACCEPT
 
 # migration
 -A INPUT -p tcp -m multiport --dports 49152:49216 -j ACCEPT
 
 
 # oVirt default firewall configuration. Automatically generated by vdsm bootstrap script.
 *filter
 :INPUT ACCEPT [0:0]
 :FORWARD ACCEPT [0:0]
 :OUTPUT ACCEPT [0:0]
 -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 
 -A INPUT -i lo -j ACCEPT
 # vdsm
 -A INPUT -p tcp --dport 54321 -j ACCEPT
 # SSH
 -A INPUT -p tcp --dport @SSH_PORT@ -j ACCEPT
 # snmp
 -A INPUT -p udp --dport 161 -j ACCEPT
 
 @CUSTOM_RULES@
 
 # Reject any other input traffic
 -A INPUT -j REJECT --reject-with icmp-host-prohibited
 -A FORWARD -m physdev ! --physdev-is-bridged -j REJECT --reject-with icmp-host-prohibited
 COMMIT
 
 
 # glusterd
 -A INPUT -p tcp -m tcp --dport 24007 -j ACCEPT
 
 # portmapper
 -A INPUT -p udp -m udp --dport 111   -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 38465 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 38466 -j ACCEPT
 
 # nfs
 -A INPUT -p tcp -m tcp --dport 111   -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 38467 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 2049  -j ACCEPT
 
 # status
 -A INPUT -p tcp -m tcp --dport 39543 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 55863 -j ACCEPT
 
 # nlockmgr
 -A INPUT -p tcp -m tcp --dport 38468 -j ACCEPT
 -A INPUT -p udp -m udp --dport 963   -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 965   -j ACCEPT
 
 # ctdbd
 -A INPUT -p tcp -m tcp --dport 4379  -j ACCEPT
 
 # smbd
 -A INPUT -p tcp -m tcp --dport 139   -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 445   -j ACCEPT
 
 # Ports for gluster volume bricks (default 100 ports)
 -A INPUT -p tcp -m tcp --dport 24009:24108 -j ACCEPT
 -A INPUT -p tcp -m tcp --dport 49152:49251 -j ACCEPT
-%-


Version-Release number of selected component (if applicable):
is22

How reproducible:
100%

Steps to Reproduce:
1. yum install rhevm ; engine-setup
2. iptables -nL | grep 6100
3.

Actual results:
ovirt-websocket-proxy cannot be access even it is configured and running

Expected results:
should work out of the box

Additional info:
Comment 1 Jiri Belka 2013-11-11 08:43:15 EST
s/is not/is now/ :D in comment #0
Comment 2 Sandro Bonazzola 2013-11-12 03:15:17 EST
It seems that your iptables configuration has not been configured at all.
It's not just a websocket issue here.
Comment 3 Sandro Bonazzola 2013-11-12 03:29:39 EST
Can you reproduce this and attach setup logs?
Comment 4 Sandro Bonazzola 2013-11-12 03:41:11 EST
I'm not able to reproduce this with IS22:
# cat /etc/sysconfig/iptables
 # Generated by ovirt-engine installer
 #filtering rules
 *filter
 :INPUT ACCEPT [0:0]
 :FORWARD ACCEPT [0:0]
 :OUTPUT ACCEPT [0:0]
 -A INPUT -i lo -j ACCEPT
 -A INPUT -p icmp -m icmp --icmp-type any -j ACCEPT
 -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
 -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
 -A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
 -A INPUT -p tcp -m state --state NEW -m tcp --dport 6100 -j ACCEPT
 -A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT

 #drop all rule
 -A INPUT -j REJECT --reject-with icmp-host-prohibited
 COMMIT
Comment 5 Sandro Bonazzola 2013-11-12 03:43:34 EST
Jiri, are you sure that the iptables file you're using is not just a copy of the template edited by hand?
Comment 6 Jiri Belka 2013-11-12 03:56:29 EST
I did not touch any iptables rules. Although it is upgrade, I'll dig to find out what is the history of my RHEVM (then provide better reproducer).
Comment 7 Sandro Bonazzola 2013-11-12 04:20:43 EST
(In reply to Jiri Belka from comment #6)
> I did not touch any iptables rules. Although it is upgrade, I'll dig to find
> out what is the history of my RHEVM (then provide better reproducer).

So, it may be a vdsm or host-deploy bug since there is:

# oVirt default firewall configuration. Automatically generated by vdsm bootstrap script.


Alon, what do you think?
Comment 8 Alon Bar-Lev 2013-11-12 04:40:48 EST
(In reply to Sandro Bonazzola from comment #7)
> (In reply to Jiri Belka from comment #6)
> > I did not touch any iptables rules. Although it is upgrade, I'll dig to find
> > out what is the history of my RHEVM (then provide better reproducer).
> 
> So, it may be a vdsm or host-deploy bug since there is:
> 
> # oVirt default firewall configuration. Automatically generated by vdsm
> bootstrap script.
> 
> 
> Alon, what do you think?

jiri, are you trying to install websocket proxy on host or on engine?
Comment 9 Jiri Belka 2013-11-12 07:10:34 EST
My RHEVM has started its life as is19, then upgraded couple of times till is22.

The issue of missing 6100 port is related to upgrade.

I tried also following scenario:

## 3.2 to 3.3

* install latest_sf
* engine-setup
* upgrade rhevm-setup to latest_is
* engine-setup
* iptables -nL # port missing
* engine-setup # port is added

Thus is anybody would upgrade from 3.2 to 3.3 websocket proxy would not be reachable. He would need to run engine-setup again. This should be either documented (engine-setup probably does not know all its features as all its deps are not installed???) or corrected.

## old 3.3 to latest 3.3

* install is19
* engine-setup
* iptables -nL # port exists
* upgrade rehvm-setup to latest_is (is22)
* engine-setup
* iptables -nL # port missing
* grep 6100 /etc/sysconfig/iptables*
  /etc/sysconfig/iptables.20131112130418:-A INPUT -p tcp -m state --state NEW -m tcp --dport 6100 -j ACCEPT
* engine-setup
* iptables -nL # port missing
* grep 6100 /etc/sysconfig/iptables*
  /etc/sysconfig/iptables.20131112130418:-A INPUT -p tcp -m state --state NEW -m tcp --dport 6100 -j ACCEPT

See attachments for logs, configuration...
Comment 10 Jiri Belka 2013-11-12 07:11:22 EST
Created attachment 822888 [details]
logs, configuration
Comment 11 Sandro Bonazzola 2013-11-14 04:42:45 EST
(In reply to Jiri Belka from comment #9)
> My RHEVM has started its life as is19, then upgraded couple of times till
> is22.
> 
> The issue of missing 6100 port is related to upgrade.
> 
> I tried also following scenario:
> 
> ## 3.2 to 3.3
> 
> * install latest_sf
> * engine-setup
> * upgrade rhevm-setup to latest_is
> * engine-setup
> * iptables -nL # port missing

Upgrading from 3.2.z don't ask at all about websocket-proxy because its configuration plugin is disabled, since the check on service existence fails in late setup stage (the websocket-proxy rpm is not yet installed at this stage when upgrading from 3.2.z)

We can skip the check on the service existence and ask about it anyway since the user upgrading rhevm will have the package installed in any case (it's a dep for rhevm.) Alon, what do you think?

Firewall rule for opening 6100 is not added to iptables if websocket plugin is not enabled


> * engine-setup # port is added



> 
> Thus is anybody would upgrade from 3.2 to 3.3 websocket proxy would not be
> reachable. He would need to run engine-setup again. This should be either
> documented (engine-setup probably does not know all its features as all its
> deps are not installed???) or corrected.
> 
> ## old 3.3 to latest 3.3
> 
> * install is19
> * engine-setup
> * iptables -nL # port exists
> * upgrade rehvm-setup to latest_is (is22)
> * engine-setup
> * iptables -nL # port missing
> * grep 6100 /etc/sysconfig/iptables*
>   /etc/sysconfig/iptables.20131112130418:-A INPUT -p tcp -m state --state
> NEW -m tcp --dport 6100 -j ACCEPT
> * engine-setup
> * iptables -nL # port missing
> * grep 6100 /etc/sysconfig/iptables*
>   /etc/sysconfig/iptables.20131112130418:-A INPUT -p tcp -m state --state
> NEW -m tcp --dport 6100 -j ACCEPT
> 
> See attachments for logs, configuration...

Same here, since PKI exists the plugin is disabled so the 6100 port is not enabled.

Alon, after all I think that http://gerrit.ovirt.org/20737 should be merged.
Comment 12 Sandro Bonazzola 2013-11-14 04:43:46 EST
(In reply to Sandro Bonazzola from comment #11)

> Alon, after all I think that http://gerrit.ovirt.org/20737 should be merged.

with the above changes about enabling / disabling the plugin.
Comment 13 Alon Bar-Lev 2013-11-14 05:00:23 EST
(In reply to Sandro Bonazzola from comment #12)
> (In reply to Sandro Bonazzola from comment #11)
> 
> > Alon, after all I think that http://gerrit.ovirt.org/20737 should be merged.
> 
> with the above changes about enabling / disabling the plugin.

Auto enable plugin on upgrade from 3.2 is for your decision.

However, we need to support running engine-setup as many times as people wants, including to do nothing or upgrade from 3.3.z->3.3.z+n.

It is unacceptable that every run of engine-setup will reset firewall rules on server, and potentially damage other services running on the host.

For example, one solution I now see for some bugs of engine is to kill running async tasks... this was done manually so far, but in 3.3 user can just run setup even if it does not upgrade anything.

So we need a solution, either manual solution or very very sensitive automatic solution.
Comment 14 Jiri Belka 2013-11-14 05:04:55 EST
About firewall... Why don't you create separate chain for RHEVM services and ensure it is always in the beginning? Then you decrease potential issues of messing with non-RHEVM iptables rules...?
Comment 15 Sandro Bonazzola 2013-11-14 05:15:09 EST
(In reply to Jiri Belka from comment #14)
> About firewall... Why don't you create separate chain for RHEVM services and
> ensure it is always in the beginning? Then you decrease potential issues of
> messing with non-RHEVM iptables rules...?

What about using a dedicated chain only for the manager and use iptables -A to that chain for adding ports and iptables-save for store the changes instead of processing templates and overwrite existing rules?

On cleanup we can just delete our own chain leaving everything else in place.
Comment 16 Sandro Bonazzola 2013-11-14 05:17:25 EST
Jiri ok, that's what you said :-)
Comment 17 Alon Bar-Lev 2013-11-14 05:19:40 EST
(In reply to Sandro Bonazzola from comment #15)
> (In reply to Jiri Belka from comment #14)
> > About firewall... Why don't you create separate chain for RHEVM services and
> > ensure it is always in the beginning? Then you decrease potential issues of
> > messing with non-RHEVM iptables rules...?
> 
> What about using a dedicated chain only for the manager and use iptables -A
> to that chain for adding ports and iptables-save for store the changes
> instead of processing templates and overwrite existing rules?
> 
> On cleanup we can just delete our own chain leaving everything else in place.

You need to produce something deterministic, whatever you choose to run can be also established by looking into the configuration file.

Please remember that we also would like to use transactional model, and delay these changes of activation to a point we sure settings are OK.
Comment 18 Sandro Bonazzola 2013-11-14 05:50:20 EST
(In reply to Alon Bar-Lev from comment #13)

> It is unacceptable that every run of engine-setup will reset firewall rules
> on server, and potentially damage other services running on the host.

What about asking about firewall configuration at each run instead of keeping that answer in post-install config?
this will allow the user to avoid firewall rules overwriting (that we currently do)
Comment 19 Yedidyah Bar David 2013-11-14 05:53:38 EST
(In reply to Sandro Bonazzola from comment #18)
> (In reply to Alon Bar-Lev from comment #13)
> 
> > It is unacceptable that every run of engine-setup will reset firewall rules
> > on server, and potentially damage other services running on the host.
> 
> What about asking about firewall configuration at each run instead of
> keeping that answer in post-install config?
> this will allow the user to avoid firewall rules overwriting (that we
> currently do)

Fine with me.

I told Sandro in private that it's annoying, but annoyed users can simply add it to their answer file.
Comment 20 Alon Bar-Lev 2013-11-14 06:04:33 EST
(In reply to Yedidyah Bar David from comment #19)
> (In reply to Sandro Bonazzola from comment #18)
> > (In reply to Alon Bar-Lev from comment #13)
> > 
> > > It is unacceptable that every run of engine-setup will reset firewall rules
> > > on server, and potentially damage other services running on the host.
> > 
> > What about asking about firewall configuration at each run instead of
> > keeping that answer in post-install config?
> > this will allow the user to avoid firewall rules overwriting (that we
> > currently do)
> 
> Fine with me.
> 
> I told Sandro in private that it's annoying, but annoyed users can simply
> add it to their answer file.

I agree, I prefer to ask only if rules are actually going to be changed... which is somewhat difficult to know during customization...
Comment 21 Yedidyah Bar David 2013-11-20 05:44:30 EST
And what about legacy? Should we simply remove ovirt-engine-setup/legacy/firewall_manager.py ?
Comment 22 Alon Bar-Lev 2013-11-20 07:06:20 EST
(In reply to Yedidyah Bar David from comment #21)
> And what about legacy? Should we simply remove
> ovirt-engine-setup/legacy/firewall_manager.py ?

why?
Comment 23 Yedidyah Bar David 2013-11-20 07:10:30 EST
(In reply to Alon Bar-Lev from comment #22)
> (In reply to Yedidyah Bar David from comment #21)
> > And what about legacy? Should we simply remove
> > ovirt-engine-setup/legacy/firewall_manager.py ?
> 
> why?

Because you object to us overwriting the local changes by user...

Not keeping firewall-manager in postinstall will prevent that only for future upgrades. For legacy we should simply remove this plugin and let the common one do its stuff.
Comment 24 Alon Bar-Lev 2013-11-20 07:14:59 EST
(In reply to Yedidyah Bar David from comment #23)
> (In reply to Alon Bar-Lev from comment #22)
> > (In reply to Yedidyah Bar David from comment #21)
> > > And what about legacy? Should we simply remove
> > > ovirt-engine-setup/legacy/firewall_manager.py ?
> > 
> > why?
> 
> Because you object to us overwriting the local changes by user...
> 
> Not keeping firewall-manager in postinstall will prevent that only for
> future upgrades. For legacy we should simply remove this plugin and let the
> common one do its stuff.

I thought that the idea of the suggestion in comment#18 is to ask user more questions.
Comment 25 Yedidyah Bar David 2013-11-20 07:23:53 EST
(In reply to Alon Bar-Lev from comment #24)
> (In reply to Yedidyah Bar David from comment #23)
> > (In reply to Alon Bar-Lev from comment #22)
> > > (In reply to Yedidyah Bar David from comment #21)
> > > > And what about legacy? Should we simply remove
> > > > ovirt-engine-setup/legacy/firewall_manager.py ?
> > > 
> > > why?
> > 
> > Because you object to us overwriting the local changes by user...
> > 
> > Not keeping firewall-manager in postinstall will prevent that only for
> > future upgrades. For legacy we should simply remove this plugin and let the
> > common one do its stuff.
> 
> I thought that the idea of the suggestion in comment#18 is to ask user more
> questions.

It is. This will not happen when the legacy plugin is active, because it will set firewall manager and the common file be skipped.

As far as I understand, the only reason we actually have ovirt-engine-setup/legacy/firewall_manager.py is because 3.2 did not keep the answer to the question "which firewall manager do you want to use", and we wanted upgrades to be as question-free as possible. I might be wrong, of course. If I am right, but we now decide the opposite - that we always want to ask the user about the firewall manager - we do not need this plugin anymore. Please read its source and tell me what we need it for.
Comment 26 Alon Bar-Lev 2013-11-20 07:27:41 EST
(In reply to Yedidyah Bar David from comment #25)
> As far as I understand, the only reason we actually have
> ovirt-engine-setup/legacy/firewall_manager.py is because 3.2 did not keep
> the answer to the question "which firewall manager do you want to use", and
> we wanted upgrades to be as question-free as possible.

If this is the case then OK.
Just keep in mind that in rhel/centos there is no firewalld and we would probably would like to avoid the entire question...

Not sure why this module disable the common...
Comment 27 Yedidyah Bar David 2013-12-09 03:20:49 EST
See https://bugzilla.redhat.com/show_bug.cgi?id=1024707#c9 for details.
Comment 28 Jiri Belka 2013-12-13 08:48:11 EST
ok, tested with updates between latest 3.2 -> 3.3, and older 3.3 to latest 3.3.
Comment 29 Itamar Heim 2014-01-21 17:22:06 EST
Closing - RHEV 3.3 Released
Comment 30 Itamar Heim 2014-01-21 17:27:15 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.