Bug 904153 - engine-setup hangs on Firewalld config
Summary: engine-setup hangs on Firewalld config
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: oVirt
Classification: Retired
Component: ovirt-engine-installer
Version: 3.2
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: ---
: ---
Assignee: Ofer Schreiber
QA Contact:
URL:
Whiteboard: integration
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-01-25 15:36 UTC by Netbulae
Modified: 2013-03-15 22:49 UTC (History)
12 users (show)

Fixed In Version: ovirt-engine-3.2.1
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-15 22:49:34 UTC
oVirt Team: ---


Attachments (Terms of Use)
engine setup log (97.30 KB, text/x-log)
2013-01-27 12:40 UTC, Netbulae
no flags Details
Comment (129.82 KB, text/plain)
2013-02-03 11:05 UTC, Netbulae
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 12587 0 None None None Never
oVirt gerrit 12590 0 None None None Never
oVirt gerrit 12762 0 None None None Never

Description Netbulae 2013-01-25 15:36:00 UTC
Description of problem:

When I run engine-setup and specify "override-firewall: Firewalld"

setup hangs on this message:
"Configuring Firewall..."

In the log I can see a NoReply:

2013-01-25 16:29:08::DEBUG::engine-setup::928::root:: Creating firewalld configuration
2013-01-25 16:29:08::DEBUG::engine-setup::956::root:: configuring firewalld
2013-01-25 16:29:08::DEBUG::common_utils::1228::root:: starting firewalld
2013-01-25 16:29:08::DEBUG::common_utils::1275::root:: executing action firewalld on service start
2013-01-25 16:29:08::DEBUG::common_utils::434::root:: Executing command --> '/sbin/service firewalld start'
2013-01-25 16:29:08::DEBUG::common_utils::472::root:: output = 
2013-01-25 16:29:08::DEBUG::common_utils::473::root:: stderr = Redirecting to /bin/systemctl start  firewalld.service

2013-01-25 16:29:08::DEBUG::common_utils::474::root:: retcode = 0
2013-01-25 16:29:33::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.12:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-01-25 16:29:33::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error

I disabled firewalld and installed with "override-firewall: None" and setup continues as normal.

Comment 1 Netbulae 2013-01-26 12:37:43 UTC
This was the patch to support firewalld:

http://gerrit.ovirt.org/#/c/10493/

packaging: engine-setup - add firewalld support

Add firewalld support for ovirt-engine-setup.
If the user will ask setup to handle firewalld, the setup will open
needed ports (JBoss ports + NFS) in all firewalld zones.

Comment 2 Alex Lourie 2013-01-27 10:31:51 UTC
Can you please attach the complete log of the setup? It would help us to handle the problem in the future.

Thank you.

Comment 3 Netbulae 2013-01-27 12:40:46 UTC
Created attachment 688418 [details]
engine setup log

No problem, I just ran the setup again for you. Here is the log file

Comment 4 Alex Lourie 2013-01-27 14:21:26 UTC
Thanks.

Could you also please run 

/bin/systemctl start  firewalld.service

and post the results?

Comment 5 Netbulae 2013-01-27 23:44:43 UTC
NP

"/bin/systemctl start  firewalld.service"

No response but in /var/log/messages the following info:

Jan 28 00:40:13 ovirt01 systemd[1]: Started firewalld - dynamic firewall daemon.

also did a "service firewalld status" and got the following output:

Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
	  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
	  Active: active (running) since Sun 2013-01-27 13:23:43 CET; 11h ago
	Main PID: 21146 (firewalld)
	  CGroup: name=systemd:/system/firewalld.service
		  └─21146 /usr/bin/python /usr/sbin/firewalld --nofork

Jan 27 13:23:43 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.
Jan 27 13:27:21 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.
Jan 28 00:40:13 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.

Comment 6 Alex Lourie 2013-01-28 11:44:35 UTC
OK, this seems as a strange state of the firewalld.

Could you please rerun the setup now?

Thanks.

Comment 7 Netbulae 2013-01-28 12:34:29 UTC
Same error. Also tried with firewalld stopped and still the same error

Comment 8 Netbulae 2013-01-28 12:44:47 UTC
With firewalld stopped it got started correctly at 13:32:42 by engine-setup. 

Still a timeout though:

2013-01-28 13:32:42::DEBUG::engine-setup::928::root:: Creating firewalld configuration
2013-01-28 13:32:42::DEBUG::engine-setup::956::root:: configuring firewalld
2013-01-28 13:32:42::DEBUG::common_utils::1228::root:: starting firewalld
2013-01-28 13:32:42::DEBUG::common_utils::1275::root:: executing action firewalld on service start
2013-01-28 13:32:42::DEBUG::common_utils::434::root:: Executing command --> '/sbin/service firewalld start'
2013-01-28 13:32:42::DEBUG::common_utils::472::root:: output = 
2013-01-28 13:32:42::DEBUG::common_utils::473::root:: stderr = Redirecting to /bin/systemctl start  firewalld.service
2013-01-28 13:32:42::DEBUG::common_utils::474::root:: retcode = 0
2013-01-28 13:33:07::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.184:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-01-28 13:33:07::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error


service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
	  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
	  Active: active (running) since Mon 2013-01-28 13:32:42 CET; 11min ago
	Main PID: 24699 (firewalld)
	  CGroup: name=systemd:/system/firewalld.service
		  └─24699 /usr/bin/python /usr/sbin/firewalld --nofork

Jan 28 13:32:42 ****.****.** systemd[1]: Started firewalld - dynamic firewall daemon.

Comment 9 Netbulae 2013-01-28 13:37:47 UTC
Did some more testing and I thought it could be a problem that I removed NetworkManager. 

I really hate NM but I reinstalled and configured it, but it still fails with the same error.

When I run "firewall-cmd --get-services", I can see ovirt listed

cluster-suite pop3s bacula-client smtp ipp radius bacula ovirt ftp mdns samba dhcpv6-client https openvpn imaps samba-client http dns ntp vnc-server telnet libvirt ssh ipsec ipp-client amanda-client tftp-client nfs tftp libvirt-tls

The file "/etc/firewalld/services/ovirt.xml" is there but the ports are not actually picked up. When I restart firewalld the ports remain unopened and it looks like ovirt doesn't get added to the zone:

"firewall-cmd --list-all-zones"
drop
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

work
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

internal
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh samba-client
  ports: 
  forward-ports: 
  icmp-blocks: 

external
  interfaces: 
  services: ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

trusted
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

home
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh samba-client
  ports: 
  forward-ports: 
  icmp-blocks: 

dmz
  interfaces: 
  services: ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

public
  interfaces: eth0
  services: mdns dhcpv6-client ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

block
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

"firewall-cmd --get-default-zone"
public

Comment 10 Pavel Zhukov 2013-02-02 00:18:51 UTC
Problem still persists. 
Patch http://gerrit.ovirt.org/#/c/10493/ has been applied

2013-02-02 01:17:44::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.2:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-02-02 01:17:44::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error

Comment 11 Alex Lourie 2013-02-03 09:45:07 UTC
This seems to have something to do with selinux-policies - like firewalld can't speak to the NM. Take a look at similar issue https://bugzilla.redhat.com/810508.

Also, please check the version of the selinux-policy package, and the mode of SELINUX on the system.

Comment 12 Netbulae 2013-02-03 11:05:00 UTC
Created attachment 915667 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).

Comment 13 Jason Brooks 2013-02-12 20:05:01 UTC
I was hitting this same issue today with oVirt 3.2 beta on F18. Putting selinux in permissive mode worked as a workaround.

Comment 14 Joop van de Wege 2013-02-17 12:16:15 UTC
kieppie on IRC had problems with 3.2 and I tried to help him and disabling firewalld and using None in engine-setup let the setup finish. After that I tried with firewalld on and ran into this same problem. Setting SELinux in permissive mode didn't help.
I'm using a minimal install F18 with the steps outlined on the Wiki.

[root@localhost ~]# rpm -aq | grep firewall
firewalld-0.2.12-2.fc18.noarch
[root@localhost ~]# rpm -aq | grep SE
[root@localhost ~]# rpm -aq | grep selin
selinux-policy-targeted-3.11.1-76.fc18.noarch
libselinux-2.1.12-7.fc18.x86_64
eclipselink-2.3.2-1.fc18.noarch
selinux-policy-3.11.1-76.fc18.noarch
libselinux-python-2.1.12-7.fc18.x86_64
libselinux-utils-2.1.12-7.fc18.x86_64
[root@localhost ~]# rpm -aq | grep ovirt
ovirt-release-fedora-5-2.noarch
ovirt-iso-uploader-3.2.0-1.fc18.noarch
ovirt-engine-userportal-3.2.0-4.fc18.noarch
ovirt-engine-backend-3.2.0-4.fc18.noarch
ovirt-engine-genericapi-3.2.0-4.fc18.noarch
ovirt-engine-cli-3.2.0.9-2.fc18.noarch
ovirt-engine-webadmin-portal-3.2.0-4.fc18.noarch
ovirt-engine-restapi-3.2.0-4.fc18.noarch
ovirt-engine-tools-common-3.2.0-4.fc18.noarch
ovirt-engine-dbscripts-3.2.0-4.fc18.noarch
ovirt-host-deploy-1.0.0-1.fc18.noarch
ovirt-log-collector-3.2.0-1.fc18.noarch
ovirt-engine-config-3.2.0-4.fc18.noarch
ovirt-image-uploader-3.2.0-1.fc18.noarch
ovirt-engine-notification-service-3.2.0-4.fc18.noarch
ovirt-host-deploy-java-1.0.0-1.fc18.noarch
ovirt-engine-sdk-3.2.0.8-1.fc18.noarch
ovirt-engine-setup-3.2.0-4.fc18.noarch
ovirt-engine-3.2.0-4.fc18.noarch

Comment 15 Ofer Schreiber 2013-02-27 13:45:36 UTC
OK, it seems like a selinux + firewalld integration issue.

Joop - Did you restart the machine after moving selinux to permissive? can you attach your logs?

Comment 16 Ofer Schreiber 2013-02-27 17:09:27 UTC
Did some more research:
it seems that the problematic line is:

zones = fw.getActiveZones()

in engine_firewalld.py, which is causing:
type=USER_AVC msg=audit(1361984756.212:1776): pid=461 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.24 spid=19402 tpid=19678 scontext=system_u:system_r:firewalld_t:s0 tcontext=unconfined_u:system_r:rpm_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

in audit.log

Comment 17 Ofer Schreiber 2013-02-27 17:37:50 UTC
It seems that miniyum.py is changing the proccess selinux context to rpm_t, which is causing the selinux permissions issue.
Alon - any reasonable solution to that?

Comment 18 Alon Bar-Lev 2013-02-27 19:08:17 UTC
(In reply to comment #17)
> It seems that miniyum.py is changing the proccess selinux context to rpm_t,
> which is causing the selinux permissions issue.
> Alon - any reasonable solution to that?

We need the rpm_t context... for rpm operations.
What do we need for firewalld? I am a bit afraid that we won't be able to have both rpm_t and another one... I thought that rpm_t should have all what rpm post scripts can do, including work with dbus...

Comment 19 Juan Hernández 2013-02-28 09:27:27 UTC
What specific rpm operations require the rpm_t context?

Comment 20 Alon Bar-Lev 2013-02-28 09:36:01 UTC
(In reply to comment #19)
> What specific rpm operations require the rpm_t context?

Installing rpms via the yum api, the %post action fails for various of issues, for example useradd won't work.

There should be no problem of setting a new selinux type before executing firewald <something>...

Comment 21 Juan Hernández 2013-02-28 10:16:39 UTC
Isn't it possible to fork a child for the RPM operations and change the context only in the child?

Comment 22 Alon Bar-Lev 2013-02-28 10:20:33 UTC
(In reply to comment #21)
> Isn't it possible to fork a child for the RPM operations and change the
> context only in the child?

We need some IPC operation between the two processes, some server to provide yum api and the installer that uses the server, serialization of requests and such.

Or we can write a simple command-line program like yum, but then we lost the transactional benefit of the API.

The yum API was meant to be used in-process... (if was meant to be used at all).

But I don't see the real issue, when executing commands at the execCmd we can set whatever selinux type we like, solving this issue for now.

Comment 23 Ofer Schreiber 2013-03-03 12:37:20 UTC
We're not executing anything related to firewalld via execCmd, we use the python api.

Also, the only usage to miniYum in the setup is to create the lock list, any real reason to have the context change in the simple usage of engien-setup?

Comment 24 Alon Bar-Lev 2013-03-03 12:47:39 UTC
(In reply to comment #23)
> We're not executing anything related to firewalld via execCmd, we use the
> python api.
> 
> Also, the only usage to miniYum in the setup is to create the lock list, any
> real reason to have the context change in the simple usage of engien-setup?

Are those in engine_firewalld.py should be removed?
---
from gi.repository import GObject
import sys
---

Well, we have a mess here... of course you can remove the miniyum.selinux_role() from engine-setup main prefix. But we may bump in this in future when trying to perform the same.

We should come up with some better solution.

Comment 25 Ofer Schreiber 2013-03-03 15:33:14 UTC
Patch posted:
http://gerrit.ovirt.org/#/c/12587/

Comment 26 Alon Bar-Lev 2013-03-03 19:59:52 UTC
commit 9df1d2d269940899fc1a3e1c0c02831135e0b455
Author: Alon Bar-Lev <alonbl>
Date:   Sun Mar 3 21:50:00 2013 +0200

    miniyum: we only need system_r
    
    rpm_t type is added automatically by the rpm_execcon(),
    question is why the system_r is not added...
    the rpm_t conflict with other tasks install should do
    such as interactive with dbus.
    
    [sync miniyum with otopi]
    
    Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=904153
    Change-Id: I2c9d6e3a9f1f65f403f309bf3b0e0ce2431c8b21
    Signed-off-by: Alon Bar-Lev <alonbl>

---

Untested as at least at my f18, the dbus python implementation goes crazy at random points. But a simple test program does work.

Comment 27 Juan Hernández 2013-03-04 09:48:00 UTC
The patch in comment #25 has been merged.

Comment 28 Alon Bar-Lev 2013-03-07 05:54:12 UTC
Waiting for merging the new patch.


Note You need to log in before you can comment on or make changes to this bug.