Bug 904153 - engine-setup hangs on Firewalld config
engine-setup hangs on Firewalld config
Status: CLOSED NEXTRELEASE
Product: oVirt
Classification: Community
Component: ovirt-engine-installer (Show other bugs)
3.2
x86_64 Linux
urgent Severity high
: ---
: ---
Assigned To: Ofer Schreiber
integration
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-01-25 10:36 EST by Netbulae
Modified: 2013-03-15 18:49 EDT (History)
12 users (show)

See Also:
Fixed In Version: ovirt-engine-3.2.1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-15 18:49:34 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
engine setup log (97.30 KB, text/x-log)
2013-01-27 07:40 EST, Netbulae
no flags Details
Comment (129.82 KB, text/plain)
2013-02-03 06:05 EST, Netbulae
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 12587 None None None Never
oVirt gerrit 12590 None None None Never
oVirt gerrit 12762 None None None Never

  None (edit)
Description Netbulae 2013-01-25 10:36:00 EST
Description of problem:

When I run engine-setup and specify "override-firewall: Firewalld"

setup hangs on this message:
"Configuring Firewall..."

In the log I can see a NoReply:

2013-01-25 16:29:08::DEBUG::engine-setup::928::root:: Creating firewalld configuration
2013-01-25 16:29:08::DEBUG::engine-setup::956::root:: configuring firewalld
2013-01-25 16:29:08::DEBUG::common_utils::1228::root:: starting firewalld
2013-01-25 16:29:08::DEBUG::common_utils::1275::root:: executing action firewalld on service start
2013-01-25 16:29:08::DEBUG::common_utils::434::root:: Executing command --> '/sbin/service firewalld start'
2013-01-25 16:29:08::DEBUG::common_utils::472::root:: output = 
2013-01-25 16:29:08::DEBUG::common_utils::473::root:: stderr = Redirecting to /bin/systemctl start  firewalld.service

2013-01-25 16:29:08::DEBUG::common_utils::474::root:: retcode = 0
2013-01-25 16:29:33::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.12:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-01-25 16:29:33::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error

I disabled firewalld and installed with "override-firewall: None" and setup continues as normal.
Comment 1 Netbulae 2013-01-26 07:37:43 EST
This was the patch to support firewalld:

http://gerrit.ovirt.org/#/c/10493/

packaging: engine-setup - add firewalld support

Add firewalld support for ovirt-engine-setup.
If the user will ask setup to handle firewalld, the setup will open
needed ports (JBoss ports + NFS) in all firewalld zones.
Comment 2 Alex Lourie 2013-01-27 05:31:51 EST
Can you please attach the complete log of the setup? It would help us to handle the problem in the future.

Thank you.
Comment 3 Netbulae 2013-01-27 07:40:46 EST
Created attachment 688418 [details]
engine setup log

No problem, I just ran the setup again for you. Here is the log file
Comment 4 Alex Lourie 2013-01-27 09:21:26 EST
Thanks.

Could you also please run 

/bin/systemctl start  firewalld.service

and post the results?
Comment 5 Netbulae 2013-01-27 18:44:43 EST
NP

"/bin/systemctl start  firewalld.service"

No response but in /var/log/messages the following info:

Jan 28 00:40:13 ovirt01 systemd[1]: Started firewalld - dynamic firewall daemon.

also did a "service firewalld status" and got the following output:

Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
	  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
	  Active: active (running) since Sun 2013-01-27 13:23:43 CET; 11h ago
	Main PID: 21146 (firewalld)
	  CGroup: name=systemd:/system/firewalld.service
		  └─21146 /usr/bin/python /usr/sbin/firewalld --nofork

Jan 27 13:23:43 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.
Jan 27 13:27:21 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.
Jan 28 00:40:13 ***.******.** systemd[1]: Started firewalld - dynamic firewall daemon.
Comment 6 Alex Lourie 2013-01-28 06:44:35 EST
OK, this seems as a strange state of the firewalld.

Could you please rerun the setup now?

Thanks.
Comment 7 Netbulae 2013-01-28 07:34:29 EST
Same error. Also tried with firewalld stopped and still the same error
Comment 8 Netbulae 2013-01-28 07:44:47 EST
With firewalld stopped it got started correctly at 13:32:42 by engine-setup. 

Still a timeout though:

2013-01-28 13:32:42::DEBUG::engine-setup::928::root:: Creating firewalld configuration
2013-01-28 13:32:42::DEBUG::engine-setup::956::root:: configuring firewalld
2013-01-28 13:32:42::DEBUG::common_utils::1228::root:: starting firewalld
2013-01-28 13:32:42::DEBUG::common_utils::1275::root:: executing action firewalld on service start
2013-01-28 13:32:42::DEBUG::common_utils::434::root:: Executing command --> '/sbin/service firewalld start'
2013-01-28 13:32:42::DEBUG::common_utils::472::root:: output = 
2013-01-28 13:32:42::DEBUG::common_utils::473::root:: stderr = Redirecting to /bin/systemctl start  firewalld.service
2013-01-28 13:32:42::DEBUG::common_utils::474::root:: retcode = 0
2013-01-28 13:33:07::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.184:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-01-28 13:33:07::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error


service firewalld status
Redirecting to /bin/systemctl status  firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
	  Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
	  Active: active (running) since Mon 2013-01-28 13:32:42 CET; 11min ago
	Main PID: 24699 (firewalld)
	  CGroup: name=systemd:/system/firewalld.service
		  └─24699 /usr/bin/python /usr/sbin/firewalld --nofork

Jan 28 13:32:42 ****.****.** systemd[1]: Started firewalld - dynamic firewall daemon.
Comment 9 Netbulae 2013-01-28 08:37:47 EST
Did some more testing and I thought it could be a problem that I removed NetworkManager. 

I really hate NM but I reinstalled and configured it, but it still fails with the same error.

When I run "firewall-cmd --get-services", I can see ovirt listed

cluster-suite pop3s bacula-client smtp ipp radius bacula ovirt ftp mdns samba dhcpv6-client https openvpn imaps samba-client http dns ntp vnc-server telnet libvirt ssh ipsec ipp-client amanda-client tftp-client nfs tftp libvirt-tls

The file "/etc/firewalld/services/ovirt.xml" is there but the ports are not actually picked up. When I restart firewalld the ports remain unopened and it looks like ovirt doesn't get added to the zone:

"firewall-cmd --list-all-zones"
drop
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

work
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

internal
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh samba-client
  ports: 
  forward-ports: 
  icmp-blocks: 

external
  interfaces: 
  services: ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

trusted
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

home
  interfaces: 
  services: ipp-client mdns dhcpv6-client ssh samba-client
  ports: 
  forward-ports: 
  icmp-blocks: 

dmz
  interfaces: 
  services: ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

public
  interfaces: eth0
  services: mdns dhcpv6-client ssh
  ports: 
  forward-ports: 
  icmp-blocks: 

block
  interfaces: 
  services: 
  ports: 
  forward-ports: 
  icmp-blocks: 

"firewall-cmd --get-default-zone"
public
Comment 10 Pavel Zhukov 2013-02-01 19:18:51 EST
Problem still persists. 
Patch http://gerrit.ovirt.org/#/c/10493/ has been applied

2013-02-02 01:17:44::ERROR::proxies::410::dbus.proxies:: Introspect error on :1.2:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2013-02-02 01:17:44::DEBUG::proxies::413::dbus.proxies:: Executing introspect queue due to error
Comment 11 Alex Lourie 2013-02-03 04:45:07 EST
This seems to have something to do with selinux-policies - like firewalld can't speak to the NM. Take a look at similar issue https://bugzilla.redhat.com/810508.

Also, please check the version of the selinux-policy package, and the mode of SELINUX on the system.
Comment 12 Netbulae 2013-02-03 06:05:00 EST
Created attachment 915667 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Comment 13 Jason Brooks 2013-02-12 15:05:01 EST
I was hitting this same issue today with oVirt 3.2 beta on F18. Putting selinux in permissive mode worked as a workaround.
Comment 14 Joop van de Wege 2013-02-17 07:16:15 EST
kieppie on IRC had problems with 3.2 and I tried to help him and disabling firewalld and using None in engine-setup let the setup finish. After that I tried with firewalld on and ran into this same problem. Setting SELinux in permissive mode didn't help.
I'm using a minimal install F18 with the steps outlined on the Wiki.

[root@localhost ~]# rpm -aq | grep firewall
firewalld-0.2.12-2.fc18.noarch
[root@localhost ~]# rpm -aq | grep SE
[root@localhost ~]# rpm -aq | grep selin
selinux-policy-targeted-3.11.1-76.fc18.noarch
libselinux-2.1.12-7.fc18.x86_64
eclipselink-2.3.2-1.fc18.noarch
selinux-policy-3.11.1-76.fc18.noarch
libselinux-python-2.1.12-7.fc18.x86_64
libselinux-utils-2.1.12-7.fc18.x86_64
[root@localhost ~]# rpm -aq | grep ovirt
ovirt-release-fedora-5-2.noarch
ovirt-iso-uploader-3.2.0-1.fc18.noarch
ovirt-engine-userportal-3.2.0-4.fc18.noarch
ovirt-engine-backend-3.2.0-4.fc18.noarch
ovirt-engine-genericapi-3.2.0-4.fc18.noarch
ovirt-engine-cli-3.2.0.9-2.fc18.noarch
ovirt-engine-webadmin-portal-3.2.0-4.fc18.noarch
ovirt-engine-restapi-3.2.0-4.fc18.noarch
ovirt-engine-tools-common-3.2.0-4.fc18.noarch
ovirt-engine-dbscripts-3.2.0-4.fc18.noarch
ovirt-host-deploy-1.0.0-1.fc18.noarch
ovirt-log-collector-3.2.0-1.fc18.noarch
ovirt-engine-config-3.2.0-4.fc18.noarch
ovirt-image-uploader-3.2.0-1.fc18.noarch
ovirt-engine-notification-service-3.2.0-4.fc18.noarch
ovirt-host-deploy-java-1.0.0-1.fc18.noarch
ovirt-engine-sdk-3.2.0.8-1.fc18.noarch
ovirt-engine-setup-3.2.0-4.fc18.noarch
ovirt-engine-3.2.0-4.fc18.noarch
Comment 15 Ofer Schreiber 2013-02-27 08:45:36 EST
OK, it seems like a selinux + firewalld integration issue.

Joop - Did you restart the machine after moving selinux to permissive? can you attach your logs?
Comment 16 Ofer Schreiber 2013-02-27 12:09:27 EST
Did some more research:
it seems that the problematic line is:

zones = fw.getActiveZones()

in engine_firewalld.py, which is causing:
type=USER_AVC msg=audit(1361984756.212:1776): pid=461 uid=81 auid=4294967295 ses=4294967295 subj=system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 msg='avc:  denied  { send_msg } for msgtype=method_return dest=:1.24 spid=19402 tpid=19678 scontext=system_u:system_r:firewalld_t:s0 tcontext=unconfined_u:system_r:rpm_t:s0-s0:c0.c1023 tclass=dbus  exe="/usr/bin/dbus-daemon" sauid=81 hostname=? addr=? terminal=?'

in audit.log
Comment 17 Ofer Schreiber 2013-02-27 12:37:50 EST
It seems that miniyum.py is changing the proccess selinux context to rpm_t, which is causing the selinux permissions issue.
Alon - any reasonable solution to that?
Comment 18 Alon Bar-Lev 2013-02-27 14:08:17 EST
(In reply to comment #17)
> It seems that miniyum.py is changing the proccess selinux context to rpm_t,
> which is causing the selinux permissions issue.
> Alon - any reasonable solution to that?

We need the rpm_t context... for rpm operations.
What do we need for firewalld? I am a bit afraid that we won't be able to have both rpm_t and another one... I thought that rpm_t should have all what rpm post scripts can do, including work with dbus...
Comment 19 Juan Hernández 2013-02-28 04:27:27 EST
What specific rpm operations require the rpm_t context?
Comment 20 Alon Bar-Lev 2013-02-28 04:36:01 EST
(In reply to comment #19)
> What specific rpm operations require the rpm_t context?

Installing rpms via the yum api, the %post action fails for various of issues, for example useradd won't work.

There should be no problem of setting a new selinux type before executing firewald <something>...
Comment 21 Juan Hernández 2013-02-28 05:16:39 EST
Isn't it possible to fork a child for the RPM operations and change the context only in the child?
Comment 22 Alon Bar-Lev 2013-02-28 05:20:33 EST
(In reply to comment #21)
> Isn't it possible to fork a child for the RPM operations and change the
> context only in the child?

We need some IPC operation between the two processes, some server to provide yum api and the installer that uses the server, serialization of requests and such.

Or we can write a simple command-line program like yum, but then we lost the transactional benefit of the API.

The yum API was meant to be used in-process... (if was meant to be used at all).

But I don't see the real issue, when executing commands at the execCmd we can set whatever selinux type we like, solving this issue for now.
Comment 23 Ofer Schreiber 2013-03-03 07:37:20 EST
We're not executing anything related to firewalld via execCmd, we use the python api.

Also, the only usage to miniYum in the setup is to create the lock list, any real reason to have the context change in the simple usage of engien-setup?
Comment 24 Alon Bar-Lev 2013-03-03 07:47:39 EST
(In reply to comment #23)
> We're not executing anything related to firewalld via execCmd, we use the
> python api.
> 
> Also, the only usage to miniYum in the setup is to create the lock list, any
> real reason to have the context change in the simple usage of engien-setup?

Are those in engine_firewalld.py should be removed?
---
from gi.repository import GObject
import sys
---

Well, we have a mess here... of course you can remove the miniyum.selinux_role() from engine-setup main prefix. But we may bump in this in future when trying to perform the same.

We should come up with some better solution.
Comment 25 Ofer Schreiber 2013-03-03 10:33:14 EST
Patch posted:
http://gerrit.ovirt.org/#/c/12587/
Comment 26 Alon Bar-Lev 2013-03-03 14:59:52 EST
commit 9df1d2d269940899fc1a3e1c0c02831135e0b455
Author: Alon Bar-Lev <alonbl@redhat.com>
Date:   Sun Mar 3 21:50:00 2013 +0200

    miniyum: we only need system_r
    
    rpm_t type is added automatically by the rpm_execcon(),
    question is why the system_r is not added...
    the rpm_t conflict with other tasks install should do
    such as interactive with dbus.
    
    [sync miniyum with otopi]
    
    Bug-Url: https://bugzilla.redhat.com/show_bug.cgi?id=904153
    Change-Id: I2c9d6e3a9f1f65f403f309bf3b0e0ce2431c8b21
    Signed-off-by: Alon Bar-Lev <alonbl@redhat.com>

---

Untested as at least at my f18, the dbus python implementation goes crazy at random points. But a simple test program does work.
Comment 27 Juan Hernández 2013-03-04 04:48:00 EST
The patch in comment #25 has been merged.
Comment 28 Alon Bar-Lev 2013-03-07 00:54:12 EST
Waiting for merging the new patch.

Note You need to log in before you can comment on or make changes to this bug.