Bug 1138731 - firewall-cmd gets stuck when run as superVDSM subsubprocess
Summary: firewall-cmd gets stuck when run as superVDSM subsubprocess
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy
Version: 7.0
Hardware: x86_64
OS: Linux
Target Milestone: rc
: ---
Assignee: Miroslav Grepl
QA Contact: Milos Malik
Whiteboard: network
Depends On: 1159363
Blocks: 1073943 1143873 1172146
TreeView+ depends on / blocked
Reported: 2014-09-05 14:04 UTC by Ondřej Svoboda
Modified: 2015-09-30 11:53 UTC (History)
8 users (show)

Fixed In Version: selinux-policy-3.13.1-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1143873 1172146 (view as bug list)
Last Closed: 2015-03-05 10:40:38 UTC
Target Upstream Version:

Attachments (Terms of Use)
spec file before configuration (50.47 KB, text/plain)
2014-09-05 15:49 UTC, Ondřej Svoboda
no flags Details
spec file after configuration (50.44 KB, text/plain)
2014-09-05 15:50 UTC, Ondřej Svoboda
no flags Details
audit.log send_msg denied (3.75 KB, text/plain)
2014-09-11 08:39 UTC, Petr Horáček
no flags Details

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0458 normal SHIPPED_LIVE selinux-policy bug fix and enhancement update 2015-03-05 15:17:00 UTC

Description Ondřej Svoboda 2014-09-05 14:04:57 UTC
Description of problem:

VDSM component of oVirt project is used (among many other tasks) to manage and monitor select network devices of a virtualization slave. To do so, it currently uses initscripts (/sbin/ifup and /sbin/ifdown), although alternative approaches are in development.

When asked to create a network on an interface (here, 'dummy_x') VDSM first prepares ifcfg-* file(s) and runs ifup/ifdown under its root-privileged supervdsm service.

For a reason unknown, firewall-cmd (run by ifup/ifdown scripts) takes forever when run under VSDM with firewalld service running. ifup/ifdown alone terminate successfully and promptly.

/sbin/ifdown dummy_x results in calling 'firewall-cmd --zone= --remove-interface=dummy_x'. Processes at the time supervdsm's ifdown is stuck are the following:

  │ ├─144709 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
  │ ├─145326 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
  │ ├─145373 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
  │ └─145386 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x

When debugging firewall-cmd with winpdb remotely I reached a blocking D-Bus call to method getDefaultZone on org.fedoraproject.FirewallD1 interface (/org/fedoraproject/FirewallD1 object).

In /usr/bin/firewall-cmd this was the trigger:
676 if options_zone_ops and not zone:
677     default = fw.getDefaultZone()

firewalld apparently never manages to receive or process or reply to the call. Maybe the call is even not actually made by firewall-cmd (but by stepping through firewall-cmd's code I have found no indication for this claim, even though D-Bus introspection fails for the method).

dbus-monitor --system shows the following output for ifdown dummy_x when invoked alone:

signal sender=:1.1996 -> dest=(null destination) serial=265 path=/org/fedoraproject/FirewallD1; interface=org.fedoraproject.FirewallD1.zone; member=InterfaceRemoved
   string "public"
   string "dummy_x"

No communication is shown between firewall-cmd and firewalld when run under supervdsm.

Do you have any hints what apparently causes firewall-cmd to block on calls to firewalld? Could privileges be capped by supervdsmd, D-Bus, PolicyKit, systemd?

I can offer access to my host and any assistance/information necessary.


Version-Release number of selected component (if applicable):
vdsm Git master

How reproducible:

Steps to Reproduce:

1. Create the dummy network interface.

ip link add name dummy_x type dummy

2. Let VDSM create the ifcfg file and ifdown it (in preparation for ifup'ing it again).

vdsClient -s 0 setupNetworks networks='{test-firewalld:{nic:dummy_x,bridged:false}}'

Actual results:

supervdsmd is blocked by firewall-cmd, which never finishes.

Expected results:

firewall-cmd contacts firewalld and exits successfully.

Additional info:

Running /usr/sbin/ifdown dummy_x with the following ifcfg file is successful.

[root@localhost network-scripts]# cat ifcfg-dummy_x
# Generated by VDSM version 4.16.0-278.git189f68c.el7

Comment 5 Ondřej Svoboda 2014-09-05 15:46:18 UTC
After running setenforce 0 VDSM/firewall-cmd was no longer blocked.

The VDSM packages have been created by running

./autogen --system
make rpm

in VDSM Git repository cloned from git://gerrit.ovirt.org/vdsm

I will attach vdsm.spec(.in) files.

Comment 6 Ondřej Svoboda 2014-09-05 15:49:29 UTC
Created attachment 934851 [details]
spec file before configuration

Comment 7 Ondřej Svoboda 2014-09-05 15:50:50 UTC
Created attachment 934852 [details]
spec file after configuration

Comment 8 Miroslav Grepl 2014-09-08 15:26:42 UTC
#============= firewalld_t ==============
allow firewalld_t init_t:dbus send_msg;

What does

# ps -efZ |grep init_t

during testing?

Comment 9 Miroslav Grepl 2014-09-08 15:32:31 UTC
Also what does

# ls -Z /usr/share/vdsm/supervdsmServer

Comment 10 Dan Kenigsberg 2014-09-08 17:12:53 UTC
Here is a different condition - I hope it helps:


# ps -efZ |grep init_t
system_u:system_r:init_t:s0     root         1     0  0 Sep04 ?        00:00:39 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root       627     1  0 Sep04 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root     17475     1  0 18:00 ?        00:00:00 /bin/sh /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
system_u:system_r:init_t:s0     root     17537 17475  1 18:00 ?        00:00:00 /usr/bin/python /usr/share/vdsm/vdsm-restore-net-config
system_u:system_r:init_t:s0     root     17559 17537  0 18:00 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-test-network
system_u:system_r:init_t:s0     root     17580 17559  0 18:00 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-test-network
system_u:system_r:init_t:s0     root     17590 17580  1 18:00 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 17596 17456  0 18:00 pts/0 00:00:00 grep --color=auto init_t

# ls -lZ /usr/share/vdsm/vdsm-restore-net-config
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/vdsm-restore-net-config

# ls -lZ /usr/share/vdsm/supervdsmServer 
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 11 Ondřej Svoboda 2014-09-09 08:44:51 UTC
And the original :-)

[root@localhost ~]# ps -efZ |grep init_t

system_u:system_r:init_t:s0     root          1      0  0 Sep05 ?        00:00:08 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      56667      1  0 10:38 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      56670      1  0 10:38 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root      56701  56667  0 10:38 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --zone= --change-interface=dummy_x
system_u:system_r:init_t:s0     vdsm      56834      1  5 10:38 ?        00:00:06 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root      57033  56670  0 10:39 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57080  57033  0 10:39 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57093  57080  0 10:39 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 57105 56921  0 10:40 pts/1 00:00:00 grep --color=auto init_t

[root@localhost ~]# ls -Z /usr/share/vdsm/supervdsmServer 

-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 12 Petr Horáček 2014-09-09 10:24:15 UTC
While VDSM, iproute2 configurator, network functional tests, this happens to me:

Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 101, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 201, in addNetwork
    return addNetwork(bridge, **options)
  File "/usr/share/vdsm/network/api.py", line 247, in wrapped
    ret = func(**attrs)
  File "/usr/share/vdsm/network/api.py", line 338, in addNetwork
  File "/usr/share/vdsm/network/models.py", line 169, in configure
    self.configurator.configureBridge(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 73, in configureBridge
  File "/usr/share/vdsm/network/models.py", line 133, in configure
    self.configurator.configureVlan(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 83, in configureVlan
  File "/usr/share/vdsm/network/models.py", line 215, in configure
    self.configurator.configureBond(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 93, in configureBond
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 342, in addBondOptions
IOError: [Errno 1] Operation not permitted

It's ok with disabled SELinux. It may be because of this bug.

CentOS 7

Comment 13 Miroslav Grepl 2014-09-09 13:46:47 UTC
Did you test it with the latest policy build?

Comment 14 Petr Horáček 2014-09-09 15:12:08 UTC
It's still the same with selinux-policy-3.13.1-42.fc21.

one network's functional test failed with 'Operation not permitted' for the first time. For the second time it passed, but another failed.
UPDATE: it raises that error even with SELinux disabled

tests are blocked, it just show name of the first test.

i'll do more investigation to be sure, that it is not my environment's problem.

Comment 15 Dan Kenigsberg 2014-09-09 16:06:36 UTC
Petr, would you check /var/log/audit/audit.log for (possibly different) denials? Attach it here, if you find any.

Comment 16 Miroslav Grepl 2014-09-10 07:15:52 UTC
So it does not work also in permissive mode, right?

Comment 17 Petr Horáček 2014-09-11 08:39:14 UTC
Created attachment 936440 [details]
audit.log send_msg denied

Comment 18 Petr Horáček 2014-09-11 08:46:37 UTC
In permissive mode it works.

Comment 19 Miroslav Grepl 2014-09-11 14:20:51 UTC
(In reply to Petr Horáček from comment #18)
> In permissive mode it works.

Ok and how does 

# ps -efZ |grep init_t

look for you on F21?

Comment 20 Petr Horáček 2014-09-17 08:48:11 UTC
(In reply to Miroslav Grepl from comment #19)
> (In reply to Petr Horáček from comment #18)
> > In permissive mode it works.
> Ok and how does 
> # ps -efZ |grep init_t
> look for you on F21?

I'm not able to install current nightly Fedora 21.

On CentOS 7, permissive mode:
- ifcfg configurator:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      1860     1  1 04:36 ?        00:00:04 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm      1939     1  7 04:36 ?        00:00:21 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root     10104  1860  0 04:41 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 10161 20524  0 04:41 pts/1 00:00:00 grep --color=auto init_t

- iproute2 configuraor:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root     10426     1  4 04:42 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm     10505     1 13 04:42 ?        00:00:02 /usr/bin/python /usr/share/vdsm/vdsm

Comment 24 Milos Malik 2014-12-16 11:51:25 UTC
The automated TC passed. The bug is fixed on selinux-policy side.

Comment 27 errata-xmlrpc 2015-03-05 10:40:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.