Bug 1138731

Summary: firewall-cmd gets stuck when run as superVDSM subsubprocess
Product: Red Hat Enterprise Linux 7 Reporter: Ondřej Svoboda <osvoboda>
Component: selinux-policyAssignee: Miroslav Grepl <mgrepl>
Status: CLOSED ERRATA QA Contact: Milos Malik <mmalik>
Severity: high Docs Contact:
Priority: high    
Version: 7.0CC: danken, hkario, mmalik, osvoboda, phoracek, s.kieske, tlavigne, todoleza
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: network
Fixed In Version: selinux-policy-3.13.1-1.el7 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1143873 1172146 (view as bug list) Environment:
Last Closed: 2015-03-05 10:40:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1159363    
Bug Blocks: 1073943, 1143873, 1172146    
Attachments:
Description Flags
spec file before configuration
none
spec file after configuration
none
audit.log send_msg denied none

Description Ondřej Svoboda 2014-09-05 14:04:57 UTC
Description of problem:

VDSM component of oVirt project is used (among many other tasks) to manage and monitor select network devices of a virtualization slave. To do so, it currently uses initscripts (/sbin/ifup and /sbin/ifdown), although alternative approaches are in development.

When asked to create a network on an interface (here, 'dummy_x') VDSM first prepares ifcfg-* file(s) and runs ifup/ifdown under its root-privileged supervdsm service.

For a reason unknown, firewall-cmd (run by ifup/ifdown scripts) takes forever when run under VSDM with firewalld service running. ifup/ifdown alone terminate successfully and promptly.

/sbin/ifdown dummy_x results in calling 'firewall-cmd --zone= --remove-interface=dummy_x'. Processes at the time supervdsm's ifdown is stuck are the following:

  ├─supervdsmd.service
  │ ├─144709 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
  │ ├─145326 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
  │ ├─145373 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
  │ └─145386 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x

When debugging firewall-cmd with winpdb remotely I reached a blocking D-Bus call to method getDefaultZone on org.fedoraproject.FirewallD1 interface (/org/fedoraproject/FirewallD1 object).

In /usr/bin/firewall-cmd this was the trigger:
676 if options_zone_ops and not zone:
677     default = fw.getDefaultZone()

firewalld apparently never manages to receive or process or reply to the call. Maybe the call is even not actually made by firewall-cmd (but by stepping through firewall-cmd's code I have found no indication for this claim, even though D-Bus introspection fails for the method).

dbus-monitor --system shows the following output for ifdown dummy_x when invoked alone:

signal sender=:1.1996 -> dest=(null destination) serial=265 path=/org/fedoraproject/FirewallD1; interface=org.fedoraproject.FirewallD1.zone; member=InterfaceRemoved
   string "public"
   string "dummy_x"

No communication is shown between firewall-cmd and firewalld when run under supervdsm.

Do you have any hints what apparently causes firewall-cmd to block on calls to firewalld? Could privileges be capped by supervdsmd, D-Bus, PolicyKit, systemd?

I can offer access to my host and any assistance/information necessary.

Thanks!

Version-Release number of selected component (if applicable):
firewalld-0.3.9-7.el7.noarch
vdsm Git master

How reproducible:
100%

Steps to Reproduce:

1. Create the dummy network interface.

ip link add name dummy_x type dummy

2. Let VDSM create the ifcfg file and ifdown it (in preparation for ifup'ing it again).

vdsClient -s 0 setupNetworks networks='{test-firewalld:{nic:dummy_x,bridged:false}}'

Actual results:

supervdsmd is blocked by firewall-cmd, which never finishes.

Expected results:

firewall-cmd contacts firewalld and exits successfully.

Additional info:

Running /usr/sbin/ifdown dummy_x with the following ifcfg file is successful.

[root@localhost network-scripts]# cat ifcfg-dummy_x
# Generated by VDSM version 4.16.0-278.git189f68c.el7
DEVICE=dummy_x
HWADDR=f6:6e:b6:1e:1e:71
ONBOOT=no
DEFROUTE=no
NM_CONTROLLED=no

Comment 5 Ondřej Svoboda 2014-09-05 15:46:18 UTC
After running setenforce 0 VDSM/firewall-cmd was no longer blocked.

The VDSM packages have been created by running

./autogen --system
make rpm

in VDSM Git repository cloned from git://gerrit.ovirt.org/vdsm

I will attach vdsm.spec(.in) files.

Comment 6 Ondřej Svoboda 2014-09-05 15:49:29 UTC
Created attachment 934851 [details]
spec file before configuration

Comment 7 Ondřej Svoboda 2014-09-05 15:50:50 UTC
Created attachment 934852 [details]
spec file after configuration

Comment 8 Miroslav Grepl 2014-09-08 15:26:42 UTC
#============= firewalld_t ==============
allow firewalld_t init_t:dbus send_msg;

What does

# ps -efZ |grep init_t

during testing?

Comment 9 Miroslav Grepl 2014-09-08 15:32:31 UTC
Also what does

# ls -Z /usr/share/vdsm/supervdsmServer

Comment 10 Dan Kenigsberg 2014-09-08 17:12:53 UTC
Here is a different condition - I hope it helps:

systemd(1)---vdsmd_init_comm(18331)---python(18393)---ifdown-eth(18406)---ifdown-post(18436)---firewall-cmd(18446)

# ps -efZ |grep init_t
system_u:system_r:init_t:s0     root         1     0  0 Sep04 ?        00:00:39 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root       627     1  0 Sep04 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root     17475     1  0 18:00 ?        00:00:00 /bin/sh /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
system_u:system_r:init_t:s0     root     17537 17475  1 18:00 ?        00:00:00 /usr/bin/python /usr/share/vdsm/vdsm-restore-net-config
system_u:system_r:init_t:s0     root     17559 17537  0 18:00 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-test-network
system_u:system_r:init_t:s0     root     17580 17559  0 18:00 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-test-network
system_u:system_r:init_t:s0     root     17590 17580  1 18:00 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 17596 17456  0 18:00 pts/0 00:00:00 grep --color=auto init_t

# ls -lZ /usr/share/vdsm/vdsm-restore-net-config
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/vdsm-restore-net-config

# ls -lZ /usr/share/vdsm/supervdsmServer 
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 11 Ondřej Svoboda 2014-09-09 08:44:51 UTC
And the original :-)

[root@localhost ~]# ps -efZ |grep init_t

system_u:system_r:init_t:s0     root          1      0  0 Sep05 ?        00:00:08 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      56667      1  0 10:38 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      56670      1  0 10:38 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root      56701  56667  0 10:38 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --zone= --change-interface=dummy_x
system_u:system_r:init_t:s0     vdsm      56834      1  5 10:38 ?        00:00:06 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root      57033  56670  0 10:39 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57080  57033  0 10:39 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57093  57080  0 10:39 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 57105 56921  0 10:40 pts/1 00:00:00 grep --color=auto init_t

[root@localhost ~]# ls -Z /usr/share/vdsm/supervdsmServer 

-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 12 Petr Horáček 2014-09-09 10:24:15 UTC
While VDSM, iproute2 configurator, network functional tests, this happens to me:

Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 101, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 201, in addNetwork
    return addNetwork(bridge, **options)
  File "/usr/share/vdsm/network/api.py", line 247, in wrapped
    ret = func(**attrs)
  File "/usr/share/vdsm/network/api.py", line 338, in addNetwork
    netEnt.configure(**options)
  File "/usr/share/vdsm/network/models.py", line 169, in configure
    self.configurator.configureBridge(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 73, in configureBridge
    bridge.port.configure(**opts)
  File "/usr/share/vdsm/network/models.py", line 133, in configure
    self.configurator.configureVlan(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 83, in configureVlan
    vlan.device.configure(**opts)
  File "/usr/share/vdsm/network/models.py", line 215, in configure
    self.configurator.configureBond(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 93, in configureBond
    self.configApplier.addBondOptions(bond)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 342, in addBondOptions
    f.write(value)
IOError: [Errno 1] Operation not permitted

It's ok with disabled SELinux. It may be because of this bug.

CentOS 7

Comment 13 Miroslav Grepl 2014-09-09 13:46:47 UTC
Did you test it with the latest policy build?

Comment 14 Petr Horáček 2014-09-09 15:12:08 UTC
It's still the same with selinux-policy-3.13.1-42.fc21.

iproute2:
one network's functional test failed with 'Operation not permitted' for the first time. For the second time it passed, but another failed.
UPDATE: it raises that error even with SELinux disabled

ifcfg:
tests are blocked, it just show name of the first test.

i'll do more investigation to be sure, that it is not my environment's problem.

Comment 15 Dan Kenigsberg 2014-09-09 16:06:36 UTC
Petr, would you check /var/log/audit/audit.log for (possibly different) denials? Attach it here, if you find any.

Comment 16 Miroslav Grepl 2014-09-10 07:15:52 UTC
So it does not work also in permissive mode, right?

Comment 17 Petr Horáček 2014-09-11 08:39:14 UTC
Created attachment 936440 [details]
audit.log send_msg denied

Comment 18 Petr Horáček 2014-09-11 08:46:37 UTC
In permissive mode it works.

Comment 19 Miroslav Grepl 2014-09-11 14:20:51 UTC
(In reply to Petr Horáček from comment #18)
> In permissive mode it works.

Ok and how does 

# ps -efZ |grep init_t

look for you on F21?

Comment 20 Petr Horáček 2014-09-17 08:48:11 UTC
(In reply to Miroslav Grepl from comment #19)
> (In reply to Petr Horáček from comment #18)
> > In permissive mode it works.
> 
> Ok and how does 
> 
> # ps -efZ |grep init_t
> 
> look for you on F21?

I'm not able to install current nightly Fedora 21.

On CentOS 7, permissive mode:
 
- ifcfg configurator:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      1860     1  1 04:36 ?        00:00:04 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm      1939     1  7 04:36 ?        00:00:21 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root     10104  1860  0 04:41 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 10161 20524  0 04:41 pts/1 00:00:00 grep --color=auto init_t

- iproute2 configuraor:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root     10426     1  4 04:42 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm     10505     1 13 04:42 ?        00:00:02 /usr/bin/python /usr/share/vdsm/vdsm

Comment 24 Milos Malik 2014-12-16 11:51:25 UTC
The automated TC passed. The bug is fixed on selinux-policy side.

Comment 27 errata-xmlrpc 2015-03-05 10:40:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0458.html