1138731 – firewall-cmd gets stuck when run as superVDSM subsubprocess

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1138731 - firewall-cmd gets stuck when run as superVDSM subsubprocess

Summary: firewall-cmd gets stuck when run as superVDSM subsubprocess

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	selinux-policy
Sub Component:
Version:	7.0
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Miroslav Grepl
QA Contact:	Milos Malik
Docs Contact:
URL:
Whiteboard:	network
Depends On:	1159363
Blocks:	1073943 1143873 1172146
TreeView+	depends on / blocked

Reported:	2014-09-05 14:04 UTC by Ondřej Svoboda
Modified:	2015-09-30 11:53 UTC (History)
CC List:	8 users (show)
Fixed In Version:	selinux-policy-3.13.1-1.el7
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1143873 1172146 (view as bug list)
Environment:
Last Closed:	2015-03-05 10:40:38 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
spec file before configuration (50.47 KB, text/plain) 2014-09-05 15:49 UTC, Ondřej Svoboda	no flags	Details
spec file after configuration (50.44 KB, text/plain) 2014-09-05 15:50 UTC, Ondřej Svoboda	no flags	Details
audit.log send_msg denied (3.75 KB, text/plain) 2014-09-11 08:39 UTC, Petr Horáček	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2015:0458	0	normal	SHIPPED_LIVE	selinux-policy bug fix and enhancement update	2015-03-05 15:17:00 UTC

Description Ondřej Svoboda 2014-09-05 14:04:57 UTC

Description of problem:

VDSM component of oVirt project is used (among many other tasks) to manage and monitor select network devices of a virtualization slave. To do so, it currently uses initscripts (/sbin/ifup and /sbin/ifdown), although alternative approaches are in development.

When asked to create a network on an interface (here, 'dummy_x') VDSM first prepares ifcfg-* file(s) and runs ifup/ifdown under its root-privileged supervdsm service.

For a reason unknown, firewall-cmd (run by ifup/ifdown scripts) takes forever when run under VSDM with firewalld service running. ifup/ifdown alone terminate successfully and promptly.

/sbin/ifdown dummy_x results in calling 'firewall-cmd --zone= --remove-interface=dummy_x'. Processes at the time supervdsm's ifdown is stuck are the following:

├─supervdsmd.service
│ ├─144709 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
│ ├─145326 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
│ ├─145373 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
│ └─145386 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x

When debugging firewall-cmd with winpdb remotely I reached a blocking D-Bus call to method getDefaultZone on org.fedoraproject.FirewallD1 interface (/org/fedoraproject/FirewallD1 object).

In /usr/bin/firewall-cmd this was the trigger:
676 if options_zone_ops and not zone:
677 default = fw.getDefaultZone()

firewalld apparently never manages to receive or process or reply to the call. Maybe the call is even not actually made by firewall-cmd (but by stepping through firewall-cmd's code I have found no indication for this claim, even though D-Bus introspection fails for the method).

dbus-monitor --system shows the following output for ifdown dummy_x when invoked alone:

signal sender=:1.1996 -> dest=(null destination) serial=265 path=/org/fedoraproject/FirewallD1; interface=org.fedoraproject.FirewallD1.zone; member=InterfaceRemoved
string "public"
string "dummy_x"

No communication is shown between firewall-cmd and firewalld when run under supervdsm.

Do you have any hints what apparently causes firewall-cmd to block on calls to firewalld? Could privileges be capped by supervdsmd, D-Bus, PolicyKit, systemd?

I can offer access to my host and any assistance/information necessary.

Thanks!

Version-Release number of selected component (if applicable):
firewalld-0.3.9-7.el7.noarch
vdsm Git master

How reproducible:
100%

Steps to Reproduce:

1. Create the dummy network interface.

ip link add name dummy_x type dummy

2. Let VDSM create the ifcfg file and ifdown it (in preparation for ifup'ing it again).

vdsClient -s 0 setupNetworks networks='{test-firewalld:{nic:dummy_x,bridged:false}}'

Actual results:

supervdsmd is blocked by firewall-cmd, which never finishes.

Expected results:

firewall-cmd contacts firewalld and exits successfully.

Additional info:

Running /usr/sbin/ifdown dummy_x with the following ifcfg file is successful.

[root@localhost network-scripts]# cat ifcfg-dummy_x
# Generated by VDSM version 4.16.0-278.git189f68c.el7
DEVICE=dummy_x
HWADDR=f6:6e:b6:1e:1e:71
ONBOOT=no
DEFROUTE=no
NM_CONTROLLED=no

Comment 5 Ondřej Svoboda 2014-09-05 15:46:18 UTC

After running setenforce 0 VDSM/firewall-cmd was no longer blocked.

The VDSM packages have been created by running

./autogen --system
make rpm

in VDSM Git repository cloned from git://gerrit.ovirt.org/vdsm

I will attach vdsm.spec(.in) files.

Comment 6 Ondřej Svoboda 2014-09-05 15:49:29 UTC

Created attachment 934851 [details]
spec file before configuration

Comment 7 Ondřej Svoboda 2014-09-05 15:50:50 UTC

Created attachment 934852 [details]
spec file after configuration

Comment 8 Miroslav Grepl 2014-09-08 15:26:42 UTC

#============= firewalld_t ==============
allow firewalld_t init_t:dbus send_msg;

What does

# ps -efZ |grep init_t

during testing?

Comment 9 Miroslav Grepl 2014-09-08 15:32:31 UTC

Also what does

# ls -Z /usr/share/vdsm/supervdsmServer

Comment 10 Dan Kenigsberg 2014-09-08 17:12:53 UTC

Here is a different condition - I hope it helps:

systemd(1)---vdsmd_init_comm(18331)---python(18393)---ifdown-eth(18406)---ifdown-post(18436)---firewall-cmd(18446)

# ps -efZ |grep init_t
system_u:system_r:init_t:s0     root         1     0  0 Sep04 ?        00:00:39 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root       627     1  0 Sep04 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root     17475     1  0 18:00 ?        00:00:00 /bin/sh /usr/libexec/vdsm/vdsmd_init_common.sh --pre-start
system_u:system_r:init_t:s0     root     17537 17475  1 18:00 ?        00:00:00 /usr/bin/python /usr/share/vdsm/vdsm-restore-net-config
system_u:system_r:init_t:s0     root     17559 17537  0 18:00 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-test-network
system_u:system_r:init_t:s0     root     17580 17559  0 18:00 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-test-network
system_u:system_r:init_t:s0     root     17590 17580  1 18:00 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 17596 17456  0 18:00 pts/0 00:00:00 grep --color=auto init_t

# ls -lZ /usr/share/vdsm/vdsm-restore-net-config
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/vdsm-restore-net-config

# ls -lZ /usr/share/vdsm/supervdsmServer 
-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 11 Ondřej Svoboda 2014-09-09 08:44:51 UTC

And the original :-)

[root@localhost ~]# ps -efZ |grep init_t

system_u:system_r:init_t:s0     root          1      0  0 Sep05 ?        00:00:08 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      56667      1  0 10:38 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      56670      1  0 10:38 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     root      56701  56667  0 10:38 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --zone= --change-interface=dummy_x
system_u:system_r:init_t:s0     vdsm      56834      1  5 10:38 ?        00:00:06 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root      57033  56670  0 10:39 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifdown-eth ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57080  57033  0 10:39 ?        00:00:00 /bin/sh /etc/sysconfig/network-scripts/ifdown-post ifcfg-dummy_x
system_u:system_r:init_t:s0     root      57093  57080  0 10:39 ?        00:00:00 /usr/bin/python -Es /usr/bin/firewall-cmd --remove-interface=dummy_x
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 57105 56921  0 10:40 pts/1 00:00:00 grep --color=auto init_t

[root@localhost ~]# ls -Z /usr/share/vdsm/supervdsmServer 

-rwxr-xr-x. root root system_u:object_r:usr_t:s0       /usr/share/vdsm/supervdsmServer

Comment 12 Petr Horáček 2014-09-09 10:24:15 UTC

While VDSM, iproute2 configurator, network functional tests, this happens to me:

Traceback (most recent call last):
  File "/usr/share/vdsm/supervdsmServer", line 101, in wrapper
    res = func(*args, **kwargs)
  File "/usr/share/vdsm/supervdsmServer", line 201, in addNetwork
    return addNetwork(bridge, **options)
  File "/usr/share/vdsm/network/api.py", line 247, in wrapped
    ret = func(**attrs)
  File "/usr/share/vdsm/network/api.py", line 338, in addNetwork
    netEnt.configure(**options)
  File "/usr/share/vdsm/network/models.py", line 169, in configure
    self.configurator.configureBridge(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 73, in configureBridge
    bridge.port.configure(**opts)
  File "/usr/share/vdsm/network/models.py", line 133, in configure
    self.configurator.configureVlan(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 83, in configureVlan
    vlan.device.configure(**opts)
  File "/usr/share/vdsm/network/models.py", line 215, in configure
    self.configurator.configureBond(self, **opts)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 93, in configureBond
    self.configApplier.addBondOptions(bond)
  File "/usr/share/vdsm/network/configurators/iproute2.py", line 342, in addBondOptions
    f.write(value)
IOError: [Errno 1] Operation not permitted

It's ok with disabled SELinux. It may be because of this bug.

CentOS 7

Comment 13 Miroslav Grepl 2014-09-09 13:46:47 UTC

Did you test it with the latest policy build?

Comment 14 Petr Horáček 2014-09-09 15:12:08 UTC

It's still the same with selinux-policy-3.13.1-42.fc21.

iproute2:
one network's functional test failed with 'Operation not permitted' for the first time. For the second time it passed, but another failed.
UPDATE: it raises that error even with SELinux disabled

ifcfg:
tests are blocked, it just show name of the first test.

i'll do more investigation to be sure, that it is not my environment's problem.

Comment 15 Dan Kenigsberg 2014-09-09 16:06:36 UTC

Petr, would you check /var/log/audit/audit.log for (possibly different) denials? Attach it here, if you find any.

Comment 16 Miroslav Grepl 2014-09-10 07:15:52 UTC

So it does not work also in permissive mode, right?

Comment 17 Petr Horáček 2014-09-11 08:39:14 UTC

Created attachment 936440 [details]
audit.log send_msg denied

Comment 18 Petr Horáček 2014-09-11 08:46:37 UTC

In permissive mode it works.

Comment 19 Miroslav Grepl 2014-09-11 14:20:51 UTC

(In reply to Petr Horáček from comment #18)
> In permissive mode it works.

Ok and how does 

# ps -efZ |grep init_t

look for you on F21?

Comment 20 Petr Horáček 2014-09-17 08:48:11 UTC

(In reply to Miroslav Grepl from comment #19)
> (In reply to Petr Horáček from comment #18)
> > In permissive mode it works.
> 
> Ok and how does 
> 
> # ps -efZ |grep init_t
> 
> look for you on F21?

I'm not able to install current nightly Fedora 21.

On CentOS 7, permissive mode:
 
- ifcfg configurator:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root      1860     1  1 04:36 ?        00:00:04 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm      1939     1  7 04:36 ?        00:00:21 /usr/bin/python /usr/share/vdsm/vdsm
system_u:system_r:init_t:s0     root     10104  1860  0 04:41 ?        00:00:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-test-network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 10161 20524  0 04:41 pts/1 00:00:00 grep --color=auto init_t

- iproute2 configuraor:
system_u:system_r:init_t:s0     root         1     0  0 04:25 ?        00:00:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 23
system_u:system_r:init_t:s0     root     10426     1  4 04:42 ?        00:00:00 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock
system_u:system_r:init_t:s0     vdsm     10505     1 13 04:42 ?        00:00:02 /usr/bin/python /usr/share/vdsm/vdsm

Comment 24 Milos Malik 2014-12-16 11:51:25 UTC

The automated TC passed. The bug is fixed on selinux-policy side.

Comment 27 errata-xmlrpc 2015-03-05 10:40:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0458.html

Note You need to log in before you can comment on or make changes to this bug.