Bug 889782 - dnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory (due to selinux)
dnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lea...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-selinux (Show other bugs)
2.0 (Folsom)
Unspecified Unspecified
high Severity high
: snapshot3
: 2.1
Assigned To: Lon Hohberger
Ofer Blaut
: Reopened, Triaged
Depends On: 902486
Blocks: quantum_ovs_tracker
  Show dependency treegraph
 
Reported: 2012-12-23 09:16 EST by Gary Kotton
Modified: 2016-04-26 23:55 EDT (History)
7 users (show)

See Also:
Fixed In Version: openstack-selinux-0.1.2-4.el6ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-21 15:04:11 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
audit log with AVCs (372.50 KB, text/x-log)
2013-01-25 12:49 EST, Bob Kukura
no flags Details
add corecmd_exec_shell(dnsmasq_t) (638 bytes, patch)
2013-02-19 16:50 EST, Lon Hohberger
no flags Details | Diff
Current full openstack-selinux-quantum policy file incl. patch. (889 bytes, text/plain)
2013-02-19 16:53 EST, Lon Hohberger
no flags Details

  None (edit)
Description Gary Kotton 2012-12-23 09:16:32 EST
Description of problem:
Quantum DHCP log file shows that it is unable to run the dhcp lease process.
When the command is invoked manually it works (looks like a rootwrap issue)

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Start quantum service
2. Create a network
3. Create a subnet on the network
  
Actual results:


Expected results:


Additional info:
Trace from the logfile is as follows:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=58eea38a-eddd-4240-a0a3-c752631f84b9', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tapa6f95c07-8c', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=set:tag0,10.0.0.0,static,120s']
Exit code: 3
Stdout: ''
Stderr: '\ndnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory\n'
2012-12-23 15:50:02     INFO [quantum.agent.dhcp_agent] Synchronizing state
2012-12-23 15:50:04    ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 91, in call_driver
    getattr(driver, action)()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 109, in enable
    self.restart()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 83, in restart
    self.enable()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 112, in enable
    self.spawn_process()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 261, in spawn_process
    utils.execute(cmd, self.root_helper)
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 63, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=58eea38a-eddd-4240-a0a3-c752631f84b9', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tapa6f95c07-8c', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/58eea38a-eddd-4240-a0a3-c752631f84b9/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=set:tag0,10.0.0.0,static,120s']
Exit code: 3
Stdout: ''
Stderr: '\ndnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory\n'
Comment 2 Gary Kotton 2012-12-24 09:35:05 EST

*** This bug has been marked as a duplicate of bug 887369 ***
Comment 3 Chris Wright 2012-12-24 13:40:02 EST
(In reply to comment #2)
> 
> *** This bug has been marked as a duplicate of bug 887369 ***

Hmm, this doesn't look like a dupe.  It looks like the dhcp agent is installed w/out quantum installed.  The options cmdline doesn't create an error, just doesn't work.  Whereas a non-existent --dhcp-script file will create the "No such file of directory" error.  Is it an SELinux issue, or packaging problem?  Does /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update exist on that network node?
Comment 4 Gary Kotton 2012-12-25 02:20:00 EST
Hi,
Chris you are correct. This is an SELinux issue. When the mode is permissive this no longer occurs.
We need to ensure that this is documented.
Thanks
Gary
Comment 5 Yaniv Kaul 2012-12-25 02:49:58 EST
(In reply to comment #4)
> Hi,
> Chris you are correct. This is an SELinux issue. When the mode is permissive
> this no longer occurs.
> We need to ensure that this is documented.
> Thanks
> Gary

I'm re-opening it, as there are two options here - either there should be an selinux policy for this (and the package should depend on it, RPM-wise), or the installation (RPM's post install script) should enable some selinux parameter.
Documenting selinux issues rarely help (and gives selinux a bad name!).
Comment 6 Daniel Walsh 2013-01-17 15:31:58 EST
When you get an SELinux issue please CC dwalsh@redhat.com and mgrepl@redhat.com to it.  Or open it as a bug on selinux-policy so we see it.

Do you have any AVC information on this?
Comment 7 Miroslav Grepl 2013-01-17 15:56:40 EST
We need to see AVC msgs.
Comment 8 Bob Kukura 2013-01-25 12:49:04 EST
Created attachment 687597 [details]
audit log with AVCs
Comment 9 Bob Kukura 2013-01-25 13:22:35 EST
I've attached an audit.log extract containing AVCs while reproducing this issue (in enforcing mode).

Note that the selinux policies for quantum were created about a year ago for the OpenStack essex release on fedora 17, and I'm not sure they've been updated for the folsom release that's in fedora 18 and is the basis for RHOS 2.X. 

I see in /etc/selinux/targeted/contexts/files/file_contexts:

/usr/bin/quantum-server --      system_u:object_r:quantum_exec_t:s0
/usr/bin/quantum-ryu-agent      --      system_u:object_r:quantum_exec_t:s0
/usr/bin/quantum-openvswitch-agent      --   system_u:object_r:quantum_exec_t:s0
/usr/bin/quantum-linuxbridge-agent      --   system_u:object_r:quantum_exec_t:s0

But I don't see anything for newer daemons that also run commands as root:

/usr/bin/quantum-dhcp-agent
/usr/bin/quantum-l3-agent
/usr/bin/quantum-ryu-agent

There are some additional quantum executables that may also need policies:

/usr/bin/quantum-debug
/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
/usr/bin/quantum-netns-cleanup
/usr/bin/quantum-rootwrap

And a new executable run by an init script at boot is about to go in:

/usr/bin/quantum-ovs-cleanup

Note that all/most of these run commands as root via /usr/bin/quantum-rootwrap, similar to nova and other OpenStack components. The configuration files for quantum-rootwrap may be a good summary of the set of commands that quantum services need to execute.

RHEL 6.4 and Fedora 18 would seem to need similar selinux policies for quantum. Fedora rawhide will include the grizzly release of quantum, with additional daemons, etc..

Also, https://bugzilla.redhat.com/show_bug.cgi?id=878846 is related to selinux policies for quantum.
Comment 10 Bob Kukura 2013-01-28 12:53:15 EST
Miroslav,

I've provided the AVCs and some additional info in comment 9. Can you please see if we can get the policies updated in time for our release, or if any additional info is needed?

Thanks,

-Bob
Comment 11 Miroslav Grepl 2013-01-30 03:46:40 EST
I apologize, I was sick.


So which of these binaries are executed by an init script? Just

/usr/bin/quantum-ovs-cleanup

? 

We need to change labeling for binaries which are executed by an init script.

# chcon -t quantum_exec_t $TARGET_BINARIES


Also please execute

# chcon -t bin_t /usr/lib/python2.6/site-packages/quantum/agent/linux/*.py

And then add this local policy module

# cat mypol.te
policy_module(mypol,1.0)

require{
 type dnsmasq_t;
}
corecmd_exec_bin(dnsmasq_t)


and run

# make -f /usr/share/selinux/devel/Makefile mypol.pp
# semodule -i mypol.pp

Re-test and

# ausearch -m avc -ts recent
Comment 12 Bob Kukura 2013-01-30 21:08:10 EST
Hi Miroslav,

Thanks! I will try your comment 11 suggestion as soon as possible and let you know the result.

Of the executables listed in comment 9, all of the following are executed directly by init scripts:

/usr/bin/quantum-server
/usr/bin/quantum-ryu-agent
/usr/bin/quantum-openvswitch-agent
/usr/bin/quantum-linuxbridge-agent
/usr/bin/quantum-dhcp-agent
/usr/bin/quantum-l3-agent
/usr/bin/quantum-ryu-agent
/usr/bin/quantum-ovs-cleanup


This is executed by most of the programs above in order to run various commands as root:

/usr/bin/quantum-rootwrap


And quantum-dhcp-agent, via rootwrap, executes dnsmasq, which in turn executes:

/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update
Comment 13 Miroslav Grepl 2013-01-31 02:16:06 EST
Ok, please make sure the are labeled as quantum_exec_t how I wrote above.
Comment 14 Gary Kotton 2013-01-31 05:17:43 EST
Hi,
Prior to running the code in comment 11 I was unable to get a DHCP lease for Quantum
After running the code in comment 11 it works.

Console trace:

[    1.625269] Freeing unused kernel memory: 1968k freed
[    1.676884] Freeing unused kernel memory: 1368k freed

info: initramfs: up at 1.77
NOCHANGE: partition 1 is size 64260. it cannot be grown
info: initramfs loading root from /dev/vda1
info: /etc/init.d/rc.sysinit: up at 2.67
[    2.704451] EXT3-fs (vda1): warning: checktime reached, running e2fsck is recommended
Starting logging: OK
Initializing random number generator... done.
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending select for 20.0.0.3...
Lease of 20.0.0.3 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 20.0.0.2


Thanks
Gary
Comment 21 Ofer Blaut 2013-02-18 04:45:57 EST
I still have this issue, after installting new machine

please advise

below logs


*** /var/log/quantum/dhcp log 

Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=3991fec8-d178-47d3-8f54-72def3d889f8', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tap4e694af8-36', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=tag0,55.55.1.0,static,120s', '--dhcp-range=tag1,55.53.1.0,static,120s', '--dhcp-range=tag2,55.54.1.0,static,120s', '--dhcp-range=tag3,75.54.1.0,static,120s', '--dhcp-range=tag4,71.54.1.0,static,120s', '--dhcp-range=tag5,72.54.1.0,static,120s', '--dhcp-range=tag6,78.54.1.0,static,120s', '--conf-file=']
Exit code: 3
Stdout: ''
Stderr: '\ndnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory\n'
2013-02-18 10:22:15     INFO [quantum.agent.dhcp_agent] Synchronizing state
2013-02-18 10:22:16    ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 91, in call_driver
    getattr(driver, action)()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 113, in enable
    self.spawn_process()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/dhcp.py", line 261, in spawn_process
    utils.execute(cmd, self.root_helper)
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 63, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=3991fec8-d178-47d3-8f54-72def3d889f8', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tap4e694af8-36', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/3991fec8-d178-47d3-8f54-72def3d889f8/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=tag0,55.55.1.0,static,120s', '--dhcp-range=tag1,55.53.1.0,static,120s', '--dhcp-range=tag2,55.54.1.0,static,120s', '--dhcp-range=tag3,75.54.1.0,static,120s', '--dhcp-range=tag4,71.54.1.0,static,120s', '--dhcp-range=tag5,72.54.1.0,static,120s', '--dhcp-range=tag6,78.54.1.0,static,120s', '--conf-file=']
Exit code: 3
Stdout: ''
Stderr: '\ndnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory\n'

********** SELINUX module 

[root@puma34 ~(keystone_admin)]$ semodule -v -l | grep quan
openstack-selinux-quantum	0.2	
quantum	1.0.0	



******  /var/log/messages


Feb 17 16:00:49 puma34 kernel: device tap4e694af8-36 entered promiscuous mode
Feb 17 16:00:49 puma34 kernel: type=1400 audit(1361109649.566:1855): avc:  denied  { read write } for  pid=15291 comm="ip" path="anon_inode:[eventpoll]" dev=anon_inodefs ino=3792 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:anon_inodefs_t:s0 tclass=file
Feb 17 16:00:49 puma34 kernel: type=1400 audit(1361109649.692:1856): avc:  denied  { read write } for  pid=15297 comm="ip" path="anon_inode:[eventpoll]" dev=anon_inodefs ino=3792 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:anon_inodefs_t:s0 tclass=file
Feb 17 16:00:49 puma34 kernel: type=1400 audit(1361109649.711:1857): avc:  denied  { read write } for  pid=15298 comm="ip" path="anon_inode:[eventpoll]" dev=anon_inodefs ino=3792 scontext=system_u:system_r:ifconfig_t:s0 tcontext=system_u:object_r:anon_inodefs_t:s0 tclass=file
Feb 17 16:00:49 puma34 kernel: type=1400 audit(1361109649.844:1858): avc:  denied  { execute } for  pid=15305 comm="dnsmasq" name="bash" dev=dm-0 ino=9568301 scontext=system_u:system_r:dnsmasq_t:s0 tcontext=system_u:object_r:shell_exec_t:s0 tclass=file
Feb 17 16:00:49 puma34 dnsmasq[15304]: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory
Feb 17 16:00:49 puma34 dnsmasq[15304]: FAILED to start up


openstack-quantum-2012.2.3-3.el6ost.noarch
openstack-selinux-0.1.2-3.el6ost.noarch
Comment 23 Daniel Walsh 2013-02-18 10:40:23 EST
corecmd_exec_shell(dnsmasq_t)  Should be added.

The anon_inode is a leaked file descriptor from the tool that is executing ip.


fd = inotify_init1(IN_CLOEXEC) should fix it.

Or

fd = inotify_init()
fcntl(fd, F_SETFD, FD_CLOEXEC)
Comment 24 Bob Kukura 2013-02-19 11:56:27 EST
The anon_inode FD leak is now being addressed in bug 878846.

I believe Lon is dealing with dnsmasq AVS under this bug via the openstack-selinux policy package, so I'm re-assigning this one to him.
Comment 25 Lon Hohberger 2013-02-19 16:50:18 EST
Created attachment 699689 [details]
add corecmd_exec_shell(dnsmasq_t)
Comment 26 Lon Hohberger 2013-02-19 16:53:55 EST
Created attachment 699695 [details]
Current full openstack-selinux-quantum policy file incl. patch.
Comment 31 Ofer Blaut 2013-02-26 14:20:22 EST
This specific AVS is not seen anymore

'\ndnsmasq: cannot run lease-init script /usr/bin/quantum-dhcp-agent-dnsmasq-lease-update: No such file or directory\n'


openstack-selinux-0.1.2-5.el6ost
Comment 33 errata-xmlrpc 2013-03-21 15:04:11 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0672.html

Note You need to log in before you can comment on or make changes to this bug.