Bug 1163754 - Stricter permissions for /var/lib/neutron break DHCP/dnsmasq - instances not getting IP assigned
Summary: Stricter permissions for /var/lib/neutron break DHCP/dnsmasq - instances not ...
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openstack-neutron
Version: rawhide
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Ihar Hrachyshka
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Keywords:
Depends On:
Blocks: 1163759
TreeView+ depends on / blocked
 
Reported: 2014-11-13 12:29 UTC by Fredy Neeser
Modified: 2014-12-19 18:30 UTC (History)
10 users (show)

(edit)
Clone Of:
: 1163759 (view as bug list)
(edit)
Last Closed: 2014-12-19 18:30:02 UTC


Attachments (Terms of Use)

Description Fredy Neeser 2014-11-13 12:29:53 UTC
Description of problem:
Immediately after updating to openstack-neutron 2014.2-4 (rawhide version, is being pushed on Fedora 20 with packstack & RedHat RDO installed), my working 2-node OpenStack setup stopped working - the instances failed to acquire IP addresses.

After checking /var/log/neutron/dnsmasq.log (see below), it's clear that the  permissions change in openstack-neutron 2014.2-4
 
    - Made /var/log/neutron and /var/lib/neutron permissions more strict
    (0755 -> 0750) since those directories may contain sensitive data,
    rhbz#1149688

is the culprit.

Version-Release number of selected component (if applicable):
2014.2-4

How reproducible:
Always

Steps to Reproduce:
1. Update a working OpenStack setup to openstack-neutron 2014.2-4
   In my case, SELinux is in permissive mode.
2. Start or restart a compute instance
3. Check log through dashboard --> dhcp discovery fails
   "Permission denied" can be observed in /var/log/neutron/dnsmasq.log

Actual results:
DHCP discovery fails because dnsmasq fails to access /var/lib/neutron

Expected results:
DHCP should assign IP address.

Additional info:

Problem can be resolved by reverting the permissions changes (I reverted both to become functional again) through

  # chmod o+rx /var/lib/neutron
  # chmod o+rx /var/log/neutron


Snippets from /var/log/neutron/dnsmasq.log:

Before update
-------------
Nov 12 08:59:45 dnsmasq[3155]: started, version 2.68 cachesize 150
Nov 12 08:59:45 dnsmasq[3155]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 08:59:45 dnsmasq[3155]: warning: no upstream servers configured
Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 08:59:45 dnsmasq[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses
Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host
Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts
Nov 12 09:10:43 dnsmasq[3155]: exiting on receipt of SIGTERM
Nov 12 09:30:07 dnsmasq[3871]: started, version 2.68 cachesize 150
Nov 12 09:30:07 dnsmasq[3871]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 09:30:07 dnsmasq[3871]: warning: no upstream servers configured
Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 09:30:07 dnsmasq[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses
Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host
Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts

Problems begin immediately after udpate:
----------------------------------------
Nov 12 10:24:31 dnsmasq[3871]: exiting on receipt of SIGTERM
Nov 12 10:24:32 dnsmasq[13750]: started, version 2.68 cachesize 150
Nov 12 10:24:32 dnsmasq[13750]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 10:24:32 dnsmasq[13750]: warning: no upstream servers configured
Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 10:24:32 dnsmasq[13750]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied
Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied
Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied
Nov 12 10:32:52 dnsmasq[13750]: exiting on receipt of SIGTERM
Nov 12 10:36:13 dnsmasq[4018]: started, version 2.68 cachesize 150
Nov 12 10:36:13 dnsmasq[4018]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 10:36:13 dnsmasq[4018]: warning: no upstream servers configured
Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 10:36:13 dnsmasq[4018]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied
Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied
Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied

Comment 1 Ihar Hrachyshka 2014-11-13 13:30:25 UTC
Thanks a lot for the bug report!

Indeed, dnsmasq seems to drop current permissions and falls back to nobody once started. So we'll need to stick to previous 755 permissions for /var/lib/neutron. As for /var/log/neutron, we should be ok with leaving 750 because dnsmasq does not log there (I guess you've specified a custom dnsmasq file to get the log file created there; if that's the case, you'll need to update it to point to another location where nobody user has permissions).

Comment 2 Fredy Neeser 2014-11-13 14:04:45 UTC
(In reply to Ihar Hrachyshka from comment #1)
...

> As for /var/log/neutron, we should be ok with leaving 750
> because dnsmasq does not log there (I guess you've specified a custom
> dnsmasq file to get the log file created there; if that's the case, you'll
> need to update it to point to another location where nobody user has
> permissions).

Correct, I told dnsmasq to write logs to a custom location
  /var/log/neutron/dnsmasq.log

Sure, I could direct the dnsmasq log to a different location but there are many references on the web suggesting precisely this location ...

                              ***

This is how I did it:

/etc/neutron/dhcp_agent.ini: Modified lines for dnsmasq troubleshooting:

# To assist with troubleshooting, enable verbose logging
verbose = True

dnsmasq_config_file = /etc/neutron/dnsmasq.conf


/etc/neutron/dnsmasq.conf: New file for dnsmasq troubleshooting and
setting MTU (initial guess for VXLAN)

# cat /etc/neutron/dnsmasq.conf 
# Log dnsmasq output to a file, instead of journalctl
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp

# Reduce guest MTU to avoid the need for path MTU discovery (PMTUD)
# with a VXLAN overlay (initial guess for a reasonable MTU)
dhcp-option=26,1454

Comment 3 Ihar Hrachyshka 2014-11-13 14:22:06 UTC
As a workaround, you may try to pass user=neutron in the dnsmasq conf file. I guess it should work.

The ideal fix is to make neutron to run dnsmasq with a user specified via config option. I'll follow up on this solution in upstream.

In the meantime, I'll revert permissions for /var/lib/neutron since it's an obvious breakage in default neutron setup.

Comment 4 Ihar Hrachyshka 2014-11-13 14:32:00 UTC
Once openstack-neutron-2014.2-9.fc22 reaches RDO repos, please verify the bug and report back (note: it may take some time). Thanks.

Comment 5 Alan Pevec 2014-11-14 18:30:38 UTC
In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it).

Comment 7 Alan Pevec 2014-11-15 21:29:25 UTC
> In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it).

Full downgrade command is:

# yum downgrade openstack-neutron\* python-neutron

Comment 8 Fedora Update System 2014-11-21 10:36:01 UTC
openstack-neutron-2014.1.3-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-4.fc21

Comment 9 Alan Pevec 2014-11-21 10:37:02 UTC
RDO update:
Neutron builds listed in comment 6 have been published to the RDO Icehouse and Juno repositories.

Comment 10 Fedora Update System 2014-11-22 20:20:59 UTC
Package openstack-neutron-2014.1.3-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openstack-neutron-2014.1.3-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-15594/openstack-neutron-2014.1.3-4.fc21
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-12-05 12:37:55 UTC
openstack-neutron-2014.1.3-5.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-5.fc21

Comment 12 Fedora Update System 2014-12-19 18:30:02 UTC
openstack-neutron-2014.1.3-5.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.