Bug 1163754

Summary: Stricter permissions for /var/lib/neutron break DHCP/dnsmasq - instances not getting IP assigned
Product: [Fedora] Fedora Reporter: Fredy Neeser <nfd>
Component: openstack-neutronAssignee: Ihar Hrachyshka <ihrachys>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: alvin, apevec, apevec, ihrachys, jlibosva, majopela, pasik, p, rk, twilson
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-2014.1.3-5.fc21 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1163759 (view as bug list) Environment:
Last Closed: 2014-12-19 18:30:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1163759    

Description Fredy Neeser 2014-11-13 12:29:53 UTC
Description of problem:
Immediately after updating to openstack-neutron 2014.2-4 (rawhide version, is being pushed on Fedora 20 with packstack & RedHat RDO installed), my working 2-node OpenStack setup stopped working - the instances failed to acquire IP addresses.

After checking /var/log/neutron/dnsmasq.log (see below), it's clear that the  permissions change in openstack-neutron 2014.2-4
 
    - Made /var/log/neutron and /var/lib/neutron permissions more strict
    (0755 -> 0750) since those directories may contain sensitive data,
    rhbz#1149688

is the culprit.

Version-Release number of selected component (if applicable):
2014.2-4

How reproducible:
Always

Steps to Reproduce:
1. Update a working OpenStack setup to openstack-neutron 2014.2-4
   In my case, SELinux is in permissive mode.
2. Start or restart a compute instance
3. Check log through dashboard --> dhcp discovery fails
   "Permission denied" can be observed in /var/log/neutron/dnsmasq.log

Actual results:
DHCP discovery fails because dnsmasq fails to access /var/lib/neutron

Expected results:
DHCP should assign IP address.

Additional info:

Problem can be resolved by reverting the permissions changes (I reverted both to become functional again) through

  # chmod o+rx /var/lib/neutron
  # chmod o+rx /var/log/neutron


Snippets from /var/log/neutron/dnsmasq.log:

Before update
-------------
Nov 12 08:59:45 dnsmasq[3155]: started, version 2.68 cachesize 150
Nov 12 08:59:45 dnsmasq[3155]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 08:59:45 dnsmasq[3155]: warning: no upstream servers configured
Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 08:59:45 dnsmasq[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses
Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host
Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts
Nov 12 09:10:43 dnsmasq[3155]: exiting on receipt of SIGTERM
Nov 12 09:30:07 dnsmasq[3871]: started, version 2.68 cachesize 150
Nov 12 09:30:07 dnsmasq[3871]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 09:30:07 dnsmasq[3871]: warning: no upstream servers configured
Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 09:30:07 dnsmasq[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses
Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host
Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts

Problems begin immediately after udpate:
----------------------------------------
Nov 12 10:24:31 dnsmasq[3871]: exiting on receipt of SIGTERM
Nov 12 10:24:32 dnsmasq[13750]: started, version 2.68 cachesize 150
Nov 12 10:24:32 dnsmasq[13750]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 10:24:32 dnsmasq[13750]: warning: no upstream servers configured
Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 10:24:32 dnsmasq[13750]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied
Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied
Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied
Nov 12 10:32:52 dnsmasq[13750]: exiting on receipt of SIGTERM
Nov 12 10:36:13 dnsmasq[4018]: started, version 2.68 cachesize 150
Nov 12 10:36:13 dnsmasq[4018]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth
Nov 12 10:36:13 dnsmasq[4018]: warning: no upstream servers configured
Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, static leases only on 10.0.0.0, lease time 1d
Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, sockets bound exclusively to interface tap8365c135-a6
Nov 12 10:36:13 dnsmasq[4018]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied
Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied
Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied

Comment 1 Ihar Hrachyshka 2014-11-13 13:30:25 UTC
Thanks a lot for the bug report!

Indeed, dnsmasq seems to drop current permissions and falls back to nobody once started. So we'll need to stick to previous 755 permissions for /var/lib/neutron. As for /var/log/neutron, we should be ok with leaving 750 because dnsmasq does not log there (I guess you've specified a custom dnsmasq file to get the log file created there; if that's the case, you'll need to update it to point to another location where nobody user has permissions).

Comment 2 Fredy Neeser 2014-11-13 14:04:45 UTC
(In reply to Ihar Hrachyshka from comment #1)
...

> As for /var/log/neutron, we should be ok with leaving 750
> because dnsmasq does not log there (I guess you've specified a custom
> dnsmasq file to get the log file created there; if that's the case, you'll
> need to update it to point to another location where nobody user has
> permissions).

Correct, I told dnsmasq to write logs to a custom location
  /var/log/neutron/dnsmasq.log

Sure, I could direct the dnsmasq log to a different location but there are many references on the web suggesting precisely this location ...

                              ***

This is how I did it:

/etc/neutron/dhcp_agent.ini: Modified lines for dnsmasq troubleshooting:

# To assist with troubleshooting, enable verbose logging
verbose = True

dnsmasq_config_file = /etc/neutron/dnsmasq.conf


/etc/neutron/dnsmasq.conf: New file for dnsmasq troubleshooting and
setting MTU (initial guess for VXLAN)

# cat /etc/neutron/dnsmasq.conf 
# Log dnsmasq output to a file, instead of journalctl
log-facility = /var/log/neutron/dnsmasq.log
log-dhcp

# Reduce guest MTU to avoid the need for path MTU discovery (PMTUD)
# with a VXLAN overlay (initial guess for a reasonable MTU)
dhcp-option=26,1454

Comment 3 Ihar Hrachyshka 2014-11-13 14:22:06 UTC
As a workaround, you may try to pass user=neutron in the dnsmasq conf file. I guess it should work.

The ideal fix is to make neutron to run dnsmasq with a user specified via config option. I'll follow up on this solution in upstream.

In the meantime, I'll revert permissions for /var/lib/neutron since it's an obvious breakage in default neutron setup.

Comment 4 Ihar Hrachyshka 2014-11-13 14:32:00 UTC
Once openstack-neutron-2014.2-9.fc22 reaches RDO repos, please verify the bug and report back (note: it may take some time). Thanks.

Comment 5 Alan Pevec 2014-11-14 18:30:38 UTC
In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it).

Comment 7 Alan Pevec 2014-11-15 21:29:25 UTC
> In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it).

Full downgrade command is:

# yum downgrade openstack-neutron\* python-neutron

Comment 8 Fedora Update System 2014-11-21 10:36:01 UTC
openstack-neutron-2014.1.3-4.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-4.fc21

Comment 9 Alan Pevec 2014-11-21 10:37:02 UTC
RDO update:
Neutron builds listed in comment 6 have been published to the RDO Icehouse and Juno repositories.

Comment 10 Fedora Update System 2014-11-22 20:20:59 UTC
Package openstack-neutron-2014.1.3-4.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing openstack-neutron-2014.1.3-4.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-15594/openstack-neutron-2014.1.3-4.fc21
then log in and leave karma (feedback).

Comment 11 Fedora Update System 2014-12-05 12:37:55 UTC
openstack-neutron-2014.1.3-5.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-5.fc21

Comment 12 Fedora Update System 2014-12-19 18:30:02 UTC
openstack-neutron-2014.1.3-5.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.