Bug 1163754
| Summary: | Stricter permissions for /var/lib/neutron break DHCP/dnsmasq - instances not getting IP assigned | |||
|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Fredy Neeser <nfd> | |
| Component: | openstack-neutron | Assignee: | Ihar Hrachyshka <ihrachys> | |
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | |
| Severity: | high | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | rawhide | CC: | alvin, apevec, apevec, ihrachys, jlibosva, majopela, pasik, p, rk, twilson | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-neutron-2014.1.3-5.fc21 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1163759 (view as bug list) | Environment: | ||
| Last Closed: | 2014-12-19 18:30:02 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1163759 | |||
Thanks a lot for the bug report! Indeed, dnsmasq seems to drop current permissions and falls back to nobody once started. So we'll need to stick to previous 755 permissions for /var/lib/neutron. As for /var/log/neutron, we should be ok with leaving 750 because dnsmasq does not log there (I guess you've specified a custom dnsmasq file to get the log file created there; if that's the case, you'll need to update it to point to another location where nobody user has permissions). (In reply to Ihar Hrachyshka from comment #1) ... > As for /var/log/neutron, we should be ok with leaving 750 > because dnsmasq does not log there (I guess you've specified a custom > dnsmasq file to get the log file created there; if that's the case, you'll > need to update it to point to another location where nobody user has > permissions). Correct, I told dnsmasq to write logs to a custom location /var/log/neutron/dnsmasq.log Sure, I could direct the dnsmasq log to a different location but there are many references on the web suggesting precisely this location ... *** This is how I did it: /etc/neutron/dhcp_agent.ini: Modified lines for dnsmasq troubleshooting: # To assist with troubleshooting, enable verbose logging verbose = True dnsmasq_config_file = /etc/neutron/dnsmasq.conf /etc/neutron/dnsmasq.conf: New file for dnsmasq troubleshooting and setting MTU (initial guess for VXLAN) # cat /etc/neutron/dnsmasq.conf # Log dnsmasq output to a file, instead of journalctl log-facility = /var/log/neutron/dnsmasq.log log-dhcp # Reduce guest MTU to avoid the need for path MTU discovery (PMTUD) # with a VXLAN overlay (initial guess for a reasonable MTU) dhcp-option=26,1454 As a workaround, you may try to pass user=neutron in the dnsmasq conf file. I guess it should work. The ideal fix is to make neutron to run dnsmasq with a user specified via config option. I'll follow up on this solution in upstream. In the meantime, I'll revert permissions for /var/lib/neutron since it's an obvious breakage in default neutron setup. Once openstack-neutron-2014.2-9.fc22 reaches RDO repos, please verify the bug and report back (note: it may take some time). Thanks. In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it). Here are the links for the "Fixed In Version" builds, if anyone ones to test them while they're going through the release process: * openstack-neutron-2014.1.3-4.el6 http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-6/epel-6-x86_64/openstack-neutron-2014.1.3-4.el6/ * openstack-neutron-2014.1.3-4.el7 http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-7/epel-7-x86_64/openstack-neutron-2014.1.3-4.el7/ * openstack-neutron-2014.1.3-4.fc21 https://kojipkgs.fedoraproject.org//packages/openstack-neutron/2014.1.3/4.fc21/noarch/ * openstack-neutron-2014.2-9.fc22 https://kojipkgs.fedoraproject.org//packages/openstack-neutron/2014.2/9.fc22/noarch/ * openstack-neutron-2014.2-9.el7 http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-juno-epel-7/epel-7-x86_64/openstack-neutron-2014.2-9.el7/ > In RDO repos we keep old versions, so yum downgrade openstack-neutron might help until update is pushed through (working on it).
Full downgrade command is:
# yum downgrade openstack-neutron\* python-neutron
openstack-neutron-2014.1.3-4.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-4.fc21 RDO update: Neutron builds listed in comment 6 have been published to the RDO Icehouse and Juno repositories. Package openstack-neutron-2014.1.3-4.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing openstack-neutron-2014.1.3-4.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2014-15594/openstack-neutron-2014.1.3-4.fc21 then log in and leave karma (feedback). openstack-neutron-2014.1.3-5.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/openstack-neutron-2014.1.3-5.fc21 openstack-neutron-2014.1.3-5.fc21 has been pushed to the Fedora 21 stable repository. If problems still persist, please make note of it in this bug report. |
Description of problem: Immediately after updating to openstack-neutron 2014.2-4 (rawhide version, is being pushed on Fedora 20 with packstack & RedHat RDO installed), my working 2-node OpenStack setup stopped working - the instances failed to acquire IP addresses. After checking /var/log/neutron/dnsmasq.log (see below), it's clear that the permissions change in openstack-neutron 2014.2-4 - Made /var/log/neutron and /var/lib/neutron permissions more strict (0755 -> 0750) since those directories may contain sensitive data, rhbz#1149688 is the culprit. Version-Release number of selected component (if applicable): 2014.2-4 How reproducible: Always Steps to Reproduce: 1. Update a working OpenStack setup to openstack-neutron 2014.2-4 In my case, SELinux is in permissive mode. 2. Start or restart a compute instance 3. Check log through dashboard --> dhcp discovery fails "Permission denied" can be observed in /var/log/neutron/dnsmasq.log Actual results: DHCP discovery fails because dnsmasq fails to access /var/lib/neutron Expected results: DHCP should assign IP address. Additional info: Problem can be resolved by reverting the permissions changes (I reverted both to become functional again) through # chmod o+rx /var/lib/neutron # chmod o+rx /var/log/neutron Snippets from /var/log/neutron/dnsmasq.log: Before update ------------- Nov 12 08:59:45 dnsmasq[3155]: started, version 2.68 cachesize 150 Nov 12 08:59:45 dnsmasq[3155]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Nov 12 08:59:45 dnsmasq[3155]: warning: no upstream servers configured Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, static leases only on 10.0.0.0, lease time 1d Nov 12 08:59:45 dnsmasq-dhcp[3155]: DHCP, sockets bound exclusively to interface tap8365c135-a6 Nov 12 08:59:45 dnsmasq[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host Nov 12 08:59:45 dnsmasq-dhcp[3155]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts Nov 12 09:10:43 dnsmasq[3155]: exiting on receipt of SIGTERM Nov 12 09:30:07 dnsmasq[3871]: started, version 2.68 cachesize 150 Nov 12 09:30:07 dnsmasq[3871]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Nov 12 09:30:07 dnsmasq[3871]: warning: no upstream servers configured Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, static leases only on 10.0.0.0, lease time 1d Nov 12 09:30:07 dnsmasq-dhcp[3871]: DHCP, sockets bound exclusively to interface tap8365c135-a6 Nov 12 09:30:07 dnsmasq[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts - 4 addresses Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host Nov 12 09:30:07 dnsmasq-dhcp[3871]: read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts Problems begin immediately after udpate: ---------------------------------------- Nov 12 10:24:31 dnsmasq[3871]: exiting on receipt of SIGTERM Nov 12 10:24:32 dnsmasq[13750]: started, version 2.68 cachesize 150 Nov 12 10:24:32 dnsmasq[13750]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Nov 12 10:24:32 dnsmasq[13750]: warning: no upstream servers configured Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, static leases only on 10.0.0.0, lease time 1d Nov 12 10:24:32 dnsmasq-dhcp[13750]: DHCP, sockets bound exclusively to interface tap8365c135-a6 Nov 12 10:24:32 dnsmasq[13750]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied Nov 12 10:24:32 dnsmasq[13750]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied Nov 12 10:32:52 dnsmasq[13750]: exiting on receipt of SIGTERM Nov 12 10:36:13 dnsmasq[4018]: started, version 2.68 cachesize 150 Nov 12 10:36:13 dnsmasq[4018]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Nov 12 10:36:13 dnsmasq[4018]: warning: no upstream servers configured Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, static leases only on 10.0.0.0, lease time 1d Nov 12 10:36:13 dnsmasq-dhcp[4018]: DHCP, sockets bound exclusively to interface tap8365c135-a6 Nov 12 10:36:13 dnsmasq[4018]: failed to load names from /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/addn_hosts: Permission denied Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/host: Permission denied Nov 12 10:36:13 dnsmasq[4018]: cannot read /var/lib/neutron/dhcp/0b10ce71-152b-47c6-9607-5fb7d35c2d60/opts: Permission denied