1119619 – neutron-dhcp-agent and neutron-lbaas-agent fail to start

Bug 1119619 - neutron-dhcp-agent and neutron-lbaas-agent fail to start

Summary: neutron-dhcp-agent and neutron-lbaas-agent fail to start

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	5.0 (RHEL 7)
Hardware:	Unspecified
OS:	Unspecified
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	5.0 (RHEL 7)
Assignee:	Miguel Angel Ajo
QA Contact:	yfried
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1119762 (view as bug list)
Depends On:
Blocks:	1120146
TreeView+	depends on / blocked

Reported:	2014-07-15 07:14 UTC by yfried
Modified:	2016-04-26 15:41 UTC (History)
CC List:	8 users (show)
Fixed In Version:	openstack-neutron-2014.1.1-4.el6ost openstack-neutron-2014.1.1-4.el7ost
Doc Type:	Bug Fix
Doc Text:	Prior to this update, LBaaS and DHCP agents relied on Python code specific to a certain Open vSwitch plug-in location. This code was moved into a separate 'openstack-neutron-openvswitch' package. Consequently, LBaaS and DHCP agents failed on startup if the plug-in package was not installed. With this update, all Python code from plug-in-specific packages have been pulled back to 'python-neutron' in order to resolve Python import dependencies. As a result, LBaaS and DHCP agents startup successfully when no 'openstack-neutron-openvswitch' package is installed.
Clone Of:
Clones:	1120146 (view as bug list)
Environment:
Last Closed:	2014-07-24 17:24:21 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2014:0936	0	normal	SHIPPED_LIVE	openstack-neutron bug-fix advisory	2014-07-24 21:22:04 UTC

Description yfried 2014-07-15 07:14:10 UTC

Description of problem:
neutron-dhcp-agent fails to start

Tested on RHEL7
Setup: Controller+Networker+2computes

Steps to Reproduce:
1. Install openstack
2. On network node - openstack-status

Actual results:
== neutron services ==
neutron-server:                         inactive  (disabled on boot)
neutron-dhcp-agent:                     failed
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    failed
neutron-openvswitch-agent:              active
neutron-linuxbridge-agent:              inactive  (disabled on boot)
neutron-ryu-agent:                      inactive  (disabled on boot)
neutron-nec-agent:                      inactive  (disabled on boot)
neutron-mlnx-agent:                     inactive  (disabled on boot)


Expected results:
failed agents should be active

Additional info:
tested with selinux enforcing and permissive


[root@puma46 ~]# systemctl status neutron-dhcp-agent.service 
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 08:58:26 IDT; 43min ago
 Main PID: 12211 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/neutron-dhcp-agent.service

Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.dhcp_agent import main
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp_agent.py", line 27, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import interface
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 26, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import ovs_lib
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 30, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.plugins.openvswitch.common import constants
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: ImportError: No module named openvswitch.common
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-dhcp-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-dhcp-agent.service entered failed state.


[root@puma46 ~]# systemctl status neutron-dhcp-agent.service 
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 08:58:26 IDT; 49min ago
 Main PID: 12211 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/neutron-dhcp-agent.service

Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.dhcp_agent import main
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp_agent.py", line 27, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import interface
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 26, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import ovs_lib
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 30, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.plugins.openvswitch.common import constants
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: ImportError: No module named openvswitch.common
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-dhcp-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-dhcp-agent.service entered failed state.


[root@puma46 ~]# sestatus 
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28
[root@puma46 ~]# /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --log-file /var/log/neutron/dhcp-agent.log
[root@puma46 ~]systemctl status neutron-dhcp-agent.service 
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 08:58:26 IDT; 53min ago
 Main PID: 12211 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/neutron-dhcp-agent.service

Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.dhcp_agent import main
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp_agent.py", line 27, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import interface
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 26, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.agent.linux import ovs_lib
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 30, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: from neutron.plugins.openvswitch.common import constants
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[12211]: ImportError: No module named openvswitch.common
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-dhcp-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-dhcp-agent.service entered failed state.


[root@puma46 ~]# systemctl start neutron-dhcp-agent.service 
[root@puma46 ~]# systemctl status neutron-dhcp-agent.service 
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 09:51:44 IDT; 1s ago
  Process: 50346 ExecStart=/usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --log-file /var/log/neutron/dhcp-agent.log (code=exited, status=1/FAILURE)
 Main PID: 50346 (code=exited, status=1/FAILURE)

Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: filelog = logging.handlers.WatchedFileHandler(logpath)
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: logging.FileHandler.__init__(self, filename, mode, encoding, delay)
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: StreamHandler.__init__(self, self._open())
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: stream = open(self.baseFilename, self.mode)
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50346]: IOError: [Errno 13] Permission denied: '/var/log/neutron/dhcp-agent.log'
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-dhcp-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 09:51:44 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-dhcp-agent.service entered failed state.







[root@puma46 ~]# setenforce 0
[root@puma46 ~]# systemctl stop neutron-dhcp-agent.service 
[root@puma46 ~]# systemctl status neutron-dhcp-agent.service 
neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 09:52:11 IDT; 17s ago
  Process: 50400 ExecStart=/usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/dhcp_agent.ini --log-file /var/log/neutron/dhcp-agent.log (code=exited, status=1/FAILURE)
 Main PID: 50400 (code=exited, status=1/FAILURE)

Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: _setup_logging_from_conf()
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: File "/usr/lib/python2.7/site-packages/neutron/openstack/common/log.py", line 473, in _setup_logging_from_conf
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: filelog = logging.handlers.WatchedFileHandler(logpath)
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: logging.FileHandler.__init__(self, filename, mode, encoding, delay)
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: StreamHandler.__init__(self, self._open())
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: stream = open(self.baseFilename, self.mode)
Jul 15 09:52:10 puma46.scl.lab.tlv.redhat.com neutron-dhcp-agent[50400]: IOError: [Errno 13] Permission denied: '/var/log/neutron/dhcp-agent.log'
Jul 15 09:52:11 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-dhcp-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 09:52:11 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-dhcp-agent.service entered failed state.
Jul 15 09:52:25 puma46.scl.lab.tlv.redhat.com systemd[1]: Stopped OpenStack Neutron DHCP Agent.



[root@puma46 ~]# systemctl status neutron-lbaas-agent.service 
neutron-lbaas-agent.service - OpenStack Neutron Load Balancing as a Service Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-lbaas-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 08:58:26 IDT; 1h 11min ago
 Main PID: 12237 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/neutron-lbaas-agent.service

Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.services.loadbalancer.agent.agent import main
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent.py", line 23, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.agent.linux import interface
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 26, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.agent.linux import ovs_lib
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 30, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.plugins.openvswitch.common import constants
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: ImportError: No module named openvswitch.common
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-lbaas-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-lbaas-agent.service entered failed state.


LBAAS:
[root@puma46 ~]# systemctl status neutron-lbaas-agent.service 
neutron-lbaas-agent.service - OpenStack Neutron Load Balancing as a Service Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-lbaas-agent.service; enabled)
   Active: failed (Result: exit-code) since Tue 2014-07-15 08:58:26 IDT; 1h 11min ago
 Main PID: 12237 (code=exited, status=1/FAILURE)
   CGroup: /system.slice/neutron-lbaas-agent.service

Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.services.loadbalancer.agent.agent import main
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/services/loadbalancer/agent/agent.py", line 23, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.agent.linux import interface
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 26, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.agent.linux import ovs_lib
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ovs_lib.py", line 30, in <module>
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: from neutron.plugins.openvswitch.common import constants
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com neutron-lbaas-agent[12237]: ImportError: No module named openvswitch.common
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: neutron-lbaas-agent.service: main process exited, code=exited, status=1/FAILURE
Jul 15 08:58:26 puma46.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-lbaas-agent.service entered failed state.


[root@puma46 ~]# setenforce 1

[root@puma46 ~]# systemctl restart neutron-lbaas-agent.service 
[root@puma46 ~]# systemctl status neutron-lbaas-agent.service 
neutron-lbaas-agent.service - OpenStack Neutron Load Balancing as a Service Agent
   Loaded: loaded (/usr/lib/systemd/system/neutron-lbaas-agent.service; enabled)
   Active: active (running) since Tue 2014-07-15 10:11:42 IDT; 5s ago
 Main PID: 52386 (neutron-lbaas-a)
   CGroup: /system.slice/neutron-lbaas-agent.service
           └─52386 /usr/bin/python /usr/bin/neutron-lbaas-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/lbaas_agent.ini --log-file /var/log/neutron/lbaas-age...

Jul 15 10:11:42 puma46.scl.lab.tlv.redhat.com systemd[1]: Started OpenStack Neutron Load Balancing as a Service Agent.

Comment 1 yfried 2014-07-15 07:44:48 UTC

[root@puma46 ~]# rpm -qa | grep neutron
python-neutron-2014.1.1-2.el7ost.noarch

Comment 2 Miguel Angel Ajo 2014-07-15 13:19:29 UTC

*** Bug 1119762 has been marked as a duplicate of this bug. ***

Comment 3 Ihar Hrachyshka 2014-07-15 13:21:05 UTC

The reason is that those agents use a file from ovs plugin.

In an ideal world (aka in Juno), we may merge plugins back to python-neutron package. Till that time (aka in Icehouse/[Havana?]), we need to apply some other approach, like moving that missing file with constants that is missing to python-neutron, leaving other plugin files in its package.

Comment 4 Miguel Angel Ajo 2014-07-15 13:25:46 UTC

It looks like a reincarnation of RDO bug 1019487.

The origin of this is that the sources in openstack-neutron
are split among various packages, when that's naturally distributed
in a single tar. And then a missing dependency from the package
containing neutron-dhcp-agent (openstack-neutron) to the 
neutron-openvswitch-agent package.
+
The non-predictable order of execution in the puppet script
modules unless we set some require constraints specifically 
to address that openvswitch-agent gets installed before
the neutron-*-agent service is started.

The RDO bug was fixed here (with collateral damage)
http://paste.fedoraproject.org/118063/05421694

And thats why it got reverted here:
http://paste.fedoraproject.org/118065/42185914


Looking for a fix at the packaging side...

Comment 7 yfried 2014-07-21 05:36:41 UTC

On RHEL7

openstack-neutron-2014.1.1-4.el7ost.noarch

[root@puma46 ~]# openstack-status
== neutron services ==
neutron-server:                         inactive  (disabled on boot)
neutron-dhcp-agent:                     active
neutron-l3-agent:                       active
neutron-metadata-agent:                 active
neutron-lbaas-agent:                    active
neutron-openvswitch-agent:              active

Comment 9 errata-xmlrpc 2014-07-24 17:24:21 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0936.html

Note You need to log in before you can comment on or make changes to this bug.