Bug 1325929

Summary: overcloud deployment fails with "Error: Could not start Service[nova-compute]: Execution of '/usr/bin/systemctl start openstack-nova-compute' returned 1" during ComputeNodesPostDeployment
Product: [Community] RDO Reporter: Attila Darazs <adarazs>
Component: rdo-managerAssignee: Hugh Brock <hbrock>
Status: CLOSED EOL QA Contact: Shai Revivo <srevivo>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: LibertyCC: gcerami, mburns
Target Milestone: ---Keywords: Automation
Target Release: Kilo   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-19 15:32:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Attila Darazs 2016-04-11 12:38:51 UTC 
Description of problem:
In the "current-passed-ci" CentOS CI liberty delorean job, this error appeared on the 7th of April:

1:52:23.667 \u001b[1;31mError: Could not start Service[nova-compute]: Execution of '/usr/bin/systemctl start openstack-nova-compute' returned 1: Job for openstack-nova-compute.service failed because the control process exited with error code. See \"systemctl status openstack-nova-compute.service\" and \"journalctl -xe\" for details.
01:52:23.667 Wrapped exception:
01:52:23.667 Execution of '/usr/bin/systemctl start openstack-nova-compute' returned 1: Job for openstack-nova-compute.service failed because the control process exited with error code. See \"systemctl status openstack-nova-compute.service\" and \"journalctl -xe\" for details.\u001b[0m

The relevant error in "journalctl -xe" seems to be:

Apr 11 11:55:12 overcloud-novacompute-0 systemd[1]: Starting OpenStack Nova Compute Server...
-- Subject: Unit openstack-nova-compute.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-compute.service has begun starting up.
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: No handlers could be found for logger "oslo_config.cfg"
Apr 11 11:55:13 overcloud-novacompute-0 ntpd[9082]: Listen normally on 6 br-ex fe80::aca5:70ff:fe52:764b UDP 123
Apr 11 11:55:13 overcloud-novacompute-0 ntpd[9082]: new interface(s) found: waking up resolver
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: Traceback (most recent call last):
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: sys.exit(main())
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/plugins/ovs_neutron_agent.py", line 20, in main
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: agent_main.main()
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/main.py", line 42, in main
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: common_config.init(sys.argv[1:])
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 207, in init
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: n_rpc.init(cfg.CONF)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 62, in init
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: aliases=TRANSPORT_ALIASES)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 186, in get_transport
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: invoke_kwds=kwargs)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 45, in __init__
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: verify_requirements=verify_requirements,
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: verify_requirements)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: self._on_load_failure_callback(self, ep, err)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 162, in _load_plugins
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: verify_requirements,
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: verify_requirements,
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 185, in _load_one_plugin
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: plugin = ep.load(require=verify_requirements)
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: entry = __import__(self.module_name, globals(),globals(), ['__name__'])
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 101, in <module>
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: cfg.PortOpt('rabbit_port',
Apr 11 11:55:13 overcloud-novacompute-0 neutron-openvswitch-agent[9287]: AttributeError: 'module' object has no attribute 'PortOpt'
Apr 11 11:55:14 overcloud-novacompute-0 systemd[1]: neutron-openvswitch-agent.service: main process exited, code=exited, status=1/FAILURE
Apr 11 11:55:14 overcloud-novacompute-0 systemd[1]: Unit neutron-openvswitch-agent.service entered failed state.
Apr 11 11:55:14 overcloud-novacompute-0 systemd[1]: neutron-openvswitch-agent.service failed.
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: No handlers could be found for logger "oslo_config.cfg"
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: Traceback (most recent call last):
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/bin/nova-compute", line 10, in <module>
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: sys.exit(main())
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/nova/cmd/compute.py", line 58, in main
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: config.parse_args(sys.argv)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/nova/config.py", line 60, in parse_args
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: rpc.init(CONF)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/nova/rpc.py", line 63, in init
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: aliases=TRANSPORT_ALIASES)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 186, in get_transport
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: invoke_kwds=kwargs)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 45, in __init__
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: verify_requirements=verify_requirements,
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 55, in __init__
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: verify_requirements)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 170, in _load_plugins
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: self._on_load_failure_callback(self, ep, err)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 162, in _load_plugins
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: verify_requirements,
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: verify_requirements,
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 185, in _load_one_plugin
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: plugin = ep.load(require=verify_requirements)
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 2260, in load
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: entry = __import__(self.module_name, globals(),globals(), ['__name__'])
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 101, in <module>
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: cfg.PortOpt('rabbit_port',
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: AttributeError: 'module' object has no attribute 'PortOpt'
Apr 11 11:55:14 overcloud-novacompute-0 systemd[1]: openstack-nova-compute.service: main process exited, code=exited, status=1/FAILURE
Apr 11 11:55:14 overcloud-novacompute-0 systemd[1]: Failed to start OpenStack Nova Compute Server.
-- Subject: Unit openstack-nova-compute.service has failed


Version-Release number of selected component (if applicable):
http://trunk.rdoproject.org/centos7-liberty/50/00/500098d8ebc9c5cad5dbfb89fe47c32a10faded1_0376d60d

How reproducible:
100%

Steps to Reproduce:
1. deploy command used:

openstack overcloud deploy --debug --log-file overcloud_deployment_30.log --templates /home/stack/local_tht/ --libvirt-type=qemu --ntp-server 212.199.182.150 --control-scale 1 --compute-scale 1 --ceph-storage-scale 1 --block-storage-scale 0 --swift-storage-scale 0 --control-flavor baremetal --compute-flavor baremetal --ceph-storage-flavor baremetal --block-storage-flavor baremetal --swift-storage-flavor baremetal --neutron-network-type vxlan --neutron-tunnel-types vxlan --timeout=90 -e /home/stack/local_tht/environments/storage-environment.yaml -e /home/stack/local_tht/environments/puppet-pacemaker.yaml

Additional info:
Here's a CI job that can reproduce it:

https://ci.centos.org/view/rdo/job/rdo_manager-periodic-7-rdo-liberty-delorean-centos-7.0-templates-virthost-minimal-neutron-ml2-vxlan-smoke/485/consoleFull

os-collect-config from the novacompute: https://ci.centos.org/artifacts/rdo/jenkins-rdo_manager-periodic-7-rdo-liberty-delorean-centos-7.0-templates-virthost-minimal-neutron-ml2-vxlan-smoke-483/overcloud-novacompute-0/var/log/os-collect-config.log.gz
Comment 1 Gabriele Cerami 2016-04-11 18:08:44 UTC 
Apr 11 11:55:14 overcloud-novacompute-0 nova-compute[9320]: AttributeError: 'module' object has no attribute 'PortOpt'

the use of PortOpt config option from oslo_config in neutron has been implemented only in the mitaka cycle. stable/liberty branch shouln't contain those
Comment 3 Chandan Kumar 2016-05-19 15:32:57 UTC 
This bug is against a Version which has reached End of Life.
If it's still present in supported release (http://releases.openstack.org), please update Version and reopen.