Bug 1177680

Summary: rubygem-staypuft: Puppet reports an error: Execution of '/usr/bin/systemctl start neutron-server' returned 1: Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details.
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: openstack-foreman-installerAssignee: Jason Guiditta <jguiditt>
Status: CLOSED CURRENTRELEASE QA Contact: Alexander Chuzhoy <sasha>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: cwolfe, majopela, mburns, morazi, ohochman, rhos-maint, sasha, yeylon
Target Milestone: gaKeywords: TestOnly
Target Release: Installer   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The timeout for starting the neutron server was too short Consequence: In some circumstances, the neutron server would fail to start quickly enough Fix: The timeout was extended. Result:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-10 15:12:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1177026    
Attachments:
Description Flags
foreman logs
none
logs from the controller none

Description Alexander Chuzhoy 2014-12-29 20:17:08 UTC
rubygem-staypuft: Puppet reports an error: Execution of '/usr/bin/systemctl start neutron-server' returned 1: Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details.

Environment:
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.8-1.el7ost.noarch
ruby193-rubygem-staypuft-0.5.9-1.el7ost.noarch
rhel-osp-installer-client-0.5.4-1.el7ost.noarch
openstack-puppet-modules-2014.2.8-1.el7ost.noarch
rhel-osp-installer-0.5.4-1.el7ost.noarch


Steps to reproduce:
1. Configure/run deployment (1 controller + 1 node) where the tenant network has vlan ID configured.


Result:
The puppet reports error:
Could not start Service[neutron-server]: Execution of '/usr/bin/systemctl start neutron-server' returned 1: Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details. Wrapped exception: Execution of '/usr/bin/systemctl start neutron-server' returned 1: Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details.

And the deployment gets paused with error.

I tried to start the neutron-server manually but got the error:
Job for neutron-server.service failed. See 'systemctl status neutron-server.service' and 'journalctl -xn' for details.


Expected result:
The neutron-server should start with no issues.

Comment 1 Alexander Chuzhoy 2014-12-29 20:20:56 UTC
Created attachment 974192 [details]
foreman logs

Comment 2 Alexander Chuzhoy 2014-12-29 20:22:00 UTC
Created attachment 974194 [details]
logs from the controller

Comment 4 Jason Guiditta 2015-01-06 16:20:24 UTC
This is the relevant error from server.log:

2014-12-29 14:53:41.916 14562 TRACE neutron.service ConfigFileValueError: invalid literal for int() with base 10: 'nisim'
2014-12-29 14:53:41.916 14562 TRACE neutron.service 
2014-12-29 14:53:41.931 14562 CRITICAL neutron [req-de4f627a-c647-476d-b2dd-eaf0363a78a2 None] ConfigFileValueError: invalid literal for int() with base 10: 'nisim'

I am not seeing that string in neutron configs, aside from comments of as part of the key 'mechanism_drivers', which is perfectly valid.

Comment 5 Jason Guiditta 2015-01-06 16:22:08 UTC
Oh, sasha, can you provide the version of neutron that was installed on the controller so I can compare with my setup?

Comment 6 Mike Burns 2015-01-08 14:08:43 UTC
(needinfo for comment 5)

Comment 7 Omri Hochman 2015-01-09 15:54:42 UTC
(In reply to Jason Guiditta from comment #5)
> Oh, sasha, can you provide the version of neutron that was installed on the
> controller so I can compare with my setup?


Version of neutron that installed on the Compute:
---------------------------------------------------
python-neutronclient-2.3.9-1.el7ost.noarch
openstack-neutron-2014.2.1-5.el7ost.noarch
python-neutron-2014.2.1-5.el7ost.noarch
openstack-neutron-openvswitch-2014.2.1-5.el7ost.noarch

Comment 8 Lars Kellogg-Stedman 2015-01-09 16:00:18 UTC
Don't know if this helps or not, but looking at the logs, neutron-server is failing with:

  ConfigFileValueError: invalid literal for int() with base 10: 'nisim'

Because of this in /etc/neutron/neutron.conf:

   quota_vip = nisim

Comment 9 Alexander Chuzhoy 2015-01-09 16:05:15 UTC
Looking at the value - I configured this on one setup in attempt to isolate the variable.
The reported issue was reproduced on another setup as well.
Thanks.

Comment 10 Jason Guiditta 2015-01-09 16:24:40 UTC
(In reply to Lars Kellogg-Stedman from comment #8)
> Don't know if this helps or not, but looking at the logs, neutron-server is
> failing with:
> 
>   ConfigFileValueError: invalid literal for int() with base 10: 'nisim'
> 
> Because of this in /etc/neutron/neutron.conf:
> 
>    quota_vip = nisim

not sure how I missed that when I was looking at the config file, but you are absoluelty right, and this is the kidn of wrong value I thought should be there.

Comment 11 Jason Guiditta 2015-01-09 16:26:46 UTC
Moving to ON_QA, as requested by Omri for testing

Comment 12 Omri Hochman 2015-01-13 20:26:40 UTC
Unable to reproduce with: rhel-osp-installer-0.5.5-1.el7ost.noarch

Comment 13 Omri Hochman 2015-01-13 20:50:39 UTC
reopen -- > (using  rhel-osp-installer-0.5.5-1.el7ost.noarch) 
On my setup after deployment already finished successfully - 
one of the controllers switched to 'error'- the report shows this message:  

Failed to call refresh: Could not restart Service[neutron-server]: Execution of '/usr/bin/systemctl restart neutron-server' returned 1: Job for neutron-server.service canceled.

neutron server.log looks like that :
------------------------------------- 
2015-01-13 15:14:17.422 4471 ERROR neutron.service [req-4d40a9a6-53c0-4155-9818-b080a997d188 None] Unrecoverable error: pleas
e check log for details.
2015-01-13 15:14:17.422 4471 TRACE neutron.service Traceback (most recent call last):
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 159, in
 serve_rpc
2015-01-13 15:14:17.422 4471 TRACE neutron.service     launcher.launch_service(rpc, workers=cfg.CONF.rpc_workers)
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.
py", line 341, in launch_service
2015-01-13 15:14:17.422 4471 TRACE neutron.service     self._start_child(wrap)
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.
py", line 322, in _start_child
2015-01-13 15:14:17.422 4471 TRACE neutron.service     status, signo = self._child_wait_for_exit_or_signal(launcher)
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.
py", line 280, in _child_wait_for_exit_or_signal
2015-01-13 15:14:17.422 4471 TRACE neutron.service     launcher.stop()
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.
py", line 128, in stop
2015-01-13 15:14:17.422 4471 TRACE neutron.service     self.services.stop()
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.
py", line 470, in stop
2015-01-13 15:14:17.422 4471 TRACE neutron.service     service.stop()
2015-01-13 15:14:17.422 4471 TRACE neutron.service   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 132, in
 stop
2015-01-13 15:14:17.422 4471 TRACE neutron.service     server.kill()
2015-01-13 15:14:17.422 4471 TRACE neutron.service AttributeError: 'MessageHandlingServer' object has no attribute 'kill'
2015-01-13 15:14:17.422 4471 TRACE neutron.service
2015-01-13 15:14:17.452 4471 CRITICAL neutron [req-4d40a9a6-53c0-4155-9818-b080a997d188 None] AttributeError: 'MessageHandlin
gServer' object has no attribute 'kill'

Comment 14 Omri Hochman 2015-01-13 20:51:42 UTC
looking at the neutron.conf (there's no: quota_vip = 'nisim' )

# Number of vips allowed per tenant. A negative value means unlimited.
# quota_vip = 10

# Number of pools allowed per tenant. A negative value means unlimited.
# quota_pool = 10

Comment 15 Mike Burns 2015-01-14 20:55:12 UTC
moving back to on_qa to retest

Comment 16 Alexander Chuzhoy 2015-01-20 17:56:07 UTC
Verified:
Environment:

rhel-osp-installer-client-0.5.5-2.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-puppet-modules-2014.2.8-1.el7ost.noarch
openstack-foreman-installer-3.0.10-2.el7ost.noarch
rhel-osp-installer-0.5.5-2.el7ost.noarch
ruby193-rubygem-staypuft-0.5.12-1.el7ost.noarch



The reported issue doesn't reproduce.

Comment 18 Scott Lewis 2015-02-10 15:12:12 UTC
This bug has been closed as a part of the RHEL-OSP 6 general availability release. For details, see https://rhn.redhat.com/errata/rhel7-rhos-6-errata.html