Bug 1174784 - HA | Neutron-server resource is being reported as failed.
Summary: HA | Neutron-server resource is being reported as failed.
Keywords:
Status: CLOSED DUPLICATE of bug 1174389
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhel-osp-installer
Version: 6.0 (Juno)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ga
: Installer
Assignee: Mike Burns
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-16 13:31 UTC by Leonid Natapov
Modified: 2014-12-16 14:03 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-16 14:03:19 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Leonid Natapov 2014-12-16 13:31:03 UTC
HA | Neutron-server resource  is being reported as failed. 
It seems like neutron-server service is being restarted on every puppet run.

rhel-osp-installer-client-0.5.3-1.el7ost.noarch
rhel-osp-installer-0.5.3-1.el7ost.noarch
openstack-puppet-modules-2014.2.7-2.el7ost.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.5-1.el7ost.noarch

server log shows the following:
--------------------------------

2014-12-16 14:56:53.444 75106 CRITICAL neutron [req-fc6a53d0-b85e-4b19-a56f-11c2a4961c2d None] AttributeError: 'MessageHandlingServer' object has no attribute 'kill'
2014-12-16 14:56:53.444 75106 TRACE neutron Traceback (most recent call last):
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/bin/neutron-server", line 10, in <module>
2014-12-16 14:56:53.444 75106 TRACE neutron     sys.exit(main())
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 51, in main
2014-12-16 14:56:53.444 75106 TRACE neutron     neutron_rpc = service.serve_rpc()
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 163, in serve_rpc
2014-12-16 14:56:53.444 75106 TRACE neutron     LOG.exception(_('Unrecoverable error: please check log '
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 82, in __exit__
2014-12-16 14:56:53.444 75106 TRACE neutron     six.reraise(self.type_, self.value, self.tb)
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 159, in serve_rpc
2014-12-16 14:56:53.444 75106 TRACE neutron     launcher.launch_service(rpc, workers=cfg.CONF.rpc_workers)
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 341, in launch_service
2014-12-16 14:56:53.444 75106 TRACE neutron     self._start_child(wrap)
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 322, in _start_child
2014-12-16 14:56:53.444 75106 TRACE neutron     status, signo = self._child_wait_for_exit_or_signal(launcher)
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 280, in _child_wait_for_exit_or_signal
2014-12-16 14:56:53.444 75106 TRACE neutron     launcher.stop()
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 128, in stop
2014-12-16 14:56:53.444 75106 TRACE neutron     self.services.stop()
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/service.py", line 470, in stop
2014-12-16 14:56:53.444 75106 TRACE neutron     service.stop()
2014-12-16 14:56:53.444 75106 TRACE neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 132, in stop
2014-12-16 14:56:53.444 75106 TRACE neutron     server.kill()
2014-12-16 14:56:53.444 75106 TRACE neutron AttributeError: 'MessageHandlingServer' object has no attribute 'kill'
-------------------------------------------------------------------------------

/var/log/messages shows that neutron server was restarted several times upon puppet run.

Dec 16 14:56:46 mac848f69fbc49f stonith-ng[14740]: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
Dec 16 14:56:47 mac848f69fbc49f stonith-ng[14740]: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
Dec 16 14:56:47 mac848f69fbc49f stonith-ng[14740]: notice: update_cib_cache_cb: [cib_diff_notify] Patch aborted: Application of an update diff failed (-206)
Dec 16 14:56:47 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Quickstack::Neutron::All/Exec[neutron-db-manage upgrade]/returns) executed successfully
Dec 16 14:56:47 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Neutron::Server::Notifications/Nova_admin_tenant_id_setter[nova_admin_tenant_id]/ensure) created
Dec 16 14:56:48 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Quickstack::Pacemaker::Cinder/Quickstack::Pacemaker::Resource::Service[openstack-cinder-api]/Exec[wait for pcmk_resource openstack-cinder-api]/returns) executed successfu
lly
Dec 16 14:56:49 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Quickstack::Pacemaker::Cinder/Quickstack::Pacemaker::Resource::Service[openstack-cinder-scheduler]/Exec[wait for pcmk_resource openstack-cinder-scheduler]/returns) execut
ed successfully
Dec 16 14:56:50 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Quickstack::Pacemaker::Cinder/Quickstack::Pacemaker::Resource::Service[openstack-cinder-volume]/Exec[wait for pcmk_resource openstack-cinder-volume]/returns) executed suc
cessfully
Dec 16 14:56:51 mac848f69fbc49f systemd: Stopping OpenStack Neutron Server...
Dec 16 14:56:52 mac848f69fbc49f systemd: Starting OpenStack Neutron Server...
Dec 16 14:56:53 mac848f69fbc49f crmd[66761]: notice: process_lrm_event: LRM operation neutron-server_monitor_30000 (call=419, rc=196, cib-update=1010, confirmed=false) OCF_PENDING
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_cs_dispatch: Update relayed from pcmk-mac848f69fbc643
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-neutron-server (1)
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_perform_update: Sent update 63115: fail-count-neutron-server=1
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_cs_dispatch: Update relayed from pcmk-mac848f69fbc643
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-neutron-server (1418734613)
Dec 16 14:56:53 mac848f69fbc49f attrd[14742]: notice: attrd_perform_update: Sent update 63117: last-failure-neutron-server=1418734613
Dec 16 14:56:53 mac848f69fbc49f systemd: Reloading.
Dec 16 14:56:53 mac848f69fbc49f systemd: Stopping OpenStack Neutron Server...
Dec 16 14:56:53 mac848f69fbc49f crmd[66761]: notice: process_lrm_event: LRM operation neutron-server_stop_0 (call=660, rc=0, cib-update=1011, confirmed=true) ok
Dec 16 14:56:53 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Failed to call refresh: Could not restart Service[neutron-server]: Execution of '/usr/bin/systemctl restart neutron-server' retur
ned 1: Job for neutron-server.service canceled.
Dec 16 14:56:53 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Could not restart Service[neutron-server]: Execution of '/usr/bin/systemctl restart neutron-server' returned 1: Job for neutron-s
erver.service canceled.
Dec 16 14:56:53 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Wrapped exception:
Dec 16 14:56:53 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Neutron::Server/Service[neutron-server]) Execution of '/usr/bin/systemctl restart neutron-server' returned 1: Job for neutron-server.service canceled.
Dec 16 14:56:53 mac848f69fbc49f systemd: Reloading.
Dec 16 14:56:53 mac848f69fbc49f systemd: Starting OpenStack Neutron Server...
Dec 16 14:56:53 mac848f69fbc49f crmd[66761]: notice: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Dec 16 14:56:53 mac848f69fbc49f cib[14739]: warning: cib_process_diff: Diff 0.257.1848 -> 0.257.1 from pcmk-mac848f69fbc643 not applied to 0.257.1848: Failed application of an update diff
Dec 16 14:56:53 mac848f69fbc49f puppet-agent[68008]: (/Stage[main]/Quickstack::Pacemaker::Neutron/Exec[pcs-neutron-server-set-up]/returns) executed successfully
Dec 16 14:56:53 mac848f69fbc49f cib[14739]: notice: cib_server_process_diff: Not applying diff 0.257.1 -> 0.257.2 (sync in progress)
Dec 16 14:56:53 mac848f69fbc49f systemd: Starting OpenStack Neutron Server...

Comment 1 Mike Burns 2014-12-16 14:03:19 UTC

*** This bug has been marked as a duplicate of bug 1174389 ***


Note You need to log in before you can comment on or make changes to this bug.