Bug 1595206
| Summary: | [UPDATE] After update cinder-scheduler keep exiting with error "Failed to declare consumer for topic 'cinder-schedul er'" | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Raviv Bar-Tal <rbartal> | ||||||
| Component: | openstack-cinder | Assignee: | John Eckersberg <jeckersb> | ||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Avi Avraham <aavraham> | ||||||
| Severity: | unspecified | Docs Contact: | Kim Nylander <knylande> | ||||||
| Priority: | unspecified | ||||||||
| Version: | 13.0 (Queens) | CC: | abeekhof, abishop, rbartal, srevivo | ||||||
| Target Milestone: | --- | Keywords: | ZStream | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2018-07-24 08:54:33 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1454623 [details]
cinder-scheduler.log
The log shows issues with cinder connecting to rabbitmq, so moving this to PIDONE. As a side note, I believe we don't support updating beta releases to RC/GA, but I'm not sure if that's relevant here. Created attachment 1454630 [details]
controller-0 /var/log/container
Can you provide the full rabbitmq logs? The sosreports from the controllers only have the tailed logs. I did not manage to reproduce this bug with newer puddle (2018-07-13.1), I assume this was fix unintentionally |
Description of problem: openstack-cinder-scheduler docker keeps restarting and exiting after ~2 minutes with this error: 2018-06-26 11:06:20.674 1 ERROR oslo.messaging._drivers.impl_rabbit [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Failed to declare consumer for topic 'cinder-schedul er': Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout: NotFound: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout 2018-06-26 11:06:20.675 1 ERROR oslo.messaging._drivers.impl_rabbit [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Unable to connect to AMQP server on controller-0.int ernalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout: NotFound: Q ueue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout 2018-06-26 11:06:20.676 1 ERROR oslo_service.service [req-3b52dc4a-84f7-4f6b-bca6-fd5b47583a72 - - - - -] Error starting thread.: MessageDeliveryFailure: Unable to connect t o AMQP server on controller-0.internalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhos t '/' due to timeout 2018-06-26 11:06:20.676 1 ERROR oslo_service.service Traceback (most recent call last): 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service 2018-06-26 11:06:20.676 1 ERROR oslo_service.service service.start() 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/service.py", line 239, in start 2018-06-26 11:06:20.676 1 ERROR oslo_service.service self.rpcserver.start() 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper 2018-06-26 11:06:20.676 1 ERROR oslo_service.service log_after, timeout_timer) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once 2018-06-26 11:06:20.676 1 ERROR oslo_service.service post_fn = fn() 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda> 2018-06-26 11:06:20.676 1 ERROR oslo_service.service states[state].run_once(lambda: fn(self, *args, **kwargs), 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 416, in start 2018-06-26 11:06:20.676 1 ERROR oslo_service.service self.listener = self._create_listener() 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 151, in _create_listener 2018-06-26 11:06:20.676 1 ERROR oslo_service.service return self.transport._listen(self._target, 1, None) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 146, in _listen 2018-06-26 11:06:20.676 1 ERROR oslo_service.service batch_timeout) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 572, in listen 2018-06-26 11:06:20.676 1 ERROR oslo_service.service callback=listener) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1120, in declare_topic_consumer 2018-06-26 11:06:20.676 1 ERROR oslo_service.service self.declare_consumer(consumer) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1023, in declare_consumer 2018-06-26 11:06:20.676 1 ERROR oslo_service.service error_callback=_connect_error) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure 2018-06-26 11:06:20.676 1 ERROR oslo_service.service raise exceptions.MessageDeliveryFailure(msg) 2018-06-26 11:06:20.676 1 ERROR oslo_service.service MessageDeliveryFailure: Unable to connect to AMQP server on controller-0.internalapi.localdomain:5672 after None tries: Queue.declare: (404) NOT_FOUND - failed to perform operation on queue 'cinder-scheduler' in vhost '/' due to timeout 2018-06-26 11:06:20.676 1 ERROR oslo_service.service 2018-06-26 11:06:20.679 1 CRITICAL cinder [req-831337c4-ce4b-4877-97d2-01da4fb7dc9d - - - - -] Unhandled error: AttributeError: 'NoneType' object has no attribute 'cleanup' 2018-06-26 11:06:20.679 1 ERROR cinder Traceback (most recent call last): 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/bin/cinder-scheduler", line 10, in <module> 2018-06-26 11:06:20.679 1 ERROR cinder sys.exit(main()) 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 55, in main 2018-06-26 11:06:20.679 1 ERROR cinder service.wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 629, in wait 2018-06-26 11:06:20.679 1 ERROR cinder _launcher.wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 334, in wait 2018-06-26 11:06:20.679 1 ERROR cinder super(ServiceLauncher, self).wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 237, in wait 2018-06-26 11:06:20.679 1 ERROR cinder self.services.wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in wait 2018-06-26 11:06:20.679 1 ERROR cinder service.wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 442, in wait 2018-06-26 11:06:20.679 1 ERROR cinder self.rpcserver.wait() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 270, in wrapper 2018-06-26 11:06:20.679 1 ERROR cinder log_after, timeout_timer) 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 190, in run_once 2018-06-26 11:06:20.679 1 ERROR cinder post_fn = fn() 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 269, in <lambda> 2018-06-26 11:06:20.679 1 ERROR cinder states[state].run_once(lambda: fn(self, *args, **kwargs), 2018-06-26 11:06:20.679 1 ERROR cinder File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 448, in wait 2018-06-26 11:06:20.679 1 ERROR cinder self.listener.cleanup() 2018-06-26 11:06:20.679 1 ERROR cinder AttributeError: 'NoneType' object has no attribute 'cleanup' 2018-06-26 11:06:20.679 1 ERROR cinder Version-Release number of selected component (if applicable): puppet-cinder-12.4.1-0.20180329071637.4011a82.el7ost.noarch python2-cinderclient-3.5.0-1.el7ost.noarch python-cinder-12.0.1-0.20180418194614.c476898.el7ost.noarch openstack-cinder-12.0.1-0.20180418194614.c476898.el7ost.noarch How reproducible: Steps to Reproduce: 1.Install osp13 beta puddle 2. update to 2018-06-21.2 puddle 3.check cinder-scheduler on the controller ('docker ps | grep cinder-scheduler') Actual results: The docker is restarting, remains in "Up" (running) status for 2 minutes and then restating again. root@controller-0 ~]# docker ps | grep cinder 89ddacb25105 192.168.24.1:8787/rhosp13/openstack-cinder-api:2018-06-21.2 "kolla_start" 17 hours ago Up 17 hours cinder_api_cron e2f9a7d9395b 192.168.24.1:8787/rhosp13/openstack-cinder-scheduler:2018-06-21.2 "kolla_start" 17 hours ago Up 37 seconds (healthy) cinder_scheduler c8e4abfe8344 192.168.24.1:8787/rhosp13/openstack-cinder-api:2018-06-21.2 "kolla_start" 17 hours ago Up 17 hours cinder_api Expected results: Additional info: