Bug 1337624
| Summary: | rhel-osp-director: upgrade 7.3->8.0, that follows update 7.2->7.3, times out during "major-upgrade-pacemaker-converge.yaml" step | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | rhosp-director | Assignee: | Angus Thomas <athomas> |
| Status: | CLOSED NOTABUG | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.0 (Liberty) | CC: | dbecker, mburns, michele, morazi, rhel-osp-director-maint |
| Target Milestone: | async | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-05-19 21:08:45 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
It was probably an environment issue (compute got stuck). I was able to upgrade successfully on subsequent run. |
rhel-osp-director: upgrade 7.3->8.0, that follows update 7.2->7.3, times out during "major-upgrade-pacemaker-converge.yaml" step Environment: openstack-tripleo-heat-templates-0.8.14-9.el7ost.noarch instack-undercloud-2.2.7-4.el7ost.noarch openstack-tripleo-heat-templates-kilo-0.8.14-9.el7ost.noarch openstack-puppet-modules-7.0.17-1.el7ost.noarch Steps to reproduce: 1. Deploy 7.2 HA with netiso (deployment command: openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --ceph-storage-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml) 2. populate the setup. 3. Update to 7.3 4. Add another instance to the setup. 5. Attempt to upgrade to 8.0 Result: The upgrade fails after 4 hours run during the major-upgrade-pacemaker-converge step with: 2016-05-19 03:28:25 [overcloud-CephStorageCephDeployment-yxak3hlzkhck]: UPDATE_COMPLETE Stack UPDATE completed successfully 2016-05-19 03:28:27 [ControllerCephDeployment]: UPDATE_COMPLETE state changed 2016-05-19 03:28:27 [CephStorageCephDeployment]: UPDATE_COMPLETE state changed ERROR: Authentication failed: Authentication required [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ [stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE +--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+ | resource_name | physical_resource_id | resource_type | resource_status | updated_time | stack_name | +--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+ | Compute | 520b5036-48b2-4ff7-9e22-2c0d40d9d6f4 | OS::Heat::ResourceGroup | UPDATE_FAILED | 2016-05-19T03:24:56 | overcloud | | 0 | bbea6423-3e7f-41d2-b8d9-425f9c9bd088 | OS::TripleO::Compute | UPDATE_FAILED | 2016-05-19T03:25:05 | overcloud-Compute-jfxnp5zhnqu2 | | UpdateDeployment | 2c3e65db-b54b-4c07-aa86-c1d77690747d | OS::Heat::SoftwareDeployment | CREATE_FAILED | 2016-05-19T03:26:22 | overcloud-Compute-jfxnp5zhnqu2-0-gwvboa4eenug | +--------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+---------------------+-----------------------------------------------------------------------------------------------+ [stack@instack ~]$ heat deployment-show 2c3e65db-b54b-4c07-aa86-c1d77690747d { "status": "IN_PROGRESS", "server_id": "0c6981c8-0ee7-4df2-be98-50d258992f01", "config_id": "573938c4-542d-4f49-a83a-4c825f6ff097", "output_values": null, "creation_time": "2016-05-19T03:26:29", "input_values": {}, "action": "CREATE", "status_reason": "Deploy data available", "id": "2c3e65db-b54b-4c07-aa86-c1d77690747d" } [stack@instack ~]$ The heat-engine.log (which is too big to attach) has these repeating block of erros: 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service [-] Unhandled exception 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service Traceback (most recent call last): 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 377, in _child_wait_for_exit_or_signal 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service launcher.wait() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 204, in wait 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.services.wait() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 625, in wait 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service service.wait() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return f(*args, **kwargs) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 591, in wait 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self._done.wait() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return hubs.get_hub().switch() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return self.greenlet.switch() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.wait(sleep_time) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service presult = self.do_poll(seconds) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return self.poll.poll(seconds) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 160, in _handle_signals 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service handler(signo, frame) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 356, in _sigterm 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.launcher.stop() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 196, in stop 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.services.stop() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 612, in stop 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service service.stop() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return f(*args, **kwargs) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 408, in stop 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service service_objects.Service.delete(ctxt, self.service_id) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/heat/objects/service.py", line 78, in delete 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service db_api.service_delete(context, service_id, soft_delete) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/heat/db/api.py", line 378, in service_delete 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return IMPL.service_delete(context, service_id, soft_delete) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 1006, in service_delete 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service service = service_get(context, service_id) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/heat/db/sqlalchemy/api.py", line 1016, in service_get 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service result = model_query(context, models.Service).get(service_id) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 819, in get 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return self._get_impl(ident, loading.load_on_ident) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 852, in _get_impl 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return fallback_fn(self, key) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service return q.one() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2473, in one 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service ret = list(self) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 86, in instances 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service util.raise_from_cause(err) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 67, in instances 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service fetch = cursor.fetchall() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 968, in fetchall 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.cursor, self.context) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service util.raise_from_cause(newraise, exc_info) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service reraise(type(exception), exception, tb=exc_tb) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 963, in fetchall 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self._soft_close() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 667, in _soft_close 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self.connection.close() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in close 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service conn.close() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 882, in close 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self._checkin() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 758, in _checkin 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service self._pool, None, self._echo, fairy=self) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 650, in _finalize_fairy 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service connection_record.checkin() 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 509, in checkin 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service pool.dispatch.checkin(connection, self) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service fn(*args, **kw) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 50, in _thread_yield 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service time.sleep(0) 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 31, in sleep 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service assert hub.greenlet is not current, 'do not call blocking functions from the mainloop' 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service DBError: do not call blocking functions from the mainloop 2016-05-18 17:39:11.156 31851 ERROR oslo_service.service