Description of problem: After upgrading undercloud from rhosp 9 to rhosp 10, and upload new images to openstack overcloud, when attempt to do a "openstack baremetal configure boot" throws an error below because of this, when attempting to upgrade overcloud from rhosp 9 to 10 fail [stack@vm-240 ~]$ openstack baremetal configure boot Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read. [Errno 111] Connection refused [stack@vm-240 ~]$ openstack overcloud node configure 1b242e24-40e3-4690-8447-963293287878 Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read. [Errno 111] Connection refused [stack@vm-240 ~]$ [0] 0:stack@vm-240:~* Node uuid=59f83f93-e5f8-4054-9da7-5d2bac2dded7 has an incorrectly configured driver_info/deploy_ramdisk. Expected "a4294703-8264-42da-a309-088d4cb101f8" but got "abbee89b-2893-4171-a5f2-6a296bdd4937". Node uuid=59f83f93-e5f8-4054-9da7-5d2bac2dded7 has an incorrectly configured driver_info/deploy_kernel. Expected "fe755387-7425-4fa7-b2d8-8318262d018c" but got "3174ddbf-6e64-4299-a5fd-5b3ad2b39fe3". Checking config for Node 00d6dd8b-8bec-4b45-b047-c1619578dca1 Doing boot checks for 00d6dd8b-8bec-4b45-b047-c1619578dca1 Node uuid=00d6dd8b-8bec-4b45-b047-c1619578dca1 has an incorrectly configured driver_info/deploy_ramdisk. Expected "a4294703-8264-42da-a309-088d4cb101f8" but got "abbee89b-2893-4171-a5f2-6a296bdd4937". Node uuid=00d6dd8b-8bec-4b45-b047-c1619578dca1 has an incorrectly configured driver_info/deploy_kernel. Expected "fe755387-7425-4fa7-b2d8-8318262d018c" but got "3174ddbf-6e64-4299-a5fd-5b3ad2b39fe3". Checking config for Node 01a6f81c-05b3-4205-a1b3-aef3b67b64e9 Doing boot checks for 01a6f81c-05b3-4205-a1b3-aef3b67b64e9 Node uuid=01a6f81c-05b3-4205-a1b3-aef3b67b64e9 has an incorrectly configured driver_info/deploy_ramdisk. Expected "a4294703-8264-42da-a309-088d4cb101f8" but got "abbee89b-2893-4171-a5f2-6a296bdd4937". Node uuid=01a6f81c-05b3-4205-a1b3-aef3b67b64e9 has an incorrectly configured driver_info/deploy_kernel. Expected "fe755387-7425-4fa7-b2d8-8318262d018c" but got "3174ddbf-6e64-4299-a5fd-5b3ad2b39fe3". Version-Release number of selected component (if applicable): RHOSP10 How reproducible: always Steps to Reproduce: 1. upgrade rhosp9 to rhosp10 on undercloud 2. untar rhosp10 images and upload to openstack 3. run command openstack overcloud configure boot
Seeing this in zagar.log [stack@vm-240 zaqar]$ cat zaqar.log 2017-06-21 20:17:55.034 14213 INFO zaqar.transport.wsgi.driver [-] Serving on host 192.0.2.1:8888 2017-06-21 20:24:54.941 14213 WARNING keystonemiddleware.auth_token [-] Using the in-process token cache is deprecated as of the 4.2.0 release and may be removed in the 5.0.0 release or the 'O' development cycle. The in-process cache causes inconsistent results and high memory usage. When the feature is removed the auth_token middleware will not cache tokens by default which may result in performance issues. It is recommended to use memcache for the auth_token token cache by setting the memcached_servers option. 2017-06-21 20:33:49.095 1366 ERROR zaqar.storage.utils [-] Failed to load "zaqar.data.storage" driver for "mongodb" 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils [-] 127.0.0.1:27017: [Errno 111] Connection refused 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils Traceback (most recent call last): 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/zaqar/storage/utils.py", line 140, in load_storage_driver 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils invoke_args=_invoke_args) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 46, in __init__ 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils verify_requirements=verify_requirements, 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 69, in __init__ 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils verify_requirements) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 171, in _load_plugins 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils self._on_load_failure_callback(self, ep, err) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 163, in _load_plugins 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils verify_requirements, 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 146, in _load_one_plugin 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils verify_requirements, 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 195, in _load_one_plugin 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils obj = plugin(*invoke_args, **invoke_kwds) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib/python2.7/site-packages/zaqar/storage/mongodb/driver.py", line 83, in __init__ 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils server_info = conn.server_info()['version'] 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 898, in server_info 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils read_preference=ReadPreference.PRIMARY) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 451, in command 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils with client._socket_for_reads(read_preference) as (sock_info, slave_ok): 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils return self.gen.next() 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 699, in _socket_for_reads 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils with self._get_socket(read_preference) as sock_info: 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils return self.gen.next() 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 663, in _get_socket 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils server = self._get_topology().select_server(selector) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 121, in select_server 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils address)) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils File "/usr/lib64/python2.7/site-packages/pymongo/topology.py", line 97, in select_servers 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils self._error_message(selector)) 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils ServerSelectionTimeoutError: 127.0.0.1:27017: [Errno 111] Connection refused 2017-06-21 20:33:49.170 1366 ERROR zaqar.storage.utils 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli [-] 127.0.0.1:27017: [Errno 111] Connection refused 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli Traceback (most recent call last): 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/common/cli.py", line 54, in _wrapper 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli func() 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/cmd/server.py", line 81, in run 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli server.run() 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/bootstrap.py", line 118, in run 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli self.transport.listen() 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/common/decorators.py", line 175, in getter 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli setattr(self, attr_name, fn(self)) 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/bootstrap.py", line 99, in transport 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli self.storage, 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/common/decorators.py", line 175, in getter 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli setattr(self, attr_name, fn(self)) 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/bootstrap.py", line 67, in storage 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli self.conf, self.cache, control_driver=self.control) 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli File "/usr/lib/python2.7/site-packages/zaqar/storage/utils.py", line 148, in load_storage_driver 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli raise errors.InvalidDriver(exc) 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli InvalidDriver: 127.0.0.1:27017: [Errno 111] Connection refused 2017-06-21 20:33:49.242 1366 ERROR zaqar.common.cli [stack@vm-240 zaqar]$ sudo netstat -antp | grep 27017 tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 1871/mongod [stack@vm-240 zaqar]$
the zaqar services is not running, restarting it resolve the configure boot issue. closing this bugzilla as not a problem.