Description of problem: Scaling out an overcloud with nodes registered to a Sallite 6 server fails with "msg": "Unable to reach the server at $sat_url:443/subscription", "rc": 69 2018-10-19 15:40:17,817 p=27958 u=mistral | TASK [redhat-subscription : Manage Red Hat subscription] *********************** 2018-10-19 15:40:17,817 p=27958 u=mistral | Friday 19 October 2018 15:40:17 -0400 (0:00:00.430) 0:04:37.678 ******** 2018-10-19 15:40:19,327 p=27958 u=mistral | fatal: [controller-0]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []} 2018-10-19 15:40:19,331 p=27958 u=mistral | fatal: [controller-2]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []} 2018-10-19 15:40:19,347 p=27958 u=mistral | fatal: [controller-1]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []} 2018-10-19 15:40:19,349 p=27958 u=mistral | NO MORE HOSTS LEFT ************************************************************* 2018-10-19 15:40:19,350 p=27958 u=mistral | PLAY RECAP ********************************************************************* 2018-10-19 15:40:19,350 p=27958 u=mistral | ceph-0 : ok=76 changed=20 unreachable=0 failed=0 2018-10-19 15:40:19,350 p=27958 u=mistral | ceph-1 : ok=76 changed=20 unreachable=0 failed=0 2018-10-19 15:40:19,350 p=27958 u=mistral | ceph-2 : ok=76 changed=20 unreachable=0 failed=0 2018-10-19 15:40:19,351 p=27958 u=mistral | compute-0 : ok=76 changed=20 unreachable=0 failed=0 2018-10-19 15:40:19,351 p=27958 u=mistral | compute-1 : ok=76 changed=20 unreachable=0 failed=0 2018-10-19 15:40:19,351 p=27958 u=mistral | compute-2 : ok=69 changed=21 unreachable=0 failed=0 2018-10-19 15:40:19,351 p=27958 u=mistral | controller-0 : ok=158 changed=25 unreachable=0 failed=1 2018-10-19 15:40:19,351 p=27958 u=mistral | controller-1 : ok=158 changed=25 unreachable=0 failed=1 2018-10-19 15:40:19,351 p=27958 u=mistral | controller-2 : ok=160 changed=25 unreachable=0 failed=1 2018-10-19 15:40:19,351 p=27958 u=mistral | undercloud : ok=2 changed=0 unreachable=0 failed=0 2018-10-19 15:40:19,352 p=27958 u=mistral | Friday 19 October 2018 15:40:19 -0400 (0:00:01.534) 0:04:39.212 ******** 2018-10-19 15:40:19,352 p=27958 u=mistral | =============================================================================== Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-9.0.0-0.20181001174822.90afd18.0rc2.el7ost.noarch ansible-role-redhat-subscription-1.0.1-1.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with nodes registered to satellite 2. Assert successful deployment 3. Run overcloud deploy command for a second time Actual results: Failure during TASK [redhat-subscription : Manage Red Hat subscription] Expected results: No failures. Additional info: Running the command manually on the overcloud nodes: [root@controller-0 heat-admin]# /sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev; echo $? Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription 69 [root@controller-0 heat-admin]# subscription-manager status +-------------------------------------------+ System Status Details +-------------------------------------------+ Overall Status: Current [root@controller-0 heat-admin]# cat /var/log/rhsm/rhsm.log 2018-10-19 19:46:45,227 [INFO] subscription-manager:478159:MainThread @managercli.py:452 - X-Correlation-ID: 8c21206b44b94d049caecb85527f49cd 2018-10-19 19:46:45,227 [INFO] subscription-manager:478159:MainThread @managercli.py:341 - Client Versions: {'subscription-manager': '1.20.11-1.el7_5'} 2018-10-19 19:46:45,228 [INFO] subscription-manager:478159:MainThread @connection.py:868 - Connection built: host=rhos-compute-node-08.lab.eng.rdu2.redhat.com port=443 handler=/subscription auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True 2018-10-19 19:46:45,229 [INFO] subscription-manager:478159:MainThread @connection.py:868 - Connection built: host=rhos-compute-node-08.lab.eng.rdu2.redhat.com port=443 handler=/subscription auth=none 2018-10-19 19:46:45,249 [INFO] subscription-manager:478159:MainThread @connection.py:586 - Response: status=404, request="GET /subscription/status/" 2018-10-19 19:46:45,250 [ERROR] subscription-manager:478159:MainThread @utils.py:149 - Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/subscription_manager/utils.py", line 141, in is_valid_server_info conn.ping() File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 929, in ping return self.conn.request_get("/status/") File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 690, in request_get return self._request("GET", method, headers=headers) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 716, in _request info=info, headers=headers) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 595, in _request self.validateResponse(result, request_type, handler) File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 654, in validateResponse raise RestlibException(response['status'], error_msg, response.get('headers')) RestlibException
This is happening due to the changes the katello-rhsm-consumer script is making to the system. I'm working to convert that script into Ansible tasks in order to make the role work reliably.
Fixed in the following patches: https://review.openstack.org/#/c/611334/ https://review.openstack.org/#/c/612508/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045