Bug 1641180 - Scaling out an overcloud with nodes registered to a Satellite 6 server fails with "msg": "Unable to reach the server at $sat_url:443/subscription", "rc": 69
Summary: Scaling out an overcloud with nodes registered to a Satellite 6 server fails ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: ansible-role-redhat-subscription
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 14.0 (Rocky)
Assignee: Sam Doran
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks: 1643254
TreeView+ depends on / blocked
 
Reported: 2018-10-19 19:47 UTC by Marius Cornea
Modified: 2019-01-11 11:54 UTC (History)
7 users (show)

Fixed In Version: ansible-role-redhat-subscription-1.0.1-4.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, the Satellite URL was not correctly set in the role. This prevented the system from getting the Satellite server version, and registration failed. This fix adds the capability to get the `rhsm_satellite_url` value from the `rhsm_baseurl` parameter by default, passes the URL to the registration task to allow force registration, and adds the option to ignore certificate errors. You can override the default value or configure the options as needed.
Clone Of:
: 1643254 (view as bug list)
Environment:
Last Closed: 2019-01-11 11:54:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:54:18 UTC

Description Marius Cornea 2018-10-19 19:47:15 UTC
Description of problem:
Scaling out an overcloud with nodes registered to a Sallite 6 server fails with "msg": "Unable to reach the server at $sat_url:443/subscription", "rc": 69

2018-10-19 15:40:17,817 p=27958 u=mistral |  TASK [redhat-subscription : Manage Red Hat subscription] ***********************
2018-10-19 15:40:17,817 p=27958 u=mistral |  Friday 19 October 2018  15:40:17 -0400 (0:00:00.430)       0:04:37.678 ******** 
2018-10-19 15:40:19,327 p=27958 u=mistral |  fatal: [controller-0]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []}
2018-10-19 15:40:19,331 p=27958 u=mistral |  fatal: [controller-2]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []}
2018-10-19 15:40:19,347 p=27958 u=mistral |  fatal: [controller-1]: FAILED! => {"changed": false, "cmd": "/sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev", "msg": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription", "rc": 69, "stderr": "Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription\n", "stderr_lines": ["Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription"], "stdout": "", "stdout_lines": []}
2018-10-19 15:40:19,349 p=27958 u=mistral |  NO MORE HOSTS LEFT *************************************************************
2018-10-19 15:40:19,350 p=27958 u=mistral |  PLAY RECAP *********************************************************************
2018-10-19 15:40:19,350 p=27958 u=mistral |  ceph-0                     : ok=76   changed=20   unreachable=0    failed=0   
2018-10-19 15:40:19,350 p=27958 u=mistral |  ceph-1                     : ok=76   changed=20   unreachable=0    failed=0   
2018-10-19 15:40:19,350 p=27958 u=mistral |  ceph-2                     : ok=76   changed=20   unreachable=0    failed=0   
2018-10-19 15:40:19,351 p=27958 u=mistral |  compute-0                  : ok=76   changed=20   unreachable=0    failed=0   
2018-10-19 15:40:19,351 p=27958 u=mistral |  compute-1                  : ok=76   changed=20   unreachable=0    failed=0   
2018-10-19 15:40:19,351 p=27958 u=mistral |  compute-2                  : ok=69   changed=21   unreachable=0    failed=0   
2018-10-19 15:40:19,351 p=27958 u=mistral |  controller-0               : ok=158  changed=25   unreachable=0    failed=1   
2018-10-19 15:40:19,351 p=27958 u=mistral |  controller-1               : ok=158  changed=25   unreachable=0    failed=1   
2018-10-19 15:40:19,351 p=27958 u=mistral |  controller-2               : ok=160  changed=25   unreachable=0    failed=1   
2018-10-19 15:40:19,351 p=27958 u=mistral |  undercloud                 : ok=2    changed=0    unreachable=0    failed=0   
2018-10-19 15:40:19,352 p=27958 u=mistral |  Friday 19 October 2018  15:40:19 -0400 (0:00:01.534)       0:04:39.212 ******** 
2018-10-19 15:40:19,352 p=27958 u=mistral |  =============================================================================== 


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-9.0.0-0.20181001174822.90afd18.0rc2.el7ost.noarch
ansible-role-redhat-subscription-1.0.1-1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with nodes registered to satellite
2. Assert successful deployment
3. Run overcloud deploy command for a second time

Actual results:
Failure during TASK [redhat-subscription : Manage Red Hat subscription]

Expected results:
No failures.

Additional info:
Running the command manually on the overcloud nodes:

[root@controller-0 heat-admin]# /sbin/subscription-manager register --baseurl https://rhos-compute-node-08.lab.eng.rdu2.redhat.com/pulp/repos --org Default_Organization --activationkey osp14-dev; echo $?
Unable to reach the server at rhos-compute-node-08.lab.eng.rdu2.redhat.com:443/subscription
69


[root@controller-0 heat-admin]# subscription-manager status
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Current

[root@controller-0 heat-admin]# cat /var/log/rhsm/rhsm.log 
2018-10-19 19:46:45,227 [INFO] subscription-manager:478159:MainThread @managercli.py:452 - X-Correlation-ID: 8c21206b44b94d049caecb85527f49cd
2018-10-19 19:46:45,227 [INFO] subscription-manager:478159:MainThread @managercli.py:341 - Client Versions: {'subscription-manager': '1.20.11-1.el7_5'}
2018-10-19 19:46:45,228 [INFO] subscription-manager:478159:MainThread @connection.py:868 - Connection built: host=rhos-compute-node-08.lab.eng.rdu2.redhat.com port=443 handler=/subscription auth=identity_cert ca_dir=/etc/rhsm/ca/ insecure=True
2018-10-19 19:46:45,229 [INFO] subscription-manager:478159:MainThread @connection.py:868 - Connection built: host=rhos-compute-node-08.lab.eng.rdu2.redhat.com port=443 handler=/subscription auth=none
2018-10-19 19:46:45,249 [INFO] subscription-manager:478159:MainThread @connection.py:586 - Response: status=404, request="GET /subscription/status/"
2018-10-19 19:46:45,250 [ERROR] subscription-manager:478159:MainThread @utils.py:149 - 
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/utils.py", line 141, in is_valid_server_info
    conn.ping()
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 929, in ping
    return self.conn.request_get("/status/")
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 690, in request_get
    return self._request("GET", method, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 716, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 595, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 654, in validateResponse
    raise RestlibException(response['status'], error_msg, response.get('headers'))
RestlibException

Comment 1 Sam Doran 2018-10-22 12:21:41 UTC
This is happening due to the changes the katello-rhsm-consumer script is making to the system. I'm working to convert that script into Ansible tasks in order to make the role work reliably.

Comment 2 Sam Doran 2018-10-23 14:10:39 UTC
Fixed in the following patches:

https://review.openstack.org/#/c/611334/
https://review.openstack.org/#/c/612508/

Comment 20 errata-xmlrpc 2019-01-11 11:54:07 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.