Bug 1868936

Hi Redhat team --

Any update? 

We have the same issue.

Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID 9f96ee89-f881-44a3-82e0-61709500182e has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
The system has been registered with ID: 19da4e3b-ee80-4835-adea-a2d8494aaa96
The registered system name is: nausplnsqld43
Traceback (most recent call last):
  File "/usr/sbin/subscription-manager", line 9, in <module>
    load_entry_point('subscription-manager==1.24.13', 'console_scripts', 'subscription-manager')()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/scripts/subscription_manager.py", line 85, in main
    return managercli.ManagerCLI().main()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 2922, in main
    ret = CLI.main(self)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cli.py", line 183, in main
    return cmd.main()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 506, in main
    return_code = self._do_command()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1368, in _do_command
    profile_mgr.update_check(self.cp, consumer['uuid'], True)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 456, in update_check
    return CacheManager.update_check(self, uep, consumer_uuid, force)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 183, in update_check
    raise Exception(_("Error updating system data on the server, see /var/log/rhsm/rhsm.log "
Exception: Error updating system data on the server, see /var/log/rhsm/rhsm.log for more details.

A potential workaround:

- Register via subscription-manager, receive the error
- The system should still appear in https://access.redhat.com/management/systems
- Select the system, go to the subscriptions tab and either run auto-attach or attach a subscription manually
- On the system locally run `# subscription-manager refresh` and this will grab the subscription for use.

Same issue:
RemoteServerException: Server error attempting a PUT to /subscription/consumers/...

after using subscription-manager register --auto-attach --force (which worked within last two weeks)

# rpm -q subscription-manager
subscription-manager-1.24.26-4.el7_8.x86_64

#  subscription-manager register --username=....  --auto-attach --force
...

Unregistering from: subscription.rhsm.redhat.com:443/subscription
The system with UUID BLAH-BLAH-BLAH-BLAHb795 has been unregistered
All local data removed
Registering to: subscription.rhsm.redhat.com:443/subscription
The system has been registered with ID: BLAH-BLAH-BLAH-BLAHb6bc
The registered system name is: me.mydomain.com
...
Traceback (most recent call last):
  File "/sbin/subscription-manager", line 9, in <module>
    load_entry_point('subscription-manager==1.24.26', 'console_scripts', 'subscription-manager')()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/scripts/subscription_manager.py", line 86, in main
    return managercli.ManagerCLI().main()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 2989, in main
    ret = CLI.main(self)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cli.py", line 183, in main
    return cmd.main()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 536, in main
    return_code = self._do_command()
  File "/usr/lib64/python2.7/site-packages/subscription_manager/managercli.py", line 1398, in _do_command
    profile_mgr.update_check(self.cp, consumer['uuid'], True)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 468, in update_check
    return CacheManager.update_check(self, uep, consumer_uuid, force)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 183, in update_check
    raise Exception(_("Error updating system data on the server, see /var/log/rhsm/rhsm.log "
Exception: Error updating system data on the server, see /var/log/rhsm/rhsm.log for more details.
#

============== FROM LOG FILE =======================

2020-08-15 10:27:44,792 [ERROR] rhsmcertd-worker:11112:MainThread @cache.py:182 - Server error attempting a PUT to /subscription/consumers/BLAH-BLAH-BLAH-BLAH/profiles returned status 503
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 173, in update_check
    self._sync_with_server(uep, consumer_uuid)
  File "/usr/lib64/python2.7/site-packages/subscription_manager/cache.py", line 506, in _sync_with_server
    _combined_profile
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 1187, in updateCombinedProfile
    return self.conn.request_put(method, profile)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 735, in request_put
    return self._request("PUT", method, params, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 752, in _request
    info=info, headers=headers)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 627, in _request
    self.validateResponse(result, request_type, handler)
  File "/usr/lib64/python2.7/site-packages/rhsm/connection.py", line 695, in validateResponse
    handler=handler)
RemoteServerException: Server error attempting a PUT to /subscription/consumers/d2b32369-4e99-4174-88b4-cd647dffb795/profiles returned status 503

Trying workaround...
"Paul Dudley 2020-08-14 21:39:43 UTC
A potential workaround:

- Register via subscription-manager, receive the error
- The system should still appear in https://access.redhat.com/management/systems
- Select the system, go to the subscriptions tab and either run auto-attach or attach a subscription manually
- On the system locally run `# subscription-manager refresh` and this will grab the subscription for use."

Problem hit by a colleague who caught the server response by modifying /usr/lib64/python2.7/site-packages/rhsm/connection.py like this:
 
# diff -u /usr/lib64/python2.7/site-packages/rhsm/connection.py{.ORIG,}
--- /usr/lib64/python2.7/site-packages/rhsm/connection.py.ORIG 2020-08-15 16:43:28.550936391 -0400
+++ /usr/lib64/python2.7/site-packages/rhsm/connection.py 2020-08-15 16:44:27.520160955 -0400
@@ -690,6 +690,9 @@
             else:
                 # This really needs an exception mapper too...
                 if str(response['status']) in ["404", "410", "500", "502", "503", "504"]:
+                    print("==========")
+                    print(response)
+                    print("==========")
                     raise RemoteServerException(response['status'],
                                                 request_type=request_type,
                                                 handler=handler)
 
He got this:
 
==========
{'content': u'<html>\n  <head>\n    <meta name="viewport" content="width=device-width, initial-scale=1">\n\n  <style type="text/css">\n  /*!\n   * Bootstrap v3.3.5 (http://getbootstrap.com)\n   * Copyright 2011-2015 Twitter, Inc.\n   * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n   */\n  /*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */\n  html {\n    font-family: sans-serif;\n    -ms-text-size-adjust: 100%;\n    -webkit-text-size-adjust: 100%;\n  }\n  body {\n    margin: 0;\n  }\n  h1 {\n    font-size: 1.7em;\n    font-weight: 400;\n    line-height: 1.3;\n    margin: 0.68em 0;\n  }\n  * {\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n  }\n  *:before,\n  *:after {\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n  }\n  html {\n    -webkit-tap-highlight-color: rgba(0, 0, 0, 0);\n  }\n  body {\n    font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n    line-height: 1.66666667;\n    font-size: 13px;\n    color: #333333;\n    background-color: #ffffff;\n    margin: 2em 1em;\n  }\n  p {\n    margin: 0 0 10px;\n    font-size: 13px;\n  }\n  .alert.alert-info {\n    padding: 15px;\n    margin-bottom: 20px;\n    border: 1px solid transparent;\n    background-color: #f5f5f5;\n    border-color: #8b8d8f;\n    color: #363636;\n    margin-top: 30px;\n  }\n  .alert p {\n    padding-left: 35px;\n  }\n  a {\n    color: #0088ce;\n  }\n\n  ul {\n    position: relative;\n    padding-left: 51px;\n  }\n  p.info {\n    position: relative;\n    font-size: 15px;\n    margin-bottom: 10px;\n  }\n  p.info:before, p.info:after {\n    content: "";\n    position: absolute;\n    top: 9%;\n    left: 0;\n  }\n  p.info:before {\n    content: "i";\n    left: 3px;\n    width: 20px;\n    height: 20px;\n    font-family: serif;\n    font-size: 15px;\n    font-weight: bold;\n    line-height: 21px;\n    text-align: center;\n    color: #fff;\n    background: #4d5258;\n    border-radius: 16px;\n  }\n\n  @media (min-width: 768px) {\n    body {\n      margin: 4em 3em;\n    }\n    h1 {\n      font-size: 2.15em;}\n  }\n\n  </style>\n  </head>\n  <body>\n    <div>\n      <h1>Application is not available</h1>\n      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\n\n      <div class="alert alert-info">\n        <p class="info">\n          Possible reasons you are seeing this page:\n        </p>\n        <ul>\n          <li>\n            <strong>The host doesn\'t exist.</strong>\n            Make sure the hostname was typed correctly and that a route matching this hostname exists.\n          </li>\n          <li>\n            <strong>The host exists, but doesn\'t have a matching path.</strong>\n            Check if the URL path was typed correctly and that the route was created using the desired path.\n          </li>\n          <li>\n            <strong>Route and path matches, but all pods are down.</strong>\n            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\n          </li>\n        </ul>\n      </div>\n    </div>\n  </body>\n</html>\n', 'status': 503, 'headers': {'transfer-encoding': 'chunked', 'server': 'openresty/1.17.8.2', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'private, max-age=0, no-cache, no-store', 'date': 'Sat, 15 Aug 2020 20:44:41 GMT', 'content-type': 'text/html'}}
==========
 
That "all pods are down" seems wrong.
 
For record, he had:
 
# rpm -qa | grep -e subscription-manager -e rhsm -e redhat | sort
redhat-logos-70.7.0-1.el7.noarch
redhat-release-eula-7.8-0.el7.noarch
redhat-release-server-7.8-2.el7.x86_64
redhat-support-lib-python-0.12.1-1.el7.noarch
redhat-support-tool-0.12.2-1.el7.noarch
subscription-manager-1.24.26-1.el7.x86_64
subscription-manager-rhsm-1.24.26-1.el7.x86_64
subscription-manager-rhsm-certificates-1.24.26-1.el7.x86_64

Re-running the register command many times lead to a successful registration, after a while.

Created attachment 1711554 [details]
registration fails because of "all pods are down"

I think I was getting same issue over the weekend, but now I was able to register same host with same command just fine.

When I modify /usr/lib64/python2.7/site-packages/rhsm/connection.py like this to see the response:

# diff -u /usr/lib64/python2.7/site-packages/rhsm/connection.py{.ORIG,}
--- /usr/lib64/python2.7/site-packages/rhsm/connection.py.ORIG 2020-08-15 16:43:28.550936391 -0400
+++ /usr/lib64/python2.7/site-packages/rhsm/connection.py 2020-08-15 16:44:27.520160955 -0400
@@ -690,6 +690,9 @@
             else:
                 # This really needs an exception mapper too...
                 if str(response['status']) in ["404", "410", "500", "502", "503", "504"]:
+                    print("==========")
+                    print(response)
+                    print("==========")
                     raise RemoteServerException(response['status'],
                                                 request_type=request_type,
                                                 handler=handler)

I get this:

==========
{'content': u'<html>\n  <head>\n    <meta name="viewport" content="width=device-width, initial-scale=1">\n\n  <style type="text/css">\n  /*!\n   * Bootstrap v3.3.5 (http://getbootstrap.com)\n   * Copyright 2011-2015 Twitter, Inc.\n   * Licensed under MIT (https://github.com/twbs/bootstrap/blob/master/LICENSE)\n   */\n  /*! normalize.css v3.0.3 | MIT License | github.com/necolas/normalize.css */\n  html {\n    font-family: sans-serif;\n    -ms-text-size-adjust: 100%;\n    -webkit-text-size-adjust: 100%;\n  }\n  body {\n    margin: 0;\n  }\n  h1 {\n    font-size: 1.7em;\n    font-weight: 400;\n    line-height: 1.3;\n    margin: 0.68em 0;\n  }\n  * {\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n  }\n  *:before,\n  *:after {\n    -webkit-box-sizing: border-box;\n    -moz-box-sizing: border-box;\n    box-sizing: border-box;\n  }\n  html {\n    -webkit-tap-highlight-color: rgba(0, 0, 0, 0);\n  }\n  body {\n    font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;\n    line-height: 1.66666667;\n    font-size: 13px;\n    color: #333333;\n    background-color: #ffffff;\n    margin: 2em 1em;\n  }\n  p {\n    margin: 0 0 10px;\n    font-size: 13px;\n  }\n  .alert.alert-info {\n    padding: 15px;\n    margin-bottom: 20px;\n    border: 1px solid transparent;\n    background-color: #f5f5f5;\n    border-color: #8b8d8f;\n    color: #363636;\n    margin-top: 30px;\n  }\n  .alert p {\n    padding-left: 35px;\n  }\n  a {\n    color: #0088ce;\n  }\n\n  ul {\n    position: relative;\n    padding-left: 51px;\n  }\n  p.info {\n    position: relative;\n    font-size: 15px;\n    margin-bottom: 10px;\n  }\n  p.info:before, p.info:after {\n    content: "";\n    position: absolute;\n    top: 9%;\n    left: 0;\n  }\n  p.info:before {\n    content: "i";\n    left: 3px;\n    width: 20px;\n    height: 20px;\n    font-family: serif;\n    font-size: 15px;\n    font-weight: bold;\n    line-height: 21px;\n    text-align: center;\n    color: #fff;\n    background: #4d5258;\n    border-radius: 16px;\n  }\n\n  @media (min-width: 768px) {\n    body {\n      margin: 4em 3em;\n    }\n    h1 {\n      font-size: 2.15em;}\n  }\n\n  </style>\n  </head>\n  <body>\n    <div>\n      <h1>Application is not available</h1>\n      <p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>\n\n      <div class="alert alert-info">\n        <p class="info">\n          Possible reasons you are seeing this page:\n        </p>\n        <ul>\n          <li>\n            <strong>The host doesn\'t exist.</strong>\n            Make sure the hostname was typed correctly and that a route matching this hostname exists.\n          </li>\n          <li>\n            <strong>The host exists, but doesn\'t have a matching path.</strong>\n            Check if the URL path was typed correctly and that the route was created using the desired path.\n          </li>\n          <li>\n            <strong>Route and path matches, but all pods are down.</strong>\n            Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.\n          </li>\n        </ul>\n      </div>\n    </div>\n  </body>\n</html>\n', 'status': 503, 'headers': {'transfer-encoding': 'chunked', 'server': 'openresty/1.17.8.2', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'private, max-age=0, no-cache, no-store', 'date': 'Sat, 15 Aug 2020 20:44:41 GMT', 'content-type': 'text/html'}}
==========

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5452