Bug 1507030 - Subscription-manager returns "System certificates corrupted. Please reregister." when listing available subscriptions [NEEDINFO]
Summary: Subscription-manager returns "System certificates corrupted. Please reregiste...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: subscription-manager
Version: 7.4
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Alex Wood
QA Contact: Red Hat subscription-manager QE Team
URL:
Whiteboard:
Depends On:
Blocks: 1796188
TreeView+ depends on / blocked
 
Reported: 2017-10-27 13:02 UTC by Alexander Rydekull
Modified: 2021-06-10 13:22 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-29 19:22:40 UTC
Target Upstream Version:
awood: needinfo? (arydekul)


Attachments (Terms of Use)
rhsm.log from subscription attempts. (28.80 KB, text/plain)
2017-10-27 13:06 UTC, Alexander Rydekull
no flags Details
The /var/log/rhsm/rhsm.log file (1.78 MB, text/plain)
2018-01-12 14:33 UTC, Jaromir Hradilek
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github subscription-manager/pull/1759 0 None None None 2020-09-29 09:38:41 UTC
Red Hat Bugzilla 1528699 0 medium CLOSED "subscription-manager attach --auto" hangs due to 'rhel-7-server-rt-beta-rpms' enabled. 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1566147 0 medium CLOSED Untranslated "HTTP error (%s - %s)" or "HTTP error (%s - %s): %s" is prefixed to the STDERR in some cases 2021-02-22 00:41:40 UTC
Red Hat Product Errata RHBA-2020:3866 0 None None None 2020-09-29 19:22:58 UTC

Internal Links: 1528699 1566147

Description Alexander Rydekull 2017-10-27 13:02:39 UTC
Description of problem:
Subscription-manager returns "System certificates corrupted. Please reregister." when listing available subscriptions

This is discovered using the Employee SKU, it does feel like the issue could be because of the amount of subscriptions or something of that sort. The reason that im saying this is because as soon as you add "--matches" to narrow down the search the command works as expected.

Version-Release number of selected component (if applicable):
[root@ae90ab98aaaf /]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 2.0.41-1
subscription management rules: 5.26
subscription-manager: 1.19.23-1.el7_4
python-rhsm: 1.19.10-1.el7_4
[root@ae90ab98aaaf /]# 

How reproducible:
Every time

Steps to Reproduce:
1.subscription-manager register --username arydekul@redhat.com
2.subscription-manager list --available
3.failure

Actual results:

[rydekull@duneyr ~]$ docker run -it --rm rhel7
[root@ae90ab98aaaf /]# subscription-manager register --username arydekul@redhat.com
Registering to: subscription.rhsm.redhat.com:443/subscription
Password: 
The system has been registered with ID: 07c46fac-868d-4999-b2a8-a2d1a3d8ee57 
[root@ae90ab98aaaf /]# subscription-manager list --available
System certificates corrupted. Please reregister.
[root@ae90ab98aaaf /]# subscription-manager list --available --matches "*Employee SKU*" | wc -l
1277
[root@ae90ab98aaaf /]#

Expected results:
A long listing of available subscriptions as it is supposed to work :-)

Additional info:
Please fix that error message also, since its really not informative whatsoever.  Funny thing, its even pointed out in the unit tests how bad the error message is :-)

Comment 2 Alexander Rydekull 2017-10-27 13:06:47 UTC
Created attachment 1344304 [details]
rhsm.log from subscription attempts.

Comment 3 Jaromir Hradilek 2017-11-22 18:13:42 UTC
I am experiencing the very same issue, also with the Employee SKU, except for me, the subscription-manager also randomly throws in a complaint object serialization:

  [jhradilek@server ~]$ sudo subscription-manager register
  Registering to: subscription.rhsm.redhat.com:443/subscription
  Username: jhradile@redhat.com
  Password: 
  The system has been registered with ID: cc3629b3-ede3-481a-b269-ba63a501d6cb 
  [jhradilek@server ~]$ sudo subscription-manager list --available
  Unable to serialize objects to JSON.
  [jhradilek@server ~]$ sudo subscription-manager list --available
  System certificates corrupted. Please reregister.

That this is a fresh installation of RHEL 7.4 Server installed using the rhel-server-7.4-x86_64-dvd.iso image downloaded from the Customer Portal. Please note that the command I used is actually recommended by our official Product Documentation:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-subscription_and_support-registering_a_system_and_managing_subscriptions#sect-Subscription_and_Support-Registering_a_System_and_Managing_Subscriptions-Registering_the_System_and_Attaching_Subscriptions

I want to hope that our customers are not affected by this. Has anybody looked into this yet?

Alexander, thank you for your workaround! It didn't occur to me to try using --matches to limit the number of entries, and it indeed solves the problem.

Comment 4 Alex Wood 2018-01-09 19:43:47 UTC
Jaromir,

Are you running subscription-manager in a container?  From Alexander's comments it looks like he was running subscription-manager in a container which isn't really supported (since subscription-manager uses DBus and a Docker container won't have a DBus daemon running).

If you are running outside a container, would you mind reproducing the issue and then attaching the /var/log/rhsm/rhsm.log file?  I've tried to reproduce this issue with my own account that has an Employee SKU entitlement, but didn't encounter the problem.

Comment 5 Jaromir Hradilek 2018-01-12 14:31:56 UTC
Hi Alex,

Thank you for looking into this. I was not running subscription-manager in a container, but a fresh installation of RHEL 7.4 Server running in a virtual machine for testing purposes (in QEMU/KVM to be precise).

I tried to reproduce the error today in the same virtual machine and was prompted to reregister the system by using the following two commands:

  sudo subscription-manager clean
  sudo subscription-manager register

After doing so, I can get the complete list of available subscriptions without running into the error I reported. This didn't work in November. I am, however, still not able to attach the subscription I want, but that might be a different problem:

  [jhradilek@server ~]$ subscription-manager attach --pool 8a85f98260c27fc50160c323263339ff
  You are attempting to run "subscription-manager" which requires administrative
privileges, but more information is needed in order to do so.
  Authenticating as "root"
  Password:
  Runtime Error could not extract ResultSet at com.mysql.jdbc.SQLError.createSQLException:1,078

I am going to attach my /var/log/rhsm/rhsm.log file which includes records from both today and November.

Comment 6 Jaromir Hradilek 2018-01-12 14:33:27 UTC
Created attachment 1380434 [details]
The /var/log/rhsm/rhsm.log file

Comment 7 Alex Wood 2018-01-15 16:14:39 UTC
Jaromir,

For your problem in November, I see the following in the log file:

2017-11-21 20:42:00,320 [INFO] subscription-manager:14440:MainThread @connection.py:552 - Response: status=500, requestUuid=7f89cca6-78f6-443a-811f-89a38cd2ff2e, request="GET /subscription/owners/6340056/pools?consumer=f614237c-e4e4-4387-b00d-d9de22bc18dc"

That's a 500 on the server side.  No problem with subscription-manager per se.

Likewise, your most recent error corresponds to the entry

2018-01-12 15:06:40,380 [INFO] subscription-manager:4624:MainThread @connection.py:551 - Response: status=500, requestUuid=af44a6f7-76df-440a-95f9-b23024b8b371, request="POST /subscription/consumers/26192c87-971b-4184-877c-9dacff591e8f/entitlements?pool=8a85f98260c27fc50160c323263339ff"

I see two problems:

a) The error messages here are of very poor quality.  They don't indicate that the error is on the server rather than on the client.
b) There's a server error somewhere that's keeping you from getting your subscriptions.

I will go ahead and correct the first issue in this bug.  The second issue may require a little more investigation.

Comment 8 Alex Wood 2018-01-15 19:28:19 UTC
Jaromir,

Looking back on the log you provided, I can track that on the server-side to a "Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction" error.  We see these errors occasionally (especially with the employee SKU since it contains so many products) when systems are attempted to attach to a pool.  They're just an indication that there's a lot of contention for the resources of a specific pool.

Comment 9 Alex Wood 2018-01-15 19:29:55 UTC
Alexander,

While I've made some minor client corrections, I still haven't determined what was causing the issue you were seeing.  Are you running subscription-manager in a container?  If so, I'm afraid that's not supported.

Comment 11 John Sefler 2018-04-04 20:10:03 UTC
(In reply to Alex Wood from comment #7)
> I see two problems:
> 
> a) The error messages here are of very poor quality.  They don't indicate that the error is on the server rather than on the client.
> b) There's a server error somewhere that's keeping you from getting your subscriptions.
> 
> I will go ahead and correct the first issue in this bug.


Demonstrating a small example of the behavior change introduced by the improvement in https://github.com/subscription-manager/pull/1759


Here's a "before improvement" behavior for registering with invalid credentials...

[root@jsefler-rhel7 ~]# subscription-manager register --username=foo --password=bar --serverurl=subscription.rhsm.redhat.com:443/subscription
Registering to: subscription.rhsm.redhat.com:443/subscription
Invalid username or password. To create a login, please visit https://www.redhat.com/wapps/ugc/register.html


Here's an "after improvement" behavior for registering with invalid credentials...

[root@jsefler-rhel7 ~]# subscription-manager register --username=foo --password=bar --serverurl=subscription.rhsm.redhat.com:443/subscription
Registering to: subscription.rhsm.redhat.com:443/subscription
HTTP error (401 - Unauthorized): Invalid username or password. To create a login, please visit https://www.redhat.com/wapps/ugc/register.html


Notice the additional "HTTP error (401 - Unauthorized): " code information.

Comment 14 John Sefler 2020-04-16 20:29:58 UTC
I would like to move this bug to VERIFIED, but there are several details that have been raised in this bugzilla that should be put to rest....

_______________________________
First. Regarding "running subscription-manager in a container" as shown in comment 0.  I agree with comment 4 that subscription-manager should be disabled inside a container since the entitlements from the host are shared with the container making it unnecessary and unsupported to run subscription-manager from within a running container.  If you need access to the CDN from within a running container, you need to register the container's host system and attach subscriptions to the host that provide entitlements that you wish to utilize inside the running container.  Here is the expected response from subscription-manager from within a rhel7 container.


[root@hpe-dl380pgen8-02-vm-11 ~]# subscription-manager register --username=rhelentqe --auto-attach
Registering to: subscription.rhsm.redhat.com:443/subscription
Password: 
The system has been registered with ID: 6441bec2-2050-4d16-b780-9e11747f7744
The registered system name is: hpe-dl380pgen8-02-vm-11.hpe2.lab.eng.bos.redhat.com
Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@hpe-dl380pgen8-02-vm-11 ~]# docker run -i -t --rm registry.access.redhat.com/rhel7 /bin/bash 
Unable to find image 'registry.access.redhat.com/rhel7:latest' locally
Trying to pull repository registry.access.redhat.com/rhel7 ... 
latest: Pulling from registry.access.redhat.com/rhel7
ec0a4551131f: Pull complete 
448f7cafed66: Pull complete 
Digest: sha256:b0818ebc44a7e45a4c5c839a5b63282fcc6b0ad5f92ffe316a2306a3e84d0594
Status: Downloaded newer image for registry.access.redhat.com/rhel7:latest
[root@214c9dbeb1fb /]# 
[root@214c9dbeb1fb /]# subscription-manager list --available
subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.

[root@214c9dbeb1fb /]# 
[root@214c9dbeb1fb /]# yum repolist          
Loaded plugins: ovl, product-id, search-disabled-repos, subscription-manager
rhel-7-server-rpms                                                             | 3.5 kB  00:00:00     
(1/3): rhel-7-server-rpms/7Server/x86_64/group                                 | 631 kB  00:00:00     
(2/3): rhel-7-server-rpms/7Server/x86_64/updateinfo                            | 3.7 MB  00:00:01     
(3/3): rhel-7-server-rpms/7Server/x86_64/primary_db                            |  69 MB  00:00:05     
repo id                                     repo name                                           status
rhel-7-server-rpms/7Server/x86_64           Red Hat Enterprise Linux 7 Server (RPMs)            28756
repolist: 28756
[root@214c9dbeb1fb /]# 

VERIFIED: This is the expected behaviour when running subscription-manager inside the latest "rhel7" container.  Despite the fact that "subscription-manager is disabled when running inside a container", I can still access entitled content from the CDN through the attached entitlement shared from the host system.

_______________________________
Second: Regarding the attempt to list available subscriptions from an account that appears to have an awful lot of "*Employee SKU*" subscriptions, I don't think this is a realistic customer situation.  Moreover, the errors encountered in comment 0 and comment 3 appear to be the result of a server-side 500 error (as discussed in comment 7 and comment 8) which is not always reproducible but detection of it was improved with the changes demonstrated in comment 11.  I will try to reproduce the situation one more time from an account with several Employee SKUs...

[root@hpe-dl380pgen8-02-vm-11 ~]# subscription-manager register --username=qa@redhat.com
Registering to: subscription.rhsm.redhat.com:443/subscription
Password: 
The system has been registered with ID: 15fc78d0-15db-472f-895f-86bd6f91ec0b
The registered system name is: hpe-dl380pgen8-02-vm-11.hpe2.lab.eng.bos.redhat.com
[root@hpe-dl380pgen8-02-vm-11 ~]# 
[root@hpe-dl380pgen8-02-vm-11 ~]# subscription-manager list --available --matches "*Employee SKU*" | wc -l
301
[root@hpe-dl380pgen8-02-vm-11 ~]# subscription-manager list --available 1>/tmp/stdout
[root@hpe-dl380pgen8-02-vm-11 ~]# echo $?
0
[root@hpe-dl380pgen8-02-vm-11 ~]# 
[root@hpe-dl380pgen8-02-vm-11 ~]# grep corrupted /tmp/stdout
[root@hpe-dl380pgen8-02-vm-11 ~]# 

VERIFIED: Although my account has only 301 occurrences of "*Employee SKU*" in my list of available subscription (as opposed to 1277 in comment 0), I did not encounter a server-side error that manifested itself as "System certificates corrupted".  Worksforme.

_______________________________

Moving this bug to VERIFIED.  If "System certificates corrupted." continues to be a problem, please open a new bugzilla.

[root@hpe-dl380pgen8-02-vm-11 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 2.9.21-1
subscription management rules: 5.37
subscription-manager: 1.24.32

Comment 16 errata-xmlrpc 2020-09-29 19:22:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (subscription-manager bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3866


Note You need to log in before you can comment on or make changes to this bug.