Bug 1390464

Please attach the SAM logs & the rhsm logs from the time when this error was duplicated.

I retried on RHEL-6.9-20161102.n.0, but there is no rhsm.log due to bug1389559. So I just attached the katello-debug info(katello-debug-20161104043421.tar.gz) from SAM server:

[root@dhcp-128-11 ~]# subscription-manager register
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password: 
The system has been registered with ID: e8046ca5-da52-45de-8577-99a92d0235bf 
[root@dhcp-128-11 ~]# subscription-manager attach


Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-128-11 ~]# subscription-manager release --list
No release versions available, please check subscriptions.

Created attachment 1217324 [details]
katello-debug-20161104043421.tar.gz

Okay, please see attachments 'rhsm.log' and 'katello-debug-20161108043514.tar.gz'

Verification:
[root@dhcp-128-11 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 1.4.3.28-1.el6sam_splice-Headpin
subscription management rules: Unknown
subscription-manager: 1.18.3-1.el6
python-rhsm: 1.18.4-1.el6
[root@dhcp-128-11 ~]# rpm -Uvh subscription-manager-1.18.4-1.el6.x86_64.rpm subscription-manager-debuginfo-1.18.4-1.el6.x86_64.rpm subscription-manager-firstboot-1.18.4-1.el6.x86_64.rpm subscription-manager-gui-1.18.4-1.el6.x86_64.rpm
Preparing...                ########################################### [100%]
   1:subscription-manager   ########################################### [ 25%]
   2:subscription-manager-gu########################################### [ 50%]
   3:subscription-manager-fi########################################### [ 75%]
   4:subscription-manager-de########################################### [100%]
[root@dhcp-128-11 ~]# subscription-manager version
server type: This system is currently not registered.
subscription management server: 1.4.3.28-1.el6sam_splice-Headpin
subscription management rules: Unknown
subscription-manager: 1.18.4-1.el6
python-rhsm: 1.18.4-1.el6
[root@dhcp-128-11 ~]# rm -f /var/log/rhsm/rhsm.log 
[root@dhcp-128-11 ~]# subscription-manager register --auto-attach
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password: 
The system has been registered with ID: bb913202-3cca-4d19-9faf-6e7b0cfdbe06 

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-128-11 ~]# subscription-manager release --list
No release versions available, please check subscriptions.

Created attachment 1218467 [details]
rhsm.log

Created attachment 1218468 [details]
katello-debug-20161108043514.tar.gz

1. Today, I installed a RHEL6.8 to test the scenario against SAM. And RHEL6.8 works well against SAM1.4:

[root@dhcp-129-159 ~]# subscription-manager register --auto-attach
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password:
The system has been registered with ID: f6785623-f2ae-49da-9fd3-cf9d889ccd70

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-129-159 ~]# subscription-manager release --list
+-------------------------------------------+
          Available Releases
+-------------------------------------------+
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6Server
[root@dhcp-129-159 ~]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 1.4.3.28-1.el6sam_splice-Headpin
subscription management rules: Unknown
subscription-manager: 1.16.8-8.el6
python-rhsm: 1.16.6-1.el6


2. Bug 1256615 exists both against satellite6.1 and satellite6.2. However, RHEL6.9 works well against Stage candlepin.
1) The following is RHEL-6.9-20161104.n.0 against satellite6.2:

[root@dhcp-129-1 ~]# subscription-manager register
Registering to: ent-02-vm-05.lab.eng.nay.redhat.com:443/rhsm
Username: admin
Password:
The system has been registered with ID: 7e736fa9-8096-4f24-a9af-a0d120428f0b
[root@dhcp-129-1 ~]# subscription-manager attach


Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-129-1 ~]# subscription-manager release --list
No release versions available, please check subscriptions.


2) I tried against stage, and the issue can not be reproduced:

[root@dhcp-128-11 product-default]# subscription-manager register
Registering to: subscription.rhsm.stage.redhat.com:443/subscription
Username: new_test
Password:
The system has been registered with ID: 4747ff3b-137b-44f5-94fe-78670fb1498f
[root@dhcp-128-11 product-default]# subscription-manager attach

All installed products are covered by valid entitlements. No need to update subscriptions at this time.

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux for SAP
Status:       Subscribed

Product Name: Red Hat Developer Toolset (for RHEL Server)
Status:       Subscribed

Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-128-11 product-default]# subscription-manager release --list
+-------------------------------------------+
          Available Releases
+-------------------------------------------+
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6Server
[root@dhcp-128-11 product-default]# subscription-manager version
server type: Red Hat Subscription Management
subscription management server: 0.9.51.20-1
subscription management rules: 5.15.1
subscription-manager: 1.18.4-1.el6
python-rhsm: 1.18.4-1.el6

This issue was reproduced on build RHEL-6.9-20161201.0 against SAM,and it works well against stage.so make this issue reopened.

[root@dhcp-128-6 ~]# subscription-manager register
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password: 
The system has been registered with ID: 35ca6806-b030-4e82-a801-23c2c29b71c7 
[root@dhcp-128-6 ~]# subscription-manager release --list
No release versions available, please check subscriptions.

Wei,

Can you please provide the following:
 - SAM version
 - rhsm.log
?

Created attachment 1229298 [details]
the latest rhsm.log

The SAM version is SAM-1.4.1-RHEL-6-20141113.0 and rhsm.log please see the attachments 'Rhsm.log'.
By the way if you need a SAM instance, I can provide one. Please feel free to let me know.

Wei Liu,

I would like to take you up on your offer of a SAM instance for reproducing this bug. I will continue work on this bug once I have access to a SAM instance.

Thank you very much for you assistance on this issue.

Cheers,

Chris

Hi Chris

I do some investigation referred to your comment, and when cat the entitlement certificates,the required tag matches "rhel-6-*|rhel-5-workstation$"

And as the Comment 11, it is OK on rhel 6.8,and I try on 6.8 with the same account，SKU and SAM instance, the release version can be listed successfully.
So it is really a regression issue.

Thank you 
weiliu

Hi,

After trying once more I am unable to locally reproduce this error as it is in the logs.

I am able to get subscription-manager to fail to list releases but believe this to be due to my system not being connected to the content proxy provided by SAM.
For your SAM instance is thumbslug running?

I've thoroughly checked the code base and verified the functionality of what subman is doing here. It has not changed in a very long time. I do not believe this to be a regression.

For me to further debug and investigate I'll need access the the reproducer system itself.


Thank you,
Chris

Given the inability to reproduce and lack of response on how to reproduce I am closing this issue as we do not have enough information to reproduce.

weiliu, please re-open if you can provide the information required to reproduce this issue.

(In reply to Chris Snyder from comment #25)
> Hi,
> 
> After trying once more I am unable to locally reproduce this error as it is
> in the logs.
> 
> I am able to get subscription-manager to fail to list releases but believe
> this to be due to my system not being connected to the content proxy
> provided by SAM.
> For your SAM instance is thumbslug running?
> 
> I've thoroughly checked the code base and verified the functionality of what
> subman is doing here. It has not changed in a very long time. I do not
> believe this to be a regression.
> 
> For me to further debug and investigate I'll need access the the reproducer
> system itself.
> 
> 
> Thank you,
> Chris

I tried on RHEL-6.9-20170202.0 against SAM-1.4.1-RHEL-6-20141113.0, and the bug can be reproduced. what's more the thumbslug is running:

1. 
[root@dhcp-129-99 ~]# subscription-manager register --auto-attach
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password: 
The system has been registered with ID: e0070b1c-6d9b-4726-9c2c-62226d117270 

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

[root@dhcp-129-99 ~]# subscription-manager release --list
No release versions available, please check subscriptions.

2.
[root@samserv ~]# katello-service status
tomcat6 (pid 1970) is running...                           [  OK  ]
httpd (pid  2017) is running...
thumbslug (pid  2189) is running...
elasticsearch (pid  1788) is running...
katello (2088) is running.
katello (2104) is running.
delayed_job is running.
delayed_job_monitor is running.

Hope this can help you, so reopen the bug.

I have once more recreated my reproducer system. Once more the release --list command works for me.

Here are my reproduction steps:
1) Step sam instance following: https://mojo.redhat.com/docs/DOC-1098333

2) Create a new vm installed from http://download.devel.redhat.com/rel-eng/RHEL-6.9-20170202.0/compose/Server/x86_64/os/ (Called this system rhel6.9_Latest)

3) Created an account on stage and gave the account some RHEL subscriptions (used http://account-manager-stage.app.eng.rdu2.redhat.com/#create to do that)

4) Logged into the stage portal with the new account and register a new subscription asset manager application.

5) Associated the RHEL subscriptions to the new subscription asset manager application in stage portal account.

6) Downloaded the manifest for the SAM instance.

7) Imported the manifest to SAM using the webui to the ACME_Corporation

8) Executed the following (found in the SAM webui) on the fresh rhel6.9_Latest system: rpm -Uvh http://192.168.121.146/pub/candlepin-cert-consumer-latest.noarch.rpm (NOTE: this updates the rhsm.conf values for server.hostname, rhsm.baseurl)

9) Add the following to the file /etc/rhsm/facts/custom.facts (the version of candlepin in SAM-1.4 seems not to be able to handle facts longer than 255 chars, this fact was so I updated it as follows, I believe this issue is unrelated to this bz):

'{"proc_cpuinfo.common.flags": "THIS_WAS_TOO_LONG_FOR_CP_TO_HANDLE"}'

10) Run the following on rhel6.9_Latest 'subscription-manager register --org="ACME_Corporation" --user admin --pass admin --auto-attach

[root@rhel6 ~]# subscription-manager register --org="ACME_Corporation" --auto-attach
Registering to: samserv.redhat.com:443/sam/api
Username: admin
Password:
The system has been registered with ID: ff009ec9-09f6-4d2b-89b5-ca26c3a74913

Installed Product Current Status:
Product Name: Red Hat Enterprise Linux Server
Status:       Subscribed

11) Run 'subscription-manager list --consumed' to see what we have be given by auto-attach

[root@rhel6 ~]# subscription-manager list --consumed
+-------------------------------------------+
   Consumed Subscriptions
+-------------------------------------------+
Subscription Name:   Red Hat Enterprise Linux with Smart Virtualization, Standard (2-socket)
Provides:            Oracle Java (for RHEL Server)
                     Red Hat JBoss Core Services
                     JBoss Enterprise Application Platform
                     Red Hat Enterprise Linux Atomic Host
                     Red Hat Enterprise Linux Server
                     Red Hat Beta
                     dotNET on RHEL Beta (for RHEL Server)
                     Red Hat Software Collections (for RHEL Server)
                     dotNET on RHEL (for RHEL Server)
                     Red Hat Enterprise Linux Atomic Host Beta
                     Red Hat Software Collections Beta (for RHEL Server)
                     Red Hat Enterprise Linux Fast Datapath
                     Red Hat Virtualization Host
                     Red Hat Virtualization
SKU:                 MCT2930
Contract:            11252134
Account:             5750118
Serial:              5361415144995827408
Pool ID:             4028f91259bcf2000159bcf2a347004a
Provides Management: No
Active:              True
Quantity Used:       1
Service Level:       Standard
Service Type:        L1-L3
Status Details:      Subscription is current
Subscription Type:   Stackable
Starts:              01/18/2017
Ends:                01/17/2018
System Type:         Physical

12) Run 'subscription-manager release --list'

[root@rhel6 ~]# subscription-manager release --list
+-------------------------------------------+
          Available Releases
+-------------------------------------------+
6.1
6.2
6.3
6.4
6.5
6.6
6.7
6.8
6Server


Given the above I do not believe this is an issue in current subscription-manager.

Please let me know how your reproduction steps differ from the above and/or provide access to the systems that are able to reproduce the failure for further debugging and investigation.

(In reply to Chris Snyder from comment #29)
> I have once more recreated my reproducer system. Once more the release
> --list command works for me.
> 
> Here are my reproduction steps:
> 1) Step sam instance following: https://mojo.redhat.com/docs/DOC-1098333
> 
> 2) Create a new vm installed from
> http://download.devel.redhat.com/rel-eng/RHEL-6.9-20170202.0/compose/Server/
> x86_64/os/ (Called this system rhel6.9_Latest)
> 
> 3) Created an account on stage and gave the account some RHEL subscriptions
> (used http://account-manager-stage.app.eng.rdu2.redhat.com/#create to do
> that)
> 
> 4) Logged into the stage portal with the new account and register a new
> subscription asset manager application.
> 
> 5) Associated the RHEL subscriptions to the new subscription asset manager
> application in stage portal account.
> 
> 6) Downloaded the manifest for the SAM instance.
> 
> 7) Imported the manifest to SAM using the webui to the ACME_Corporation
> 
> 8) Executed the following (found in the SAM webui) on the fresh
> rhel6.9_Latest system: rpm -Uvh
> http://192.168.121.146/pub/candlepin-cert-consumer-latest.noarch.rpm (NOTE:
> this updates the rhsm.conf values for server.hostname, rhsm.baseurl)
> 
> 9) Add the following to the file /etc/rhsm/facts/custom.facts (the version
> of candlepin in SAM-1.4 seems not to be able to handle facts longer than 255
> chars, this fact was so I updated it as follows, I believe this issue is
> unrelated to this bz):
> 
> '{"proc_cpuinfo.common.flags": "THIS_WAS_TOO_LONG_FOR_CP_TO_HANDLE"}'
> 
> 10) Run the following on rhel6.9_Latest 'subscription-manager register
> --org="ACME_Corporation" --user admin --pass admin --auto-attach
> 
> [root@rhel6 ~]# subscription-manager register --org="ACME_Corporation"
> --auto-attach
> Registering to: samserv.redhat.com:443/sam/api
> Username: admin
> Password:
> The system has been registered with ID: ff009ec9-09f6-4d2b-89b5-ca26c3a74913
> 
> Installed Product Current Status:
> Product Name: Red Hat Enterprise Linux Server
> Status:       Subscribed
> 
> 11) Run 'subscription-manager list --consumed' to see what we have be given
> by auto-attach
> 
> [root@rhel6 ~]# subscription-manager list --consumed
> +-------------------------------------------+
>    Consumed Subscriptions
> +-------------------------------------------+
> Subscription Name:   Red Hat Enterprise Linux with Smart Virtualization,
> Standard (2-socket)
> Provides:            Oracle Java (for RHEL Server)
>                      Red Hat JBoss Core Services
>                      JBoss Enterprise Application Platform
>                      Red Hat Enterprise Linux Atomic Host
>                      Red Hat Enterprise Linux Server
>                      Red Hat Beta
>                      dotNET on RHEL Beta (for RHEL Server)
>                      Red Hat Software Collections (for RHEL Server)
>                      dotNET on RHEL (for RHEL Server)
>                      Red Hat Enterprise Linux Atomic Host Beta
>                      Red Hat Software Collections Beta (for RHEL Server)
>                      Red Hat Enterprise Linux Fast Datapath
>                      Red Hat Virtualization Host
>                      Red Hat Virtualization
> SKU:                 MCT2930
> Contract:            11252134
> Account:             5750118
> Serial:              5361415144995827408
> Pool ID:             4028f91259bcf2000159bcf2a347004a
> Provides Management: No
> Active:              True
> Quantity Used:       1
> Service Level:       Standard
> Service Type:        L1-L3
> Status Details:      Subscription is current
> Subscription Type:   Stackable
> Starts:              01/18/2017
> Ends:                01/17/2018
> System Type:         Physical
> 
> 12) Run 'subscription-manager release --list'
> 
> [root@rhel6 ~]# subscription-manager release --list
> +-------------------------------------------+
>           Available Releases
> +-------------------------------------------+
> 6.1
> 6.2
> 6.3
> 6.4
> 6.5
> 6.6
> 6.7
> 6.8
> 6Server
> 
> 
> Given the above I do not believe this is an issue in current
> subscription-manager.
> 
> Please let me know how your reproduction steps differ from the above and/or
> provide access to the systems that are able to reproduce the failure for
> further debugging and investigation.

Hi, Chris

I have sent a mail to you. Please check the SAM instance and RHEL6.9. Hope this can help you.

I have taken a look at the systems used for reproduction.
The short answer (for those not interested in the details) is this is not a regression in subscription-manager or even a change in subscription-manager behaviour and as such I am closing this bug.


On to the details:

Subscription-manager/python-rhsm on RHEL 6 uses the m2crypto library for ssl communication. M2crypto, eventually, utilizes methods provided by openssl (using whatever version is installed already on the system) to do it's communication.

As a result of the logjam attack[1] (among others) openssl increased the minimum length of the keys used in diffie-hellman ciphersuites to 1024 (see [1] below). The version of openssl included in RHEL 6.9 includes this new requirement. When a key is provided that does not meet the minimum length openssl stops the communication (more or less) immediately.

In order to deal with logjam similar changes were made to Java (minimum key size was increased to 1024 and is now configurable). It seems that the change of the minimum key size was pulled into java-1.7.0-openjdk-1.7.0.99-2.6.5.1.el6.x86_64.
It may have been pulled in to slightly older versions as well but the point is the change of minimum key size is not in java-1.7.0-openjdk-1.7.0.65-2.5.1.2.el6_5.x86_64 (the version installed on the reproducer SAM instance provided to me).

As stated before the calls that subscription-manager makes to provide a listing of the available releases (in the SAM case) are directed to thumbslug on the SAM instance and passed on to the cdn.

Looking in the thumbslug logs (/var/log/thumbslug/error.log) you can see instances of the following (written immediately upon running subscription-manager release --list on the rhel 6.9 system):

"Feb 09 13:43:45 [pool-2-thread-8] ERROR org.candlepin.thumbslug.HttpRequestHandler - Exception caught!
javax.net.ssl.SSLException: Received fatal alert: handshake_failure [...]"

To check the key size returned by thumbslug I did the following (from the rhel6.9 reproducer box):
"""

[root@dhcp-129-99 ~]# openssl s_client -connect samserv.redhat.com:8088 -cipher "EDH" | grep "Server Temp
 Key"
depth=0 C = US, ST = North Carolina, L = Raleigh, O = SomeOrg, OU = SomeOrgUnit, CN = samserv.redhat.com
verify error:num=18:self signed certificate
verify return:1
depth=0 C = US, ST = North Carolina, L = Raleigh, O = SomeOrg, OU = SomeOrgUnit, CN = samserv.redhat.com
verify return:1
140006684985160:error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3345:
Server Temp Key: DH, 768 bits  # <========== BOOM
[root@dhcp-129-99 ~]#
"""

As can be see above on the line marked with BOOM, the key returned from thumbslug is of length 768.
Also note on the line above the marked one, you can find the phrase "dh key too small".
This is the same message that is wrapped in an SSLError and raised in subscription-manager.

Please note the above command /does not use any subscription-manager or python-rhsm code/.

Because a key of length 768 is less than the minimum expected in the version of openssl being used in rhel 6.9,
the connection is aborted and no information (about releases) can be retrieved by subscription manager.

Starting with a RHEL 6.8 box with SAM configured in the same way as the reproducer everything works fine.
Downgrading the version of the java openjdk installed on said system can cause the failure as described here.
Upgrading the package again (on the same system) causes the failure to go away.

Installing the newer java openjdk onto the RHEL 6.6 reproducer does not fix the issue.
I think this might have something to do with certs that are already present on the system (created during SAM configuration).

Next steps to try:

- Upgrade the rhel 6.6 SAM instance to rhel 6.8
- Further isolate the involved dependencies.


I have also tried installing an older version of openssl on an otherwise fresh install of RHEL 6.9. Doing this causes the subscription-manager release --list command to succeed.

Given all of the above I am confident in saying this is not a bug in subscription-manager or python-rhsm. Hence I am closing this bug.

[1]: https://www.openssl.org/blog/blog/2015/05/20/logjam-freak-upcoming-changes/