Bug 1010082

Summary: Compatibility issue with java-1.7.0-openjdk
Product: [Retired] Subscription Asset Manager Reporter: Og Maciel <omaciel>
Component: katello-configureAssignee: Devan Goodwin <dgoodwin>
Status: CLOSED ERRATA QA Contact: sthirugn <sthirugn>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.4CC: bkearney, cbillett, cduryee, dgoodwin, sthirugn
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1010111 (view as bug list) Environment:
Last Closed: 2014-06-10 12:13:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 971511, 1010111    

Description Og Maciel 2013-09-19 22:13:29 UTC
Description of problem:

Seems that latest SAM code has compatibility issue if java-1.7.0-openjdk is already installed in the system. After successfully installing katello-headpin-all and running katello-configuration, katello-jobs fails to start

Version-Release number of selected component (if applicable):

* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.8.26-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.26-1.el6sam.noarch
* candlepin-tomcat6-0.8.26-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.2-2.el6sat.noarch
* katello-cli-1.4.3-11.el6sat.noarch
* katello-cli-common-1.4.3-11.el6sat.noarch
* katello-common-1.4.3-14.el6sam_splice.noarch
* katello-configure-1.4.4-4.el6sat.noarch
* katello-glue-candlepin-1.4.3-14.el6sam_splice.noarch
* katello-glue-elasticsearch-1.4.3-14.el6sam_splice.noarch
* katello-headpin-1.4.3-14.el6sam_splice.noarch
* katello-headpin-all-1.4.3-14.el6sam_splice.noarch
* Katello-Katello-Sanity-ImportKeys-1.2-1.noarch
* katello-selinux-1.4.4-2.el6sat.noarch
* openldap-2.4.23-32.el6_4.1.x86_64
* openldap-devel-2.4.23-32.el6_4.1.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-2.el6sat.noarch
* thumbslug-0.0.34-1.el6sam.noarch
* thumbslug-selinux-0.0.34-1.el6sam.noarch

How reproducible:


Steps to Reproduce:
1. Provision a new RHEL system
2. Install java-1.7.0-openjdk
3. Install and configure katello-headpin-all

Actual results:

katello-jobs fails to start

Expected results:


Additional info:

[root@ibm-x3550m3-11 ~]# yum install -y java-1.7.0-openjdk
[root@ibm-x3550m3-11 ~]# rpm -qa | grep openjdk
java-1.7.0-openjdk-1.7.0.40-2.4.2.1.el6.x86_64
[root@ibm-x3550m3-11 ~]# katello-service status
tomcat6 (pid 19716) is running...[  OK  ]
httpd (pid  19834) is running...
thumbslug (pid  19883) is running...
elasticsearch (pid  19926) is running...
katello (20077) is running.
katello (20096) is running.
katello (20115) is running.
katello (20134) is running.
katello (20153) is running.
katello (20172) is running.
katello (20191) is running.
katello (20210) is running.
katello (20237) is running.
katello (20264) is running.
katello (20291) is running.
katello (20318) is running.
katello (20350) is running.
katello (20379) is running.
katello (20407) is running.
katello (20435) is running.
katello (20464) is running.
katello (20495) is running.
katello (20518) is running.
katello (20548) is running.
katello (20583) is running.
katello (20621) is running.
katello (20673) is running.
katello (20700) is running.
katello (20736) is running.
delayed_job is not running.
delayed_job_monitor is not running.
[root@ibm-x3550m3-11 ~]# headpin -u admin -p admin ping
------------------------------------------------------------------------------------------------------------------------------------------------
                                                                 Katello Status

Status Service        Result Duration Message
------------------------------------------------------------------------------------------------------------------------------------------------
FAIL
candlepin      FAIL   SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error
candlepin_auth FAIL   SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: tlsv1 alert internal error
elasticsearch  ok     38ms
katello_jobs   FAIL   katello-jobs service not running
thumbslug      ok     148ms
[root@ibm-x3550m3-11 ~]# rpm -qa | grep openjdk
java-1.7.0-openjdk-1.7.0.40-2.4.2.1.el6.x86_64

[root@ibm-x3550m3-11 ~]# yum install -y java-1.6.0-openjdk
[root@ibm-x3550m3-11 ~]# katello-service stop
[root@ibm-x3550m3-11 ~]# katello-configure --deployment=sam --user-pass=admin
root@ibm-x3550m3-11 ~]# headpin -u admin -p admin ping
Service unavailable or restarting, try later
[root@ibm-x3550m3-11 ~]# katello-service status
tomcat6 (pid 22970) is running...[  OK  ]
httpd (pid  23288) is running...
thumbslug (pid  23176) is running...
elasticsearch (pid  22721) is running...
katello is not running.
delayed_job is not running.
delayed_job_monitor is not running.
[root@ibm-x3550m3-11 ~]# rpm -qa | grep openjdk
java-1.7.0-openjdk-1.7.0.40-2.4.2.1.el6.x86_64
java-1.6.0-openjdk-1.6.0.0-1.62.1.11.11.90.el6_4.x86_64
[root@ibm-x3550m3-11 ~]# rpm -e java-1.7.0-openjdk.x86_64
[root@ibm-x3550m3-11 ~]# rpm -qa | grep openjdk
java-1.6.0-openjdk-1.6.0.0-1.62.1.11.11.90.el6_4.x86_64
[root@ibm-x3550m3-11 ~]# katello-service stop
[root@ibm-x3550m3-11 ~]# katello-configure --deployment=sam --user-pass=admin
[root@ibm-x3550m3-11 ~]# headpin -u admin -p admin ping
------------------------------------------------------------------------------------------------------------------------------------------------
                                                                 Katello Status

Status Service        Result Duration Message
------------------------------------------------------------------------------------------------------------------------------------------------
ok
candlepin      ok     50ms
candlepin_auth ok     33ms
elasticsearch  ok     27ms
katello_jobs   ok     34ms
thumbslug      ok     212ms
[root@ibm-x3550m3-11 ~]# katello-service status
tomcat6 (pid 25090) is running...[  OK  ]
httpd (pid  26165) is running...
thumbslug (pid  25295) is running...
elasticsearch (pid  24843) is running...
katello (25482) is running.
katello (25501) is running.
katello (25520) is running.
katello (25539) is running.
katello (25558) is running.
katello (25577) is running.
katello (25596) is running.
katello (25615) is running.
katello (25642) is running.
katello (25669) is running.
katello (25696) is running.
katello (25723) is running.
katello (25750) is running.
katello (25785) is running.
katello (25804) is running.
katello (25840) is running.
katello (25860) is running.
katello (25891) is running.
katello (25912) is running.
katello (25949) is running.
katello (25976) is running.
katello (26006) is running.
katello (26051) is running.
katello (26080) is running.
katello (26121) is running.
delayed_job is running.
delayed_job_monitor is running.

Comment 2 Devan Goodwin 2014-03-05 20:02:21 UTC
If anyone can reproduce this, could we get the "security.provider.*" lines from java.security file for the JDK on that system?

Comment 3 Devan Goodwin 2014-03-24 16:59:15 UTC
Ok Chris Duryee hit this and got in touch with me. His java.security featured:

security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/nss.cfg

Which is the FIPS compliant JCE provider, a new default in some JDK rpms in RHEL.

This FIPS compliant provider provides an implementation for several anonymous ECDH ciphers you will see in /etc/tomcat6/server.xml:

TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_anon_WITH_AES_128_CBC_SHA
TLS_ECDH_anon_WITH_AES_256_CBC_SHA

However the stock JDK JCE provider does *not* provide an implementation for these:

http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SUNProvider

And critically, these ciphers should not be used and cannot work with Candlepin. See https://github.com/candlepin/candlepin/commit/be8c72995e29912319bb6e622b8e648e3cbab411 for more details.

TLDR Candlepin has been shipping config that included some ciphers which cannot work with the application. The problem was hidden because those ciphers are not provided by the default JCE provider. The SunPKCS11 however does implement them, and recent JDK's have been switching to this as the default, exposing the problem.

To work around, edit your server.xml and remove the anon ciphers above.

Cipher config should be corrected as of candlepin-0.9.5-1

Comment 8 sthirugn@redhat.com 2014-04-22 16:29:29 UTC
Verified.

Steps:
1. Provisioned rhel 6.5
2. Removed older openjdk and installed java-1.7.0-openjdk:
yum install -y java-1.7.0-openjdk
3. Install SAM
4. Run katello-configure

Result:
# rpm -qa | grep openjdk
java-1.7.0-openjdk-1.7.0.55-2.4.7.1.el6_5.x86_64

# katello-service status
tomcat6 (pid 18909) is running...[  OK  ]
httpd (pid  19371) is running...
thumbslug (pid  19119) is running...
elasticsearch (pid  18333) is running...
katello (19307) is running.
katello (19324) is running.
katello (19341) is running.
delayed_job is running.
delayed_job_monitor is running.

Version tested:
* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.9.6-1.el6_5.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.9.6-1.el6_5.noarch
* candlepin-tomcat6-0.9.6-1.el6_5.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.2-2.el6sat.noarch
* katello-cli-1.4.3.1-1.el6sam.noarch
* katello-cli-common-1.4.3.1-1.el6sam.noarch
* katello-common-1.4.3.26-1.el6sam_splice.noarch
* katello-configure-1.4.5-1.el6sam.noarch
* katello-glue-candlepin-1.4.3.26-1.el6sam_splice.noarch
* katello-glue-elasticsearch-1.4.3.26-1.el6sam_splice.noarch
* katello-headpin-1.4.3.26-1.el6sam_splice.noarch
* katello-headpin-all-1.4.3.26-1.el6sam_splice.noarch
* katello-selinux-1.4.4-2.el6sat.noarch
* openldap-2.4.23-32.el6_4.1.x86_64
* openldap-devel-2.4.23-32.el6_4.1.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-1.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-2.el6sat.noarch
* thumbslug-0.0.39-1.el6sam.noarch
* thumbslug-selinux-0.0.39-1.el6sam.noarch

Comment 10 errata-xmlrpc 2014-06-10 12:13:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0677.html