Bug 1302761 - Satellite 6.1.6 cannot install atop 6.8 composes due to candlepin failures - apparent tomcat permission errors.
Satellite 6.1.6 cannot install atop 6.8 composes due to candlepin failures - ...
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: tomcat6 (Show other bugs)
Unspecified Unspecified
high Severity high
: beta
: ---
Assigned To: Coty Sutherland
Depends On: 1084426
  Show dependency treegraph
Reported: 2016-01-28 09:57 EST by Corey Welton
Modified: 2016-02-09 10:51 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2016-02-09 10:51:31 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Corey Welton 2016-01-28 09:57:54 EST
Description of problem:
When trying to install satellite6 atop recent composes of 6.8, the install fails due to candlepin not being able to start.  This, in turn, is due to some permissions issue with tomcat, which keeps candlepin from being able to access required certs.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1.  Attempt to install satellite 6.1.6 atop $compose:
   katello-installer -v -d --foreman-admin-password=changeme
2.  Note failure to install and view resulting foreman logfiles, indicating a candlepin error.

Actual results:
Install fails

Expected results:
Install should be successful.

Additional info:
I believe the issue has come up per one of the recent commits to tomcat's permission/security model

 rpm -q --changelog tomcat6
Comment 2 Coty Sutherland 2016-01-28 10:10:02 EST
Can you provide any more information regarding the failure? Assume I know nothing about Satellite. A stack trace, error message, or exact steps to reproduce would be great so I can look into this further.

As far as a possible cause for the issue...the only file permission change was made in https://bugzilla.redhat.com/show_bug.cgi?id=886653. There was also a process ownership change made in https://bugzilla.redhat.com/show_bug.cgi?id=1084426 also. Can you reverse the changes of either of those before you install Satellite and see if that resolves the issue?
Comment 4 Coty Sutherland 2016-02-04 09:32:51 EST
Still need more info. I thought I set needinfo before, but I guess not.
Comment 5 Chris Duryee 2016-02-08 12:31:55 EST
Satellite 6 uses tomcat6, and lays down a couple of private key certs that are used by candlepin (a webapp that runs inside tomcat). For example:

# ll /etc/pki/katello/private/katello-default-ca.*
-r--r-----. 1 root foreman 1675 Feb  8 10:20 /etc/pki/katello/private/katello-default-ca.key
-r--------. 1 root root      24 Feb  8 10:20 /etc/pki/katello/private/katello-default-ca.pwd

On RHEL 6.7 and older, candlepin is able to read these files. However, upon upgrading to tomcat6-6.0.24-92.el6.noarch, candlepin is no longer able to access these. This causes tomcat (and thus satellite) to not start.

The exact error in /var/log/candlepin/candlepin.log is:

Caused by: java.io.FileNotFoundException: /etc/pki/katello/private/katello-default-ca.key (Permission denied)
Comment 6 Chris Duryee 2016-02-08 15:00:24 EST
more info:

# ll -Z /etc/pki/katello/private/katello-default-ca.*
-r--r-----. root foreman system_u:object_r:cert_t:s0      /etc/pki/katello/private/katello-default-ca.key
-r--------. root root    system_u:object_r:cert_t:s0      /etc/pki/katello/private/katello-default-ca.pwd

additionally, 'audit2allow -a' does not show any denials.

The tomcat user is able to read the file:

# sudo su - tomcat -s /bin/bash
-bash-4.1$ cat /etc/pki/katello/private/katello-default-ca.key
Comment 7 Coty Sutherland 2016-02-08 15:07:58 EST
It looks like https://bugzilla.redhat.com/show_bug.cgi?id=1084426 is at fault from some testing (process of elimination). Not sure why though...it shouldn't be doing anything by default other than adding the ability to actually control the processes group owner. The default group comes from the tomcat user unless it's set elsewhere:

# Define the tomcat group
Comment 8 Chris Duryee 2016-02-08 15:13:25 EST
Is there a way to back out 1084426 easily on an existing machine to see if it helps?
Comment 9 Coty Sutherland 2016-02-08 15:21:24 EST
I am actually reproducing the issue on a VM using build 90 and the change from that bug manually applied to the init script. 

I'm retesting now with TOMCAT_GROUP=foreman in the tomcat6.conf to see if that is the issue given the file permissions are 440. Though that still doesn't explain how the behavior should have changed unless the installer was somehow setting the group differently before the patch and for some reason can't accomplish the same thing afterwards. If that doesn't resolve the issue (though I'm not even sure that is a feasible solution) I am open to dropping the bug from the release. The customer case has been long since closed and nobody (that I can see) is requesting the fix at this point.
Comment 10 Coty Sutherland 2016-02-08 20:24:54 EST
With the fix in place and using TOMCAT_GROUP=foreman, the following occurs:

Feb 08, 2016 3:15:21 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory candlepin
Feb 08, 2016 3:15:21 PM org.apache.catalina.startup.HostConfig deployDirectory
SEVERE: Error deploying web application directory candlepin
java.io.FileNotFoundException: /etc/tomcat6/Catalina/localhost/candlepin.xml (Permission denied)
        at java.io.FileOutputStream.open0(Native Method)
        at java.io.FileOutputStream.open(FileOutputStream.java:270)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1013)

because the process owner isn't in the tomcat group anymore and therefore can't write to the root:tomcat owned directories. Directory listing of the conf directory is as follows:

# ll /etc/tomcat6/ -R
total 100
drwxrwxr-x. 3 root   tomcat   4096 May 15  2015 Catalina
-rw-rw-r--. 1 root   root     8945 May 15  2015 catalina.policy

total 4
drwxrwxr-x. 2 root tomcat 4096 May 15  2015 localhost

total 0

After updating the permissions on this directory to tomcat user ownership and also updating the /var/log/tomcat6 directory to tomcat ownership things work as expected. That would mean that we need to make further updates to the permissions of the tomcat6 package to get this working correctly. Additionally, the upstream doesn't take into account the group process permissions, so I think that it's safe to revert the change for rhbz-1084426.
Comment 11 Coty Sutherland 2016-02-09 10:51:31 EST
I reverted https://bugzilla.redhat.com/show_bug.cgi?id=1084426, rebuilt (build 95), and retested to verify that it works. I've closed that bug as wontfix, so I'll close this one also.

Let me know if there are further questions.

Note You need to log in before you can comment on or make changes to this bug.