Bug 2029914

Summary: FIPS enabled RHEL7 server: Candlepin services not running after installation
Product: Red Hat Satellite Reporter: Peter Ondrejka <pondrejk>
Component: InstallationAssignee: Evgeni Golov <egolov>
Status: CLOSED ERRATA QA Contact: Peter Ondrejka <pondrejk>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.11.0CC: egolov, gpayelka, gtalreja, pcreech
Target Milestone: 6.11.0Keywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-installer-3.1.2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-05 14:30:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Peter Ondrejka 2021-12-07 14:44:09 UTC
Description of problem:

candlepin services fail to start after satellite installation

# hammer ping
database:         
    Status:          ok
    Server Response: Duration: 0ms
candlepin:        
    Status:          FAIL
    Server Response: Message: 404 Not Found
candlepin_auth:   
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events: 
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 2ms
katello_events:   
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:            
    Status:          ok
    Server Response: Duration: 454ms
pulp3_content:    
    Status:          ok
    Server Response: Duration: 189ms
foreman_tasks:    
    Status:          ok
    Server Response: Duration: 6ms


Restarting the service with foreman-maintain health check fails with:
Couldn't connect to the server:undefined method `[]' for nil:NilClass

Version-Release number of selected component (if applicable):
Satellite 7, sn 1
satellite-7.0.0-0.1.beta.el7sat.noarch
candlepin-4.0.9-1.el7sat.noarch

some avc denials occurring:

 ausearch -i -m avc
----
type=PROCTITLE msg=audit(12/07/2021 07:49:09.402:817) : proctitle=pam_tally2 --user foreman --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:49:09.402:817) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x5603e62f8ec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x7faedfff69d0 items=0 ppid=3164 pid=3169 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:49:09.402:817) : avc:  denied  { write } for  pid=3169 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:49:38.274:881) : proctitle=pam_tally2 --user tss --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:49:38.274:881) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55ac218fdec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x7ff13cdc1690 items=0 ppid=3288 pid=3293 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:49:38.274:881) : avc:  denied  { write } for  pid=3293 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:49:42.169:926) : proctitle=pam_tally2 --user saslauth --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:49:42.169:926) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55e74defdec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x7ff3eebf6aa0 items=0 ppid=3430 pid=3435 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:49:42.169:926) : avc:  denied  { write } for  pid=3435 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:49:45.475:948) : proctitle=pam_tally2 --user foreman-proxy --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:49:45.475:948) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55cf18c15ec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x0 items=0 ppid=3483 pid=3488 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:49:45.475:948) : avc:  denied  { write } for  pid=3488 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:50:13.644:1005) : proctitle=pam_tally2 --user tomcat --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:50:13.644:1005) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55c977382ec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x7fe75ccd9c40 items=0 ppid=3619 pid=3624 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:50:13.644:1005) : avc:  denied  { write } for  pid=3624 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:51:17.265:1039) : proctitle=pam_tally2 --user postgres --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:51:17.265:1039) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55b1403e2ec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x0 items=0 ppid=4335 pid=4340 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:51:17.265:1039) : avc:  denied  { write } for  pid=4340 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:51:36.792:1058) : proctitle=pam_tally2 --user redis --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:51:36.792:1058) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x55b936d41ec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x0 items=0 ppid=4424 pid=4429 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:51:36.792:1058) : avc:  denied  { write } for  pid=4429 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 
----
type=PROCTITLE msg=audit(12/07/2021 07:54:17.521:1171) : proctitle=pam_tally2 --user apache --reset --quiet 
type=SYSCALL msg=audit(12/07/2021 07:54:17.521:1171) : arch=x86_64 syscall=open success=no exit=EACCES(Permission denied) a0=0x560689b1bec4 a1=O_RDONLY|O_CREAT|O_APPEND a2=0600 a3=0x0 items=0 ppid=5412 pid=5417 auid=root uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=3 comm=pam_tally2 exe=/usr/sbin/pam_tally2 subj=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 key=(null) 
type=AVC msg=audit(12/07/2021 07:54:17.521:1171) : avc:  denied  { write } for  pid=5417 comm=pam_tally2 name=log dev="vda1" ino=12583030 scontext=unconfined_u:unconfined_r:useradd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_log_t:s0 tclass=dir permissive=0 


How reproducible:
always

Steps to Reproduce:
1. fresh installation of Satellite
2. hammer ping

Actual results:
see above

Expected results:
services running

Additional info:
this particular system is fips enabled, issue looks similar to https://bugzilla.redhat.com/show_bug.cgi?id=1897360

Comment 3 Evgeni Golov 2022-01-26 15:33:58 UTC
I couldn't repro the issue on upstream nightly (didn't try snaps yet).

Is there a reproducer for this available somewhere already?

Comment 4 Evgeni Golov 2022-01-26 15:34:58 UTC
also, the selinux denials are about pam_tally2, not really satellite related

Comment 5 Evgeni Golov 2022-01-26 16:27:39 UTC
I'm stupid. Reproduced.

Comment 6 Evgeni Golov 2022-02-07 09:03:58 UTC
Note to whoever will pick that:
you need the attached redmine *AND* the puppet-katello change which went in without a redmine

Comment 7 Nikos Moumoulidis 2022-03-02 10:57:19 UTC
*** Bug 2059351 has been marked as a duplicate of this bug. ***

Comment 10 Ganesh Payelkar 2022-05-23 19:46:49 UTC
Even if we tried to enable FIPS through below steps it is getting failed 


/etc/tomcat/server.xml

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />
to:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" FIPSMode="on" />


SEVERE: Failed to enter FIPS mode
java.lang.Error: Failed to enter FIPS mode
	at org.apache.catalina.core.AprLifecycleListener.lifecycleEvent(AprLifecycleListener.java:146)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:101)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:642)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:667)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)

May 20, 2022 2:49:05 AM org.apache.catalina.startup.Catalina load
SEVERE: Catalina.start
org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[8005]]
Caused by: java.lang.Error: Failed to enter FIPS mode


error 'Oops, we're sorry but something went wrong Failed to open TCP connection to localhost:23443 (No route to host - connect(2) for "localhost" port 23443)'

Comment 13 errata-xmlrpc 2022-07-05 14:30:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Satellite 6.11 Release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5498