Disabling selinux works around the issue:

# setenforce 0

# systemctl restart rh-mongodb34-mongod

# systemctl status rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2020-08-27 16:25:21 EDT; 11s ago

time->Thu Aug 27 16:26:57 2020
type=PROCTITLE msg=audit(1598560017.267:775): proctitle=2F6F70742F72682F72682D6D6F6E676F646233342F726F6F742F7573722F62696E2F6D6F6E676F64002D66002F6574632F6F70742F72682F72682D6D6F6E676F646233342F6D6F6E676F642E636F6E660072756E
type=SYSCALL msg=audit(1598560017.267:775): arch=c000003e syscall=2 success=no exit=-13 a0=563fdc988898 a1=241 a2=1b6 a3=7fbc3b3cf8a0 items=0 ppid=83937 pid=83938 auid=4294967295 uid=184 gid=184 euid=184 suid=184 fsuid=184 egid=184 sgid=184 fsgid=184 tty=(none) ses=4294967295 comm="mongod" exe="/opt/rh/rh-mongodb34/root/usr/bin/mongod" subj=system_u:system_r:mongod_t:s0 key=(null)
type=AVC msg=audit(1598560017.267:775): avc:  denied  { write } for  pid=83938 comm="mongod" name="mongodb" dev="dm-0" ino=1351829 scontext=system_u:system_r:mongod_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0

on the test VM:


# ls -alhZ /opt/rh/rh-mongodb34/root/usr/bin/mongod
-rwxr-xr-x. root root system_u:object_r:bin_t:s0       /opt/rh/rh-mongodb34/root/usr/bin/mongod

on my functional, fresh provisioned RHEL 7.8 Satellite 6.8 install:


# ls -alhZ /opt/rh/rh-mongodb34/root/usr/bin/mongod
-rwxr-xr-x. root root system_u:object_r:mongod_exec_t:s0 /opt/rh/rh-mongodb34/root/usr/bin/mongod

I corrected this and *many* other improperly labled files via:

# restorecon -vR /opt/*
...
restorecon reset /opt/rh/rh-mongodb34/root context system_u:object_r:root_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/boot context system_u:object_r:boot_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/dev context system_u:object_r:device_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/home context system_u:object_r:home_root_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/lib64 context system_u:object_r:lib_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/media context system_u:object_r:mnt_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/mnt context system_u:object_r:mnt_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/root context system_u:object_r:admin_home_t:s0->system_u:object_r:usr_t:s0
restorecon reset /opt/rh/rh-mongodb34/root/run context system_u:object_r:var_run_t:s0->system_u:object_r:usr_t:s0
...

There is likely something in your installation routine that is either disabling SELinux then re-enabling it, causing files to be written in the wrong context, which needs a restore to get set properly.

Full log of what was restored:

http://file.rdu.redhat.com/~mmccune/1873319-restorecon.log

Once the contexts were restored, satellite-installer runs fine:

# satellite-installer 
Preparing installation Done                                              
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
=============================================
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
foreman-rake upgrade:run finished successfully!
  Success!
  * Satellite is running at https://example.redhat.com

  * To install an additional Capsule on separate machine continue by running:

      capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE" --certs-tar "/root/$CAPSULE-certs.tar"

  * To upgrade an existing 6.7 Capsule to 6.8:
      Please see official documentation for steps and parameters to use when upgrading a 6.7 Capsule to 6.8.

  * Capsule is running at https://example.redhat.com:9090
  The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.

DEV: Hold off trying to reproduce this, this appears unique to a specific test environment which is being rebuilt and not likely a systemic issue with 6.8.

Will keep this bug open to track the issue as we progress.

The same issue occurs on Capsule during a z-stream upgrade via foreman-maintain:

# foreman-maintain upgrade run --target-version=6.8.z --whitelist="repositories-validate,repositories-setup" --assumeyes
...
--------------------------------------------------------------------------------
Procedures::Installer::Upgrade:                                       [FAIL]
...
 Systemd start for rh-mongodb34-mongod failed!
journalctl log for rh-mongodb34-mongod:
-- Logs begin at Tue 2020-09-15 13:04:58 EDT, end at Tue 2020-09-15 14:26:09 EDT. --
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com systemd[1]: Starting High-performance, schema-free document-oriented database...
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com mongodb-scl-helper[65801]: about to fork child process, waiting until server is ready for connections.
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com mongodb-scl-helper[65801]: forked process: 65806
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com mongod.27017[65806]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com mongodb-scl-helper[65801]: ERROR: child process failed, exited with error number 1
Sep 15 14:26:09 intel-knm-02.khw1.lab.eng.bos.redhat.com systemd[1]: rh-mongodb34-mongod.service: control process exited, code=exited status=1

'chcon -t mongod_log_t /var/opt/rh/rh-mongodb34/run/mongodb/' or 'restorecon -R /*' or 'setenforce 0' workarounds the issue

I saw this today on attempting to update a SNAP 17 Satellite.

AVC denial:

time->Mon Sep 28 12:09:17 2020
type=PROCTITLE msg=audit(1601309357.684:7552): proctitle=2F6F70742F72682F72682D6D6F6E676F646233342F726F6F742F7573722F62696E2F6D6F6E676F64002D66002F6574632F6F70742F72682F72682D6D6F6E676F646233342F6D6F6E676F642E636F6E660072756E
type=SYSCALL msg=audit(1601309357.684:7552): arch=c000003e syscall=2 success=no exit=-13 a0=55746fae6898 a1=241 a2=1b6 a3=7f5c8e56d8a0 items=0 ppid=15448 pid=15449 auid=4294967295 uid=184 gid=184 euid=184 suid=184 fsuid=184 egid=184 sgid=184 fsgid=184 tty=(none) ses=4294967295 comm="mongod" exe="/opt/rh/rh-mongodb34/root/usr/bin/mongod" subj=system_u:system_r:mongod_t:s0 key=(null)
type=AVC msg=audit(1601309357.684:7552): avc:  denied  { write } for  pid=15449 comm="mongod" name="mongodb" dev="dm-0" ino=1946309408 scontext=system_u:system_r:mongod_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
----

error from system:

Sep 28 12:04:25 sat-r220-10.lab.eng.rdu2.redhat.com mongod.27017[15389]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied

moving this back into 6.8

QE seems to be able to reproduce this reliably by:

1) Install OS

2) Install Satellite bits via yum

3) run 'satellite-installer'

4) installer completes properly, hammer ping, etc. all OK

5) Reboot the host

6) Now, mongo will not start until you 'restorecon -r /'

After some investigation, this issue stems from not fully updating the OS prior to installing Satellite. Per our documentation, the workflow to avoid this should be:

 1) Provision OS
 2) Fully update OS to latest RHEL 7
 3) yum install satellite
 4) satellite-installer --scenario satellite
 5) reboot


If the system is not fully up to date, when satellite is installed, SELinux packages are updated in the same yum transaction. This can lead to ordering issues with post install scripts in the RPMs that lead issues such as the context changing issue this BZ describes for Mongo.

I propose this be CLOSED NOTABUG

WORKAROUND: 

If the user did not fully execute 'yum update' before installing Satellite simply run:

# restorecon -r /
# satellite-installer

Going to close this out as there is nothing we can do natively in Satellite to resolve the issue and will be avoided if following our installation instructions.