Bug 1893291

This looks like a duplicate of bug 1846364

*** This bug has been marked as a duplicate of bug 1846364 ***

Hi Alexon, that podman version doesn't look correct - it should be podman-1.6.4-15 at least. Is the container-tools:2.0 module stream enabled?

I think bug 1866290 may be a clearer duplicate for this. Comment 30 on that bug summarises the steps required to get to the correct podman, as well the steps to follow after that to restart the libvirt containers. Note that just upgrading podman is not sufficient as the labels would still be incorrect.

The knowledge base article at https://access.redhat.com/solutions/5297991 also has a playbook to run these commands across all compute nodes more conveniently. Does this help with resolving the issue?

Hi Julie,

Just to be clear, I'm working in the RHOSP 16.1 All-In-One installation, according to the official documentation:

Creating an all-in-one OpenStack cloud for test and proof-of-concept environments:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/quick_start_guide/index

The official documentation doesn't require to enable the container-tools:2.0 module stream. The only requirements regard packages are:

===
[stack@all-in-one]$ sudo dnf install -y dnf-utils
[stack@all-in-one]$ sudo subscription-manager repos --disable=*
[stack@all-in-one]$ sudo subscription-manager repos \
--enable=rhel-8-for-x86_64-baseos-eus-rpms \
--enable=rhel-8-for-x86_64-appstream-eus-rpms \
--enable=rhel-8-for-x86_64-highavailability-eus-rpms \
--enable=ansible-2.9-for-rhel-8-x86_64-rpms \
--enable=openstack-16.1-for-rhel-8-x86_64-rpms \
--enable=fast-datapath-for-rhel-8-x86_64-rpms \
--enable=rhel-8-for-x86_64-highavailability-rpms

[stack@all-in-one]$ sudo dnf install -y python3-tripleoclient
===

As you can see from below, I have the same repositories:

===
$ sudo dnf repolist

repo id                                        repo name
ansible-2.9-for-rhel-8-x86_64-rpms             Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs)
fast-datapath-for-rhel-8-x86_64-rpms           Fast Datapath for RHEL 8 x86_64 (RPMs)
openstack-16.1-for-rhel-8-x86_64-rpms          Red Hat OpenStack Platform 16.1 for RHEL 8 x86_64 (RPMs)
rhel-8-for-x86_64-appstream-eus-rpms           Red Hat Enterprise Linux 8 for x86_64 - AppStream - Extended Update Support (RPMs)
rhel-8-for-x86_64-baseos-eus-rpms              Red Hat Enterprise Linux 8 for x86_64 - BaseOS - Extended Update Support (RPMs)
rhel-8-for-x86_64-highavailability-eus-rpms    Red Hat Enterprise Linux 8 for x86_64 - High Availability - Extended Update Support (RPMs)
rhel-8-for-x86_64-highavailability-rpms        Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs)
satellite-tools-6.7-for-rhel-8-x86_64-rpms     Red Hat Satellite Tools 6.7 for RHEL 8 x86_64 (RPMs)
===


Also, it specifies:

===
If you want to use the all-in-one Red Hat OpenStack Platform installation in a virtual environment, you must define the virtualization type with the StandaloneExtraConfig parameter:

StandaloneExtraConfig:
  NovaComputeLibvirtType: qemu
===

The problem is I just can't change everything as the KCS recommends because it's an "all-in-one" deployment. Even if I reboot it, it fails to come up and run again and I need to redeploy everything.

If I disable SELinux after deployment, it works fine, no issues. But the problem is, the way I see, if the official documentation doesn't mention that the container-tools:2.0 module stream should be enabled prior the installation, that's a flaw. Another thing is the procedure described in that other BZ doesn't apply here completely because it's an "all-in-one" deployment, so we should have a different procedure for this environment.

I agree, it seems like the documentation needs to be updated with the information about the module streams. Someone left a comment on the knowledge base article about how to adjust the paunch command for a standalone deployment which might help, though it might still need to be adjusted as it seems related to a different deployment method.

Cedric, I'm wondering if you would have an idea on how to adjust the libvirt/SELinux/podman workaround so that it can be applied in a standalone environment? (cf. comment 7 where the current one we documented in a KB doesn't work in this context.)
Also, to confirm before we update the docs, is it correct that enabling the correct streams after configuring the subcriptions will mean we get the right podman set up for the libvirt containers too?
Thank you!

I will move this to the documentation component.

Hello,

A couple of things/thoughts/details:
- iirc there actually IS a mention of "enable container-tools:2.0" somewhere in the official documentation - but apparently it's done only for upgrades, updates and LEAPP[1]

- there is a bunch of changes coming in z3 (or already in z2 - not sure anymore now) actually pinning the podman version in different fashion, being either within ansible directly, or with package dependencies

- though, with modules and streams, we can't make a dependency on a module stream (i.e. we can't make that python3-tripleoclient depends on container-tools:2.0 being active)

- the version pinning as currently implemented will NOT crash "dnf install -y python3-tripleoclient" - though there's also work in progress on that part[2][3]

- ensuring we get the correct stream before deploying will, of course, make the whole thing work, since podman will be installed at a correct version. Therefore, adding some hint about "ensure you enable the following streams before installing osp-16.1" is a must (for instance, there's also the "virt" module, must go from virt:rhel to virt:8.2 apparently).

Does it answer the questions, Julie?

Cheers,

C.



[1] https://access.redhat.com/search/#/?q=container-tools:2.0&p=1&sort=relevant&scoped&documentKind=Documentation - for instance https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/framework_for_upgrades_13_to_16.1/configuring-the-overcloud-for-a-leapp-upgrade#creating-an-upgrades-environment-file-overcloud-leapp
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1878189
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1861777

I think it does! Thank you. It sounds like the solution is to update the document Alexon linked to in comment 7 with the following step, to be run after enabling the repos/subscriptions (aka in section 3, between step 8 (subcription manager) and 9 (install python-tripleoclient)):


# dnf module disable -y container-tools:rhel8
# dnf module enable -y container-tools:2.0
# dnf module disable -y virt:rhel
# dnf module enable -y virt:8.2

This should prevent the problem from occurring.

Julie/Cédric,

As you can see from the official documentation now [1], I managed to get the Chapter 3, Step 9, updated with your recommendation. But the problem remains. I've tested to redeploy the whole environment with this new configuration, but it didn't work.

Do you have any other recommendation so I can test it?

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/quick_start_guide/installing-the-all-in-one-openstack-environment

Hi Alexon,

Great to see the documentation updated! I will leave the needinfo on Cedric as he is more familiar with the details of the problem and might see something I'm missing, but I am wondering from your "redeploy" comment if the same existing environment was reused as opposed to creating a new deployment from scratch?

If this is a redeploy, the SELinux labels are likely still wrong. https://bugzilla.redhat.com/show_bug.cgi?id=1866290#c30 and related KB article have a couple of additional steps for how to restart the libvirt container so that it uses the right labels... Could this help?

If not, can you attach the permissive logs for SELinux again to see if it's a new error as well as confirm the podman version? Thank you.

Julie/Cédric,

Just in case, I've just finished a fresh install, from scratch, brand new VM, following all your recommendations and following the quoted BZ and KCS. Check it out:

===
[root@openstack16 ~]# hostnamectl 

   Static hostname: openstack16.example.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: f85aa0aaf6894ab792bd555197a13152
           Boot ID: 7c99b506f0c9434a95430de0ceebcd6a
    Virtualization: kvm
  Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
            Kernel: Linux 4.18.0-193.29.1.el8_2.x86_64
      Architecture: x86-64


[root@openstack16 ~]# getenforce

Enforcing

[root@openstack16 ~]# rpm -qa | grep -i openstack

openstack-tripleo-common-containers-11.4.1-1.20200914165651.el8ost.noarch
python3-openstackclient-4.0.1-1.20200817092223.bff556c.el8ost.noarch
openstack-selinux-0.8.24-1.20200914163011.26243bf.el8ost.noarch
openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch
python3-openstacksdk-0.36.4-0.20200715054250.76d3b29.el8ost.noarch
python-openstackclient-lang-4.0.1-1.20200817092223.bff556c.el8ost.noarch
puppet-openstacklib-15.4.1-0.20200403203429.5fdf43c.el8ost.noarch
openstack-heat-api-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-ironic-python-agent-builder-2.1.1-1.20200914175356.65d0f80.el8ost.noarch
openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch
openstack-heat-engine-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-tripleo-puppet-elements-11.2.2-0.20200701163410.432518a.el8ost.noarch
ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.noarch
openstack-heat-monolith-13.0.3-1.20200914171254.48b730a.el8ost.noarch
puppet-openstack_extras-15.4.1-0.20200528113453.371931c.el8ost.noarch
openstack-heat-common-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.noarch
openstack-heat-agents-1.10.1-0.20200311091123.96b819c.el8ost.noarch
openstack-tripleo-common-11.4.1-1.20200914165651.el8ost.noarch

[root@openstack16 ~]# rpm -qa | grep podman

podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64


(admin_overcloud) [stack@openstack16 ~]$ openstack server list

+--------------------------------------+-----------+--------+----------+---------+--------+
| ID                                   | Name      | Status | Networks | Image   | Flavor |
+--------------------------------------+-----------+--------+----------+---------+--------+
| 50635c9c-6a14-47f8-b47e-acc408277890 | myserver2 | ERROR  |          | cirros4 | tiny   |
| ba580948-51ee-4957-baf4-aae5cf924ac9 | myserver1 | ERROR  |          | cirros5 | tiny   |
+--------------------------------------+-----------+--------+----------+---------+--------+

[root@openstack16 ~]# aureport -a

AVC Report
===============================================================
# date time comm subj syscall class permission obj result event
===============================================================
1. 01/12/2020 18:27:01 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 248
2. 01/12/2020 18:46:14 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 326
3. 01/12/2020 18:46:43 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 329
4. 01/12/2020 18:48:16 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 361
5. 01/12/2020 18:48:22 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 362
6. 01/12/2020 18:48:53 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 364
7. 09/12/2020 18:47:18 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 330
8. 09/12/2020 18:47:27 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 333
9. 09/12/2020 18:47:34 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 334
10. 09/12/2020 18:47:38 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0 (null) (null) (null) unset 337
11. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0 file read system_u:object_r:root_t:s0 denied 380
12. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0 file read system_u:object_r:root_t:s0 denied 381


[root@openstack16 ~]# ausearch -a 380
----
time->Tue Dec  1 18:30:56 2020
type=SERVICE_START msg=audit(1606847456.995:380): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
----
time->Tue Dec  1 18:57:54 2020
type=SOFTWARE_UPDATE msg=audit(1606849074.991:380): pid=5310 uid=0 auid=1001 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=install sw="python3-pbr-5.1.2-2.el8ost.noarch" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local addr=? terminal=pts/0 res=success'
----
time->Wed Dec  9 18:59:55 2020
type=AVC msg=audit(1607540395.240:380): avc:  denied  { read } for  pid=20402 comm="rhsmcertd-worke" name="satellite-5-client.module" dev="dm-0" ino=135 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0

[root@openstack16 ~]# ausearch -a 381
----
time->Tue Dec  1 18:30:56 2020
type=SERVICE_STOP msg=audit(1606847456.995:381): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
----
time->Tue Dec  1 18:57:54 2020
type=SOFTWARE_UPDATE msg=audit(1606849074.991:381): pid=5310 uid=0 auid=1001 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=install sw="python3-pyyaml-3.12-12.el8.x86_64" sw_type=rpm key_enforce=0 gpg_res=1 root_dir="/" comm="dnf" exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local addr=? terminal=pts/0 res=success'
----
time->Wed Dec  9 18:59:55 2020
type=AVC msg=audit(1607540395.240:381): avc:  denied  { read } for  pid=20402 comm="rhsmcertd-worke" name="virt.module" dev="dm-0" ino=136 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0
===

Also, after finishing the deployment, I ran the recommended playbook as follows (PS. I've commented the modules and updates parts because I've already did prior the deployment):

==
---
- hosts: localhost
  gather_facts: false
  become: true
  tasks:
#    - name: disable virt:rhel
#      command: dnf -y module disable virt:rhel

#    - name: enable virt:8.2
#      command: dnf -y module enable virt:8.2

#    - name: disable container-tools:rhel8
#      command: dnf -y module disable container-tools:rhel8

#    - name: enable container-tools:2.0
#      command: dnf -y module enable container-tools:2.0

#    - name: update remaining packages
#      dnf:
#        name: '*'
#        state: latest
#        exclude: ansible

    - name: stop nova libvirt
      service:
        name: tripleo_nova_libvirt
        state: stopped
        enabled: no

    - name: trash libvirt container
      command: podman rm nova_libvirt

    - name: recreate nova container with proper selinux settings
      command: paunch apply --file /var/lib/tripleo-config/container-startup-config/step_3/nova_libvirt.json --config-id tripleo_step3 --managed-by tripleo-Standalone

    - name: stop nova libvirt
      service:
        name: tripleo_nova_libvirt
        state: started
        enabled: yes

    - name: make selinux great again
      selinux:
        policy: targeted
        state: enforcing
==

When I did it, the Keystone and Horizon containers stopped to work and were deleted. I needed to come up with them manually like this:

===
paunch apply --file /var/lib/tripleo-config/container-startup-config/step_3/keystone.json --config-id tripleo_step3 --managed-by tripleo-Standalone
paunch apply --file /var/lib/tripleo-config/container-startup-config/step_3/horizon.json --config-id tripleo_step3 --managed-by tripleo-Standalone
===

The problem is, when I do start the Keystone container, the Horizon container stops to work. If I start the Horizon container, the Keystone container stops to work (facepalm).

Also, if I reboot the VM, nothing works anymore and I need to reinstall everything again.

I don't know what else could be done. What do y'all have in mind?

Hello,

Some comments inline.

(In reply to Alexon Oliveira from comment #15)
> Julie/Cédric,
> 
> Just in case, I've just finished a fresh install, from scratch, brand new
> VM, following all your recommendations and following the quoted BZ and KCS.
> Check it out:
> 
> ===
> [root@openstack16 ~]# hostnamectl 
> 
>    Static hostname: openstack16.example.local
>          Icon name: computer-vm
>            Chassis: vm
>         Machine ID: f85aa0aaf6894ab792bd555197a13152
>            Boot ID: 7c99b506f0c9434a95430de0ceebcd6a
>     Virtualization: kvm
>   Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
>        CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
>             Kernel: Linux 4.18.0-193.29.1.el8_2.x86_64
>       Architecture: x86-64
> 
> 
> [root@openstack16 ~]# getenforce
> 
> Enforcing
> 
> [root@openstack16 ~]# rpm -qa | grep -i openstack
> 
> openstack-tripleo-common-containers-11.4.1-1.20200914165651.el8ost.noarch
> python3-openstackclient-4.0.1-1.20200817092223.bff556c.el8ost.noarch
> openstack-selinux-0.8.24-1.20200914163011.26243bf.el8ost.noarch
> openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch
> python3-openstacksdk-0.36.4-0.20200715054250.76d3b29.el8ost.noarch
> python-openstackclient-lang-4.0.1-1.20200817092223.bff556c.el8ost.noarch
> puppet-openstacklib-15.4.1-0.20200403203429.5fdf43c.el8ost.noarch
> openstack-heat-api-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> openstack-ironic-python-agent-builder-2.1.1-1.20200914175356.65d0f80.el8ost.
> noarch
> openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch
> openstack-heat-engine-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> openstack-tripleo-puppet-elements-11.2.2-0.20200701163410.432518a.el8ost.
> noarch
> ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.
> noarch
> openstack-heat-monolith-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> puppet-openstack_extras-15.4.1-0.20200528113453.371931c.el8ost.noarch
> openstack-heat-common-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.
> noarch
> openstack-heat-agents-1.10.1-0.20200311091123.96b819c.el8ost.noarch
> openstack-tripleo-common-11.4.1-1.20200914165651.el8ost.noarch
> 
> [root@openstack16 ~]# rpm -qa | grep podman
> 
> podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64

This is the wrong podman version....... No wonder it's not working then.

> 
> 
> (admin_overcloud) [stack@openstack16 ~]$ openstack server list
> 
> +--------------------------------------+-----------+--------+----------+-----
> ----+--------+
> | ID                                   | Name      | Status | Networks |
> Image   | Flavor |
> +--------------------------------------+-----------+--------+----------+-----
> ----+--------+
> | 50635c9c-6a14-47f8-b47e-acc408277890 | myserver2 | ERROR  |          |
> cirros4 | tiny   |
> | ba580948-51ee-4957-baf4-aae5cf924ac9 | myserver1 | ERROR  |          |
> cirros5 | tiny   |
> +--------------------------------------+-----------+--------+----------+-----
> ----+--------+
> 
> [root@openstack16 ~]# aureport -a
> 
> AVC Report
> ===============================================================
> # date time comm subj syscall class permission obj result event
> ===============================================================
> 1. 01/12/2020 18:27:01 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 248
> 2. 01/12/2020 18:46:14 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 326
> 3. 01/12/2020 18:46:43 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 329
> 4. 01/12/2020 18:48:16 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 361
> 5. 01/12/2020 18:48:22 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 362
> 6. 01/12/2020 18:48:53 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 364
> 7. 09/12/2020 18:47:18 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 330
> 8. 09/12/2020 18:47:27 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 333
> 9. 09/12/2020 18:47:34 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 334
> 10. 09/12/2020 18:47:38 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> (null) (null) (null) unset 337
> 11. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0
> file read system_u:object_r:root_t:s0 denied 380
> 12. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0
> file read system_u:object_r:root_t:s0 denied 381
> 
> 
> [root@openstack16 ~]# ausearch -a 380
> ----
> time->Tue Dec  1 18:30:56 2020
> type=SERVICE_START msg=audit(1606847456.995:380): pid=1 uid=0
> auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> msg='unit=user-runtime-dir@0 comm="systemd" exe="/usr/lib/systemd/systemd"
> hostname=? addr=? terminal=? res=success'
> ----
> time->Tue Dec  1 18:57:54 2020
> type=SOFTWARE_UPDATE msg=audit(1606849074.991:380): pid=5310 uid=0 auid=1001
> ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> msg='op=install sw="python3-pbr-5.1.2-2.el8ost.noarch" sw_type=rpm
> key_enforce=0 gpg_res=1 root_dir="/" comm="dnf"
> exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local
> addr=? terminal=pts/0 res=success'
> ----
> time->Wed Dec  9 18:59:55 2020
> type=AVC msg=audit(1607540395.240:380): avc:  denied  { read } for 
> pid=20402 comm="rhsmcertd-worke" name="satellite-5-client.module" dev="dm-0"
> ino=135 scontext=system_u:system_r:rhsmcertd_t:s0
> tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0
> 
> [root@openstack16 ~]# ausearch -a 381
> ----
> time->Tue Dec  1 18:30:56 2020
> type=SERVICE_STOP msg=audit(1606847456.995:381): pid=1 uid=0 auid=4294967295
> ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@0
> comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
> res=success'
> ----
> time->Tue Dec  1 18:57:54 2020
> type=SOFTWARE_UPDATE msg=audit(1606849074.991:381): pid=5310 uid=0 auid=1001
> ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> msg='op=install sw="python3-pyyaml-3.12-12.el8.x86_64" sw_type=rpm
> key_enforce=0 gpg_res=1 root_dir="/" comm="dnf"
> exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local
> addr=? terminal=pts/0 res=success'
> ----
> time->Wed Dec  9 18:59:55 2020
> type=AVC msg=audit(1607540395.240:381): avc:  denied  { read } for 
> pid=20402 comm="rhsmcertd-worke" name="virt.module" dev="dm-0" ino=136
> scontext=system_u:system_r:rhsmcertd_t:s0
> tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0
> ===
> 
> Also, after finishing the deployment, I ran the recommended playbook as
> follows (PS. I've commented the modules and updates parts because I've
> already did prior the deployment):


Did you REALLY switch the module streams? Since you have the -12, I'm pretty sure you didn't do it...
Though I thought there were some hard limitations/Depends on the exact podman version that should have prevented the deploy..... Adding a needinfo on Lon..

> 
> ==
> ---
> - hosts: localhost
>   gather_facts: false
>   become: true
>   tasks:
> #    - name: disable virt:rhel
> #      command: dnf -y module disable virt:rhel
> 
> #    - name: enable virt:8.2
> #      command: dnf -y module enable virt:8.2
> 
> #    - name: disable container-tools:rhel8
> #      command: dnf -y module disable container-tools:rhel8
> 
> #    - name: enable container-tools:2.0
> #      command: dnf -y module enable container-tools:2.0
> 
> #    - name: update remaining packages
> #      dnf:
> #        name: '*'
> #        state: latest
> #        exclude: ansible
> 
>     - name: stop nova libvirt
>       service:
>         name: tripleo_nova_libvirt
>         state: stopped
>         enabled: no
> 
>     - name: trash libvirt container
>       command: podman rm nova_libvirt
> 
>     - name: recreate nova container with proper selinux settings
>       command: paunch apply --file
> /var/lib/tripleo-config/container-startup-config/step_3/nova_libvirt.json
> --config-id tripleo_step3 --managed-by tripleo-Standalone
> 
>     - name: stop nova libvirt
>       service:
>         name: tripleo_nova_libvirt
>         state: started
>         enabled: yes
> 
>     - name: make selinux great again
>       selinux:
>         policy: targeted
>         state: enforcing
> ==
> 
> When I did it, the Keystone and Horizon containers stopped to work and were
> deleted. I needed to come up with them manually like this:

There is no reason Horizon or Keyston stop working when you trash nova_libvirt container. At least with the right podman version... But since you have an unsupported one, there might be some tricks there.

> 
> ===
> paunch apply --file
> /var/lib/tripleo-config/container-startup-config/step_3/keystone.json
> --config-id tripleo_step3 --managed-by tripleo-Standalone
> paunch apply --file
> /var/lib/tripleo-config/container-startup-config/step_3/horizon.json
> --config-id tripleo_step3 --managed-by tripleo-Standalone
> ===
> 
> The problem is, when I do start the Keystone container, the Horizon
> container stops to work. If I start the Horizon container, the Keystone
> container stops to work (facepalm).

might also be due to a configuration you've passed and services are trying to hit on the same port/device at some point? Seems unrelated to the whole SELinux story, unless you can point to denials. Still. Wrong podman version, can't do anything good to the whole stability.

> 
> Also, if I reboot the VM, nothing works anymore and I need to reinstall
> everything again.
> 
> I don't know what else could be done. What do y'all have in mind?

The podman version is still wrong indeed (minimum is podman-1.6.4-15).

Additionally the SELinux denials look new and are related to rhsmcertd_t this time? Could there be a problem even earlier in the process when setting up the modules and subscription manager? This may explain why the virt module stream didn't show up either:

type=AVC msg=audit(1607540395.240:381): avc:  denied  { read } for  pid=20402 comm="rhsmcertd-worke" name="virt.module" dev="dm-0" ino=136 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0

This looks like the virt module configuration file couldn't be read? I haven't seen this before. Was there any error when setting up the subscriptions and modules otherwise?

The *.module read errors look a lot like bug 1775975 which indicate that the label for those files should be etc_t, not root_t like in the error message. 

# restorecon -Rv /etc/dnf/modules.d

should resolve this. This can be confirmed after with `ls -alZ /etc/dnf/modules.d`.

I think the advanced-virt-for-rhel-8-x86_64-rpms is missing from the list of repos though, that's where virt:8.2 comes from... But I don't think this should impact the podman version issue, as this comes from the container-tools:2.0 stream.

(In reply to Cédric Jeanneret from comment #16)
> Hello,
> 
> Some comments inline.
> 
> (In reply to Alexon Oliveira from comment #15)
> > Julie/Cédric,
> > 
> > Just in case, I've just finished a fresh install, from scratch, brand new
> > VM, following all your recommendations and following the quoted BZ and KCS.
> > Check it out:
> > 
> > ===
> > [root@openstack16 ~]# hostnamectl 
> > 
> >    Static hostname: openstack16.example.local
> >          Icon name: computer-vm
> >            Chassis: vm
> >         Machine ID: f85aa0aaf6894ab792bd555197a13152
> >            Boot ID: 7c99b506f0c9434a95430de0ceebcd6a
> >     Virtualization: kvm
> >   Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
> >        CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
> >             Kernel: Linux 4.18.0-193.29.1.el8_2.x86_64
> >       Architecture: x86-64
> > 
> > 
> > [root@openstack16 ~]# getenforce
> > 
> > Enforcing
> > 
> > [root@openstack16 ~]# rpm -qa | grep -i openstack
> > 
> > openstack-tripleo-common-containers-11.4.1-1.20200914165651.el8ost.noarch
> > python3-openstackclient-4.0.1-1.20200817092223.bff556c.el8ost.noarch
> > openstack-selinux-0.8.24-1.20200914163011.26243bf.el8ost.noarch
> > openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch
> > python3-openstacksdk-0.36.4-0.20200715054250.76d3b29.el8ost.noarch
> > python-openstackclient-lang-4.0.1-1.20200817092223.bff556c.el8ost.noarch
> > puppet-openstacklib-15.4.1-0.20200403203429.5fdf43c.el8ost.noarch
> > openstack-heat-api-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> > openstack-ironic-python-agent-builder-2.1.1-1.20200914175356.65d0f80.el8ost.
> > noarch
> > openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch
> > openstack-heat-engine-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> > openstack-tripleo-puppet-elements-11.2.2-0.20200701163410.432518a.el8ost.
> > noarch
> > ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.
> > noarch
> > openstack-heat-monolith-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> > puppet-openstack_extras-15.4.1-0.20200528113453.371931c.el8ost.noarch
> > openstack-heat-common-13.0.3-1.20200914171254.48b730a.el8ost.noarch
> > openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.
> > noarch
> > openstack-heat-agents-1.10.1-0.20200311091123.96b819c.el8ost.noarch
> > openstack-tripleo-common-11.4.1-1.20200914165651.el8ost.noarch
> > 
> > [root@openstack16 ~]# rpm -qa | grep podman
> > 
> > podman-1.6.4-12.module+el8.2.0+6669+dde598ec.x86_64
> 
> This is the wrong podman version....... No wonder it's not working then.
> 

You're right. The playbook I ran, for some reason, didn't work and didn't change the container-tools version.

> > 
> > 
> > (admin_overcloud) [stack@openstack16 ~]$ openstack server list
> > 
> > +--------------------------------------+-----------+--------+----------+-----
> > ----+--------+
> > | ID                                   | Name      | Status | Networks |
> > Image   | Flavor |
> > +--------------------------------------+-----------+--------+----------+-----
> > ----+--------+
> > | 50635c9c-6a14-47f8-b47e-acc408277890 | myserver2 | ERROR  |          |
> > cirros4 | tiny   |
> > | ba580948-51ee-4957-baf4-aae5cf924ac9 | myserver1 | ERROR  |          |
> > cirros5 | tiny   |
> > +--------------------------------------+-----------+--------+----------+-----
> > ----+--------+
> > 
> > [root@openstack16 ~]# aureport -a
> > 
> > AVC Report
> > ===============================================================
> > # date time comm subj syscall class permission obj result event
> > ===============================================================
> > 1. 01/12/2020 18:27:01 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 248
> > 2. 01/12/2020 18:46:14 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 326
> > 3. 01/12/2020 18:46:43 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 329
> > 4. 01/12/2020 18:48:16 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 361
> > 5. 01/12/2020 18:48:22 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 362
> > 6. 01/12/2020 18:48:53 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 364
> > 7. 09/12/2020 18:47:18 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 330
> > 8. 09/12/2020 18:47:27 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 333
> > 9. 09/12/2020 18:47:34 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 334
> > 10. 09/12/2020 18:47:38 ? system_u:system_r:system_dbusd_t:s0-s0:c0.c1023 0
> > (null) (null) (null) unset 337
> > 11. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0
> > file read system_u:object_r:root_t:s0 denied 380
> > 12. 09/12/2020 18:59:55 rhsmcertd-worke system_u:system_r:rhsmcertd_t:s0 0
> > file read system_u:object_r:root_t:s0 denied 381
> > 
> > 
> > [root@openstack16 ~]# ausearch -a 380
> > ----
> > time->Tue Dec  1 18:30:56 2020
> > type=SERVICE_START msg=audit(1606847456.995:380): pid=1 uid=0
> > auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0
> > msg='unit=user-runtime-dir@0 comm="systemd" exe="/usr/lib/systemd/systemd"
> > hostname=? addr=? terminal=? res=success'
> > ----
> > time->Tue Dec  1 18:57:54 2020
> > type=SOFTWARE_UPDATE msg=audit(1606849074.991:380): pid=5310 uid=0 auid=1001
> > ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > msg='op=install sw="python3-pbr-5.1.2-2.el8ost.noarch" sw_type=rpm
> > key_enforce=0 gpg_res=1 root_dir="/" comm="dnf"
> > exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local
> > addr=? terminal=pts/0 res=success'
> > ----
> > time->Wed Dec  9 18:59:55 2020
> > type=AVC msg=audit(1607540395.240:380): avc:  denied  { read } for 
> > pid=20402 comm="rhsmcertd-worke" name="satellite-5-client.module" dev="dm-0"
> > ino=135 scontext=system_u:system_r:rhsmcertd_t:s0
> > tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0
> > 
> > [root@openstack16 ~]# ausearch -a 381
> > ----
> > time->Tue Dec  1 18:30:56 2020
> > type=SERVICE_STOP msg=audit(1606847456.995:381): pid=1 uid=0 auid=4294967295
> > ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=user-runtime-dir@0
> > comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=?
> > res=success'
> > ----
> > time->Tue Dec  1 18:57:54 2020
> > type=SOFTWARE_UPDATE msg=audit(1606849074.991:381): pid=5310 uid=0 auid=1001
> > ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
> > msg='op=install sw="python3-pyyaml-3.12-12.el8.x86_64" sw_type=rpm
> > key_enforce=0 gpg_res=1 root_dir="/" comm="dnf"
> > exe="/usr/libexec/platform-python3.6" hostname=openstack16.example.local
> > addr=? terminal=pts/0 res=success'
> > ----
> > time->Wed Dec  9 18:59:55 2020
> > type=AVC msg=audit(1607540395.240:381): avc:  denied  { read } for 
> > pid=20402 comm="rhsmcertd-worke" name="virt.module" dev="dm-0" ino=136
> > scontext=system_u:system_r:rhsmcertd_t:s0
> > tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0
> > ===
> > 
> > Also, after finishing the deployment, I ran the recommended playbook as
> > follows (PS. I've commented the modules and updates parts because I've
> > already did prior the deployment):
> 
> 
> Did you REALLY switch the module streams? Since you have the -12, I'm pretty
> sure you didn't do it...
> Though I thought there were some hard limitations/Depends on the exact
> podman version that should have prevented the deploy..... Adding a needinfo
> on Lon..

I thought I did, but I didn't. I have fixed it later.

> 
> > 
> > ==
> > ---
> > - hosts: localhost
> >   gather_facts: false
> >   become: true
> >   tasks:
> > #    - name: disable virt:rhel
> > #      command: dnf -y module disable virt:rhel
> > 
> > #    - name: enable virt:8.2
> > #      command: dnf -y module enable virt:8.2
> > 
> > #    - name: disable container-tools:rhel8
> > #      command: dnf -y module disable container-tools:rhel8
> > 
> > #    - name: enable container-tools:2.0
> > #      command: dnf -y module enable container-tools:2.0
> > 
> > #    - name: update remaining packages
> > #      dnf:
> > #        name: '*'
> > #        state: latest
> > #        exclude: ansible
> > 
> >     - name: stop nova libvirt
> >       service:
> >         name: tripleo_nova_libvirt
> >         state: stopped
> >         enabled: no
> > 
> >     - name: trash libvirt container
> >       command: podman rm nova_libvirt
> > 
> >     - name: recreate nova container with proper selinux settings
> >       command: paunch apply --file
> > /var/lib/tripleo-config/container-startup-config/step_3/nova_libvirt.json
> > --config-id tripleo_step3 --managed-by tripleo-Standalone
> > 
> >     - name: stop nova libvirt
> >       service:
> >         name: tripleo_nova_libvirt
> >         state: started
> >         enabled: yes
> > 
> >     - name: make selinux great again
> >       selinux:
> >         policy: targeted
> >         state: enforcing
> > ==
> > 
> > When I did it, the Keystone and Horizon containers stopped to work and were
> > deleted. I needed to come up with them manually like this:
> 
> There is no reason Horizon or Keyston stop working when you trash
> nova_libvirt container. At least with the right podman version... But since
> you have an unsupported one, there might be some tricks there.
> 

I agree with that, but for some reason that was happening then. Dunno why.

> > 
> > ===
> > paunch apply --file
> > /var/lib/tripleo-config/container-startup-config/step_3/keystone.json
> > --config-id tripleo_step3 --managed-by tripleo-Standalone
> > paunch apply --file
> > /var/lib/tripleo-config/container-startup-config/step_3/horizon.json
> > --config-id tripleo_step3 --managed-by tripleo-Standalone
> > ===
> > 
> > The problem is, when I do start the Keystone container, the Horizon
> > container stops to work. If I start the Horizon container, the Keystone
> > container stops to work (facepalm).
> 
> might also be due to a configuration you've passed and services are trying
> to hit on the same port/device at some point? Seems unrelated to the whole
> SELinux story, unless you can point to denials. Still. Wrong podman version,
> can't do anything good to the whole stability.
> 
> > 
> > Also, if I reboot the VM, nothing works anymore and I need to reinstall
> > everything again.
> > 
> > I don't know what else could be done. What do y'all have in mind?

I managed to correct all of this later.

(In reply to Julie Pichon from comment #17)
> The podman version is still wrong indeed (minimum is podman-1.6.4-15).
> 
> Additionally the SELinux denials look new and are related to rhsmcertd_t
> this time? Could there be a problem even earlier in the process when setting
> up the modules and subscription manager? This may explain why the virt
> module stream didn't show up either:
> 
> type=AVC msg=audit(1607540395.240:381): avc:  denied  { read } for 
> pid=20402 comm="rhsmcertd-worke" name="virt.module" dev="dm-0" ino=136
> scontext=system_u:system_r:rhsmcertd_t:s0
> tcontext=system_u:object_r:root_t:s0 tclass=file permissive=0

I don't know why this happened, but I'm sure I've set the right repositories. The problem is the playbook I ran didn't change the modules to the right version. That could be the cause (?).
> 
> This looks like the virt module configuration file couldn't be read? I
> haven't seen this before. Was there any error when setting up the
> subscriptions and modules otherwise?

See above comment.

(In reply to Julie Pichon from comment #18)
> The *.module read errors look a lot like bug 1775975 which indicate that the
> label for those files should be etc_t, not root_t like in the error message. 
> 
> # restorecon -Rv /etc/dnf/modules.d
> 
> should resolve this. This can be confirmed after with `ls -alZ
> /etc/dnf/modules.d`.

Julie, this one is the golden trick, because I managed to solve the selinux context permission with this later and that solved the problem.

> 
> I think the advanced-virt-for-rhel-8-x86_64-rpms is missing from the list of
> repos though, that's where virt:8.2 comes from... But I don't think this
> should impact the podman version issue, as this comes from the
> container-tools:2.0 stream.

You're right, but the official documentation doesn't mention it. I'll ask them to update the documentation with this information, otherwise it won't be possible to use the correct module version.

Julie/Cédric,

Good news! After all your tips and recommendations, I finally managed to deploy the All-In-One correctly (despite the official documentation is wrong regarding these steps) and the SELinux issue was solved. Check it out:

+++

$ openstack server list

+--------------------------------------+-----------+--------+-------------------------------------+---------+--------+
| ID                                   | Name      | Status | Networks                            | Image   | Flavor |
+--------------------------------------+-----------+--------+-------------------------------------+---------+--------+
| c2533014-7f3a-45f8-a97e-d09b67fa65da | myserver2 | ACTIVE | private=192.168.4.77, 192.168.1.113 | cirros4 | tiny   |
| 333a8678-7369-4e1e-956b-c1b281814e0d | myserver1 | ACTIVE | private=192.168.4.56, 192.168.1.112 | cirros5 | tiny   |
+--------------------------------------+-----------+--------+-------------------------------------+---------+--------+

$ getenforce 

Enforcing

$ hostnamectl 

   Static hostname: openstack16.example.local
         Icon name: computer-vm
           Chassis: vm
        Machine ID: f85aa0aaf6894ab792bd555197a13152
           Boot ID: f28e812e50714cbea29a85e64b1662a5
    Virtualization: kvm
  Operating System: Red Hat Enterprise Linux 8.2 (Ootpa)
       CPE OS Name: cpe:/o:redhat:enterprise_linux:8.2:GA
            Kernel: Linux 4.18.0-193.29.1.el8_2.x86_64
      Architecture: x86-64

$ sudo dnf repolist

repo id                                             repo name
advanced-virt-for-rhel-8-x86_64-rpms                Advanced Virtualization for RHEL 8 x86_64 (RPMs)
ansible-2.9-for-rhel-8-x86_64-rpms                  Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs)
fast-datapath-for-rhel-8-x86_64-rpms                Fast Datapath for RHEL 8 x86_64 (RPMs)
openstack-16.1-for-rhel-8-x86_64-rpms               Red Hat OpenStack Platform 16.1 for RHEL 8 x86_64 (RPMs)
rhel-8-for-x86_64-appstream-eus-rpms                Red Hat Enterprise Linux 8 for x86_64 - AppStream - Extended Update Support (RPMs)
rhel-8-for-x86_64-baseos-eus-rpms                   Red Hat Enterprise Linux 8 for x86_64 - BaseOS - Extended Update Support (RPMs)
rhel-8-for-x86_64-highavailability-eus-rpms         Red Hat Enterprise Linux 8 for x86_64 - High Availability - Extended Update Support (RPMs)
rhel-8-for-x86_64-highavailability-rpms             Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs)

$ sudo dnf module list --enabled

Updating Subscription Management repositories.

Fast Datapath for RHEL 8 x86_64 (RPMs)                                                                                 27 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - BaseOS - Extended Update Support (RPMs)                                        22 kB/s | 2.4 kB     00:00    
Red Hat OpenStack Platform 16.1 for RHEL 8 x86_64 (RPMs)                                                               36 kB/s | 2.4 kB     00:00    
Advanced Virtualization for RHEL 8 x86_64 (RPMs)                                                                       26 kB/s | 2.8 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs)                                                       27 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - High Availability - Extended Update Support (RPMs)                             33 kB/s | 2.4 kB     00:00    
Red Hat Ansible Engine 2.9 for RHEL 8 x86_64 (RPMs)                                                                    36 kB/s | 2.4 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - AppStream - Extended Update Support (RPMs)                                     39 kB/s | 2.8 kB     00:00    

Advanced Virtualization for RHEL 8 x86_64 (RPMs)
Name                          Stream               Profiles                             Summary                                                       
virt                          8.2 [e]              common                               Virtualization module                                         

Red Hat Enterprise Linux 8 for x86_64 - AppStream - Extended Update Support (RPMs)
Name                          Stream               Profiles                             Summary                                                       
container-tools               2.0 [e]              common [d]                           Common tools and dependencies for container runtimes          
httpd                         2.4 [d][e]           common [d], devel, minimal           Apache HTTP Server                                            
python36                      3.6 [d][e]           build, common [d]                    Python programming language, version 3.6                      
ruby                          2.5 [d][e]           common [d]                           An interpreter of object-oriented scripting language          
satellite-5-client            1.0 [d][e]           common [d], gui                      Red Hat Satellite 5 client packages                           

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

$ sudo ls -alZ /etc/dnf/modules.d

total 24
drwxr-xr-x. 2 root root system_u:object_r:etc_t:s0     150 dez 14 20:34 .
drwxr-xr-x. 8 root root system_u:object_r:etc_t:s0     128 mai  7  2020 ..
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  74 dez 11 19:10 container-tools.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  54 dez 14 20:34 httpd.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  60 dez  1 18:13 python36.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  52 dez  1 18:42 ruby.module
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      80 set 24 18:58 satellite-5-client.module
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      52 dez 11 19:22 virt.module

$ sudo rpm -qa | grep openstack

openstack-tripleo-common-containers-11.4.1-1.20200914165651.el8ost.noarch
python3-openstackclient-4.0.1-1.20200817092223.bff556c.el8ost.noarch
openstack-selinux-0.8.24-1.20200914163011.26243bf.el8ost.noarch
openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch
python3-openstacksdk-0.36.4-0.20200715054250.76d3b29.el8ost.noarch
python-openstackclient-lang-4.0.1-1.20200817092223.bff556c.el8ost.noarch
puppet-openstacklib-15.4.1-0.20200403203429.5fdf43c.el8ost.noarch
openstack-heat-api-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-ironic-python-agent-builder-2.1.1-1.20200914175356.65d0f80.el8ost.noarch
openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch
openstack-heat-engine-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-tripleo-puppet-elements-11.2.2-0.20200701163410.432518a.el8ost.noarch
ansible-role-openstack-operations-0.0.1-0.20200311080930.274739e.el8ost.noarch
openstack-heat-monolith-13.0.3-1.20200914171254.48b730a.el8ost.noarch
puppet-openstack_extras-15.4.1-0.20200528113453.371931c.el8ost.noarch
openstack-heat-common-13.0.3-1.20200914171254.48b730a.el8ost.noarch
openstack-tripleo-image-elements-10.6.2-0.20200528043425.7dc0fa1.el8ost.noarch
openstack-heat-agents-1.10.1-0.20200311091123.96b819c.el8ost.noarch
openstack-tripleo-common-11.4.1-1.20200914165651.el8ost.noarch

$ sudo rpm -qa | grep podman

podman-1.6.4-16.module+el8.2.0+7659+b700d80e.x86_64

+++

So here is the correct steps in order to deploy it without SELinux issues:

===

Chapter3:

1 - For the Step 8, the right repositories to be enable are:

[stack@all-in-one]$ sudo subscription-manager repos \
--enable=rhel-8-for-x86_64-baseos-eus-rpms \
--enable=rhel-8-for-x86_64-appstream-eus-rpms \
--enable=rhel-8-for-x86_64-highavailability-eus-rpms \
--enable=ansible-2.9-for-rhel-8-x86_64-rpms \
--enable=openstack-16.1-for-rhel-8-x86_64-rpms \
--enable=fast-datapath-for-rhel-8-x86_64-rpms \
--enable=rhel-8-for-x86_64-highavailability-rpms \
--enable=advanced-virt-for-rhel-8-x86_64-rpms

Chapter 5:

2 - After Step 2 and before Step 3, run restorecon and make sure the /etc/dnf/modules.d gets the right SELinux context:

$ sudo restorecon -Rv /etc/dnf/modules.d
$ sudo ls -alZ /etc/dnf/modules.d

total 24
drwxr-xr-x. 2 root root system_u:object_r:etc_t:s0     150 dez 14 20:34 .
drwxr-xr-x. 8 root root system_u:object_r:etc_t:s0     128 mai  7  2020 ..
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  74 dez 11 19:10 container-tools.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  54 dez 14 20:34 httpd.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  60 dez  1 18:13 python36.module
-rw-r--r--. 1 root root unconfined_u:object_r:etc_t:s0  52 dez  1 18:42 ruby.module
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      80 set 24 18:58 satellite-5-client.module
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      52 dez 11 19:22 virt.module

===

Only doing this way the deployment is going to be able to set and use the right SELinux contexts and the instances' spawning are going to run flawlessly.

teamwork++

That's great news, thank you for the update!

Note that the restorecon step is only a workaround for another bug (I think bug 1775975), hopefully it won't be necessary forever. It may be useful to add a link to that bug in the documentation for that step as well, for the context?

I'm clearing everyone's needinfo because I don't think any further information is necessary on our part, but feel free to add it back if I missed a question! I agree on leaving this bug open until the documentation is up to date.

Hello there,

Good to see everything's fine now.

Alexon: I'm a bit surprised a restorecon is mandatory once you enable the repositories. Would be interesting to investigate a bit further, there might be an issue somewhere if this is actually needed.

Julie: there's still the question mark about dependency on the right podman version - not 100% sure about the state here.... Maybe it will hit another zstream? Since Lon took the relevant BZs, I'll let him clear things up on that field.

Cheers,

C.

(I see I actually didn't check the box to clear everybody's needinfo earlier, sorry about that!)

I ran a few additional tests about module enablement with subscription manager. I was able to confirm a couple of things, but I couldn't reproduce the bug.

1. Looking at a fresh 8.2 system and another one I keep up-to-date (but not with subscription manager), both are displaying the *wrong* file context for module files (root_t vs etc_t):

$ ls -lZ /etc/dnf/modules.d
total 12
-rw-r--r--. 1 root root system_u:object_r:root_t:s0 74 Dec 16 14:48 container-tools.module
-rw-r--r--. 1 root root system_u:object_r:root_t:s0 80 Dec 16 14:33 satellite-5-client.module
-rw-r--r--. 1 root root system_u:object_r:root_t:s0 53 Dec 16 14:33 virt.module

And it looks like bug 1775975 will only fix the issue from RHEL 8.3.

2. However, this does not prevent me from setting up modules:

$ sudo dnf module disable container-tools:rhel8
[...]
$ sudo dnf module enable container-tools:2.0
[...]
$ sudo dnf module list --enabled
Updating Subscription Management repositories.
Last metadata expiration check: 0:17:52 ago on Wed 16 Dec 2020 14:52:48 GMT.
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
Name                 Stream        Profiles         Summary                                              
container-tools      2.0 [e]       common [d]       Common tools and dependencies for container runtimes 
satellite-5-client   1.0 [d][e]    common [d], gui  Red Hat Satellite 5 client packages                  
virt                 rhel [d][e]   common [d]       Virtualization module                                

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

$ sudo grep denied /var/log/audit/audit.log
# Returns nothing

The expected podman comes up too. I'm using the Employee SKU in a fresh RHEL 8.2 VM, and couldn't reproduce the denial mentioned in comment 17. Not sure what could have caused the issue.

The document in comment 7 now shows the information about enabling modules and the right list of repositories. From comment 26, it doesn't look like the restorecon step is mandatory or related to the module enablement issues, so I'm closing this as resolved. If the bug where the module silently failed the 'enable' step can be produced, please open a new bug. Thank you!